A Fixed Pattern Noise Calibration System and Method for Image Sensors Based on FPGA

Abstract

This application relates to an FPGA-based image sensor fixed pattern noise calibration system and method. The system includes: a parameter storage module for storing calibrated row FPN calibration parameters and column FPN calibration parameters, wherein the row FPN calibration parameters are stored according to the number of rows in the image sensor, and the column FPN calibration parameters are stored according to the number of columns in the image sensor; a row calibration module for receiving image data from the image sensor and performing calibration based on the row FPN calibration parameters to obtain row calibration result parameters; and a column calibration module, connected to the row calibration module, for receiving the row calibration result parameters and performing calibration based on the row calibration result parameters and column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode. This addresses the problem in related technologies of how to effectively correct fixed pattern noise in multi-slope HDR mode under limited storage resource constraints.

Description

Technical Field

[0001] This application relates to the field of image sensor calibration technology, and in particular to an FPGA-based image sensor fixed pattern noise calibration system and method. Background Technology

[0002] CMOS image sensors are widely used due to their high integration and low power consumption, but their ubiquitous fixed pattern noise (FPN) severely affects image uniformity and quality. FPN mainly originates from differences in pixel dark current, column amplifier offset / gain inconsistency, and row readout path deviation, resulting in a fixed striped pattern on the captured image. This problem is particularly prominent in multi-slope HDR mode: to extend the dynamic range, the sensor needs to rapidly acquire and fuse multiple frames with different exposure times. Changes in exposure time significantly affect the response curves of the row and column circuits, causing the FPN performance to dynamically change with the exposure range, drastically increasing the difficulty of calibration.

[0003] For FPN correction, existing technical solutions typically utilize high-precision pixel-by-pixel calibration. While this approach can effectively suppress noise, it requires a huge amount of storage for calibration parameters (for example, for a 2048×1216 resolution sensor, several MB of storage is needed in a single mode, and more than 50 MB in multi-slope HDR mode), far exceeding the on-chip storage capacity of low-cost FPGAs (Field Programmable Gate Arrays), making it difficult to implement on embedded platforms.

[0004] Therefore, no effective solution has yet been proposed for FPGA-based image sensor FPN calibration systems to achieve effective correction of fixed pattern noise in multi-slope HDR modes under limited storage resource constraints. Summary of the Invention

[0005] This application provides an FPGA-based image sensor fixed pattern noise calibration system and method to at least solve the problem in related technologies of how to effectively correct fixed pattern noise in multi-slope HDR modes under limited storage resource constraints.

[0006] In a first aspect, embodiments of this application provide an FPGA-based image sensor fixed pattern noise calibration system for correcting fixed pattern noise in multi-slope HDR mode, the system comprising:

[0007] The parameter storage module is used to store the calibrated row FPN calibration parameters and column FPN calibration parameters, wherein the row FPN calibration parameters are stored according to the number of rows of the image sensor, and the column FPN calibration parameters are stored according to the number of columns of the image sensor.

[0008] The line calibration module is used to receive image data from the image sensor, perform calibration based on the line FPN calibration parameters, and obtain line calibration result parameters.

[0009] The column calibration module, connected to the row calibration module, is used to receive the row calibration result parameters and perform calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode.

[0010] In one embodiment, when storing the calibrated row FPN calibration parameters and column FPN calibration parameters, the parameter storage module is further configured to:

[0011] The row FPN calibration parameters obtained by the image sensor through multiple HDR modes are acquired, and the row FPN calibration parameters are stored in the calibration parameter RAM in the parameter storage module.

[0012] Obtain the column FPN calibration parameters obtained from the calibration based on the black point image and the white point image, and store the column FPN calibration parameters in the calibration parameter RAM;

[0013] Multiple random images with uniform gray levels acquired by the image sensor constitute a random image test set. The mean square error value of each image in the random image test set is calculated, and an evaluation index for evaluating the uniformity of the image sensor is determined based on the mean square error value of each image.

[0014] The evaluation index and the calibration parameters RAM are stored in the non-volatile memory; wherein, the evaluation index is used to verify whether the correction effect of the row FPN calibration parameters and the column FPN calibration parameters meets the standard.

[0015] In one embodiment, when the parameter storage module acquires the row FPN calibration parameters obtained by calibrating the image sensor through multiple HDR modes, it is further configured to:

[0016] The image sensor is controlled to acquire at least four levels of uniform grayscale images in each HDR mode;

[0017] For each row of pixels in the uniform grayscale image, the slope of the high-gain dark area, the intercept of the high-gain dark area, the slope of the low-gain bright area, and the intercept of the low-gain bright area are obtained by fitting.

[0018] Based on the high-gain dark area slope, the high-gain dark area intercept, the low-gain bright area slope, and the low-gain bright area intercept, calculate the knee pixel value and scaling factor for each row of pixels;

[0019] The knee pixel value and the scaling factor are used as the row FPN calibration parameters.

[0020] In one embodiment, when the parameter storage module obtains the column FPN calibration parameters calibrated based on the black and white images, it is further configured to:

[0021] The image sensor is controlled to acquire black and white images;

[0022] Based on the black field image and the white field image, the gain coefficient and offset are fitted to each column of pixels;

[0023] The gain coefficient and the offset are used as the column FPN calibration parameters.

[0024] In one embodiment, when the line calibration module receives image data from the image sensor, performs calibration based on the line FPN calibration parameters, and obtains line calibration result parameters, it is further configured to:

[0025] Receive image data from the image sensor, and obtain the corresponding row FPN calibration parameters from the parameter storage module based on the row index of the image data;

[0026] The original pixel values ​​in the image data are compared with the knee pixel values ​​of the row FPN calibration parameters;

[0027] If the original pixel value is less than the knee pixel value, then the original pixel value S is output as the row calibration result parameter;

[0028] If the original pixel value is greater than or equal to the knee pixel value, the difference between the original pixel value and the knee pixel value is scaled based on the scaling factor in the row FPN calibration parameters, and the scaling result is added to the knee pixel value to obtain the row calibration result parameters.

[0029] In one embodiment, when the column calibration module receives the row calibration result parameters and performs calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode, it is further configured to:

[0030] Based on the column index of the image data, the corresponding column FPN calibration parameters are obtained from the parameter storage module;

[0031] Receive the row calibration result parameters, and subtract the offset from the column FPN calibration parameters to obtain the offset correction value;

[0032] The offset correction value is divided by the gain coefficient in the column FPN calibration parameters to obtain the calibrated image data.

[0033] In one embodiment, the image data output by the image sensor, the line calibration result parameters output by the line calibration module, and the image data output by the line calibration module are all transmitted within the system via an AXI4-Stream signal.

[0034] In one embodiment, the system further includes a RISC-V soft core mounted on an AXI bus; the RISC-V soft core is used to control the row FPN calibration parameters and column FPN calibration parameters in the non-volatile memory to be loaded into the calibration parameter RAM of the parameter storage module when the system is powered on.

[0035] In one embodiment, the system further includes an image caching module and an Ethernet communication module;

[0036] The image caching module is connected to the column calibration module and is used to receive and cache the calibrated image data output by the column calibration module.

[0037] The Ethernet communication module is connected to the image cache module and is used to package the image data cached in the image cache module and send it to an external host.

[0038] The Ethernet communication module is also used to receive control commands and calibration parameter files from the external host, and to store the calibration parameter RAM in the non-volatile memory or write it into the parameter storage module according to the control commands.

[0039] Secondly, embodiments of this application provide a fixed-pattern noise calibration method for an image sensor based on an FPGA. The method is applied to the aforementioned system and includes:

[0040] The calibrated row FPN calibration parameters and column FPN calibration parameters are stored, wherein the row FPN calibration parameters are stored according to the number of rows of the image sensor, and the column FPN calibration parameters are stored according to the number of columns of the image sensor.

[0041] Receive image data from the image sensor and perform calibration based on the row FPN calibration parameters to obtain row calibration result parameters;

[0042] The system receives the row calibration result parameters and performs calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode.

[0043] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an FPGA-based image sensor fixed pattern noise calibration method as described in the second aspect above.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the second aspect above.

[0045] The fixed pattern noise calibration system and method for image sensors based on FPGA provided in this application have at least the following technical effects.

[0046] A parameter storage module stores calibrated row FPN calibration parameters and column FPN calibration parameters. The row FPN calibration parameters are stored based on the number of rows in the image sensor, and the column FPN calibration parameters are stored based on the number of columns. By compressing the parameter storage from the total number of pixels to the sum of the number of rows and columns, storage resource overhead is significantly reduced, enabling the system to be implemented on a low-cost FPGA platform with limited storage resources. This effectively solves the problem of large-capacity parameter storage in embedded scenarios. A row calibration module receives image data from the image sensor and performs calibration based on the row FPN calibration parameters to obtain row calibration result parameters. By applying targeted corrections row by row, it can accurately compensate for horizontal fixed noise caused by factors such as row readout path deviation and row buffer inconsistency, significantly improving the uniformity of the image in the row direction. A column calibration module, connected to the row calibration module, receives the row calibration result parameters and performs calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode. Building upon row calibration, further column-based correction is applied to address vertical stripe noise caused by inconsistencies in column amplifier gain / offset. This cascaded row-column correction achieves synergistic suppression of bidirectional fixed-pattern noise. Particularly in multi-slope HDR mode, it adapts to the dynamic changes in column response curves under different exposure times, effectively suppressing FPN exacerbated by exposure time switching, and significantly improving the image quality uniformity and overall visual effect of high dynamic range images. This addresses the challenge in related technologies of effectively correcting fixed-pattern noise in multi-slope HDR modes under limited storage resource constraints.

[0047] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1This is a system structure block diagram illustrating an FPGA-based image sensor fixed pattern noise calibration system according to an exemplary embodiment;

[0050] Figure 2 This is a schematic diagram illustrating the structure of a lookup table according to an exemplary embodiment;

[0051] Figure 3 This is a schematic diagram illustrating the relationship between the knee point and the slope according to an exemplary embodiment;

[0052] Figure 4 This is a flowchart of a fixed-pattern noise calibration method for image sensors based on FPGA;

[0053] Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0055] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0056] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0057] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0058] In modern imaging systems, CMOS image sensors are widely used due to their high integration and low power consumption. However, their ubiquitous fixed pattern noise (FPN) severely affects image uniformity and quality. FPN mainly originates from differences in pixel dark current, column amplifier offset / gain inconsistency, and row readout path deviation, and it presents a fixed stripe or grid pattern in space. This problem is particularly prominent in multi-slope HDR mode: in order to extend the dynamic range, the sensor needs to rapidly acquire and fuse multiple frames of images with different exposure times. Changes in exposure time significantly affect the response curves of row and column circuits, causing the performance of FPN to dynamically change with the exposure range, drastically increasing the difficulty of calibration.

[0059] The fundamental source of line FPN is the inter-line bias difference caused by knee reset. Multi-slope HDR performs a "forced reset" on the pixel capacitors when each slope segment reaches the knee. This reset is triggered row by row by the line drive circuit, and different rows have slight differences in voltage, timing, residual charge, and knee threshold during reset. Furthermore, parasitic coupling exists between the reset line and the line drive network, and the reset transient creates a fixed bias in the line direction, further amplifying the level difference between the CDS (reset readout, signal readout) of each line. Because multi-slope HDR repeatedly experiences multiple reset segments, these line biases accumulate in different exposure segments, resulting in a stable brightness difference between lines under the same lighting conditions. Combined with the process variations in the line readout link (line selection switch, driver) itself, this ultimately forms a structural line Fixed Pattern Noise.

[0060] The root cause of column-level FPN lies in the inherent inconsistencies of the column-level analog readout link. Each column pixel signal must pass through an independent column amplifier, sample-and-hold capacitor, comparator, or column ADC. Due to unavoidable process differences in analog circuitry, the input bias voltage, amplifier bias current, reference voltage, sampling capacitor size, and ADC zero and gain of each column will vary slightly, causing a fixed offset or linear scaling difference in the signals of each column under the same illumination. In addition, the parallel structure of column readout means that parasitic resistance, capacitance, and coupling differences with row drive or reset lines between columns will create additional offsets. In multi-slope HDR, the pixel signal after knee reset needs to be sampled across columns, and the sampling timing and switching charge injection of different columns will also amplify the mismatch between columns. These structural differences remain constant when the timing is stable and the illumination is consistent, forming vertical stripe noise, i.e., column fixed pattern noise, which mainly includes two types: offset and gain.

[0061] Existing technologies for FPN calibration typically face several insurmountable drawbacks. First, while high-precision pixel-by-pixel calibration schemes effectively suppress noise, the required storage of calibration parameters is enormous (e.g., for a 2048×1216 resolution sensor, several MB of storage is needed in a single mode, easily exceeding 50 MB in multi-exposure mode), far exceeding the on-chip storage capacity of low-cost FPGAs, making implementation impossible on embedded platforms. Second, simplified schemes such as grid interpolation, used to reduce storage, while compressing parameters, introduce a large number of multiply-accumulate operations (DSP), failing to meet the real-time processing requirements of high frame rates exceeding 200fps. Third, traditional pure column calibration or global polynomial fitting methods have extremely poor suppression of row-direction noise, with residual noise typically exceeding 3DN, failing to meet the stringent requirements of high-end applications for image non-uniformity below 0.3%. Furthermore, existing solutions generally lack efficient on-site recalibration mechanisms, unable to cope with parameter changes caused by sensor temperature drift and aging. In summary, existing technologies struggle to simultaneously achieve high-precision, multi-exposure compatible, and high-frame-rate FPN correction on resource-constrained FPGA platforms, creating significant technological bottlenecks and market gaps.

[0062] Therefore, no effective solution has yet been proposed for FPGA-based image sensor FPN calibration systems to achieve effective correction of fixed pattern noise in multi-slope HDR modes under limited storage resource constraints.

[0063] Based on the above, this application provides an FPGA-based image sensor fixed pattern noise calibration system and method.

[0064] In this document, it should be understood that the terms used may be technical means used to implement part of the present invention or other summary technical terms. For example, the terms may include:

[0065] Multiple-Slope HDR: A technique that optimizes highlight and shadow details through piecewise nonlinear response / multi-frame fusion to achieve wide dynamic range imaging;

[0066] FPN (Fixed Pattern Noise): A repetitive noise pattern in an image sensor that is spatially fixed and independent of the incident light, caused by differences in hardware manufacturing processes.

[0067] In a first aspect, embodiments of this application provide an FPGA-based image sensor fixed pattern noise calibration system. Figure 1 This is a system architecture block diagram illustrating an FPGA-based image sensor fixed pattern noise calibration system according to an exemplary embodiment. Figure 1 As shown, the system is used to correct fixed pattern noise in multi-slope HDR mode. The system includes:

[0068] The parameter storage module is used to store the calibrated row FPN calibration parameters and column FPN calibration parameters. The row FPN calibration parameters are stored according to the number of rows of the image sensor, and the column FPN calibration parameters are stored according to the number of columns of the image sensor.

[0069] The line calibration module is used to receive image data from the image sensor, perform calibration based on the line FPN calibration parameters, and obtain the line calibration result parameters.

[0070] The column calibration module, connected to the row calibration module, is used to receive row calibration result parameters and perform calibration based on the row calibration result parameters and column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode.

[0071] In summary, this application provides an FPGA-based image sensor fixed pattern noise calibration system. A parameter storage module stores calibrated row FPN calibration parameters and column FPN calibration parameters. The row FPN calibration parameters are stored according to the number of rows in the image sensor, and the column FPN calibration parameters are stored according to the number of columns. By compressing the parameter storage from the total number of pixels to the sum of the number of rows and columns, the storage resource overhead is significantly reduced, enabling the system to be implemented on a low-cost FPGA platform with limited storage resources. This effectively solves the problem of large-capacity parameter storage in embedded scenarios. The row calibration module receives image data from the image sensor and performs calibration based on the row FPN calibration parameters to obtain row calibration result parameters. By applying targeted corrections row by row, it can accurately compensate for horizontal fixed noise caused by factors such as row readout path deviation and row buffer inconsistency, significantly improving the uniformity of the image in the row direction. The column calibration module, connected to the row calibration module, receives the row calibration result parameters and performs calibration based on the row calibration result parameters and column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode. Building upon row calibration, further column-based correction is applied to address vertical stripe noise caused by inconsistencies in column amplifier gain / offset. This cascaded row-column correction achieves synergistic suppression of bidirectional fixed-pattern noise. Particularly in multi-slope HDR mode, it adapts to the dynamic changes in column response curves under different exposure times, effectively suppressing FPN exacerbated by exposure time switching, and significantly improving the image quality uniformity and overall visual effect of high dynamic range images. This addresses the challenge in related technologies of effectively correcting fixed-pattern noise in multi-slope HDR modes under limited storage resource constraints.

[0072] In one embodiment, Figure 2 This is a schematic diagram illustrating the structure of a lookup table according to an exemplary embodiment, such as... Figure 2As shown, the image sensor in this embodiment has a resolution of 1216 pixels high and 2048 pixels wide, and also supports three different HDR modes. If per-pixel FPN calibration is used, a 2490368 (2048*1216) long FPN calibration parameter lookup table is required; this embodiment only requires a 3264 (2048+1216) long FPN calibration parameter lookup table. Because of the smaller storage space required, multiple HDR modes can be supported simultaneously, such as... Figure 2 As shown. Figure 2 The row FPN calibration parameter lookup table is shared by the three HDR modes. The main difference between the three HDR modes is the knee point. The three HDR modes can cope with different shooting scenarios. The main reason for the row FPN is the difference in the knee point, so three different row FPN calibration parameter lookup tables are needed.

[0073] In one embodiment, the mathematical model of row FPN is:

[0074] S = Original signal value

[0075] I = Light intensity

[0076] KP = Knee Point

[0077] I KP = x-coordinate of the knee point

[0078] S KP =Y-coordinate of the knee point

[0079] A1 = Slope of the high-gain dark region (first segment of the response curve)

[0080] B1 = Intercept of the high-gain dark region (first segment of the response curve)

[0081] A2 = Slope of the low-gain bright region (second segment of the response curve)

[0082] B2 = Intercept of the low-gain bright region (second segment of the response curve)

[0083] A two-slope HDR response curve can be represented by a piecewise function:

[0084]

[0085] In the formula, S1 represents the signal value of the high-gain dark area, and S2 represents the signal value of the low-gain bright area.

[0086] Solve for the x-coordinate of the knee point I KP (Light intensity at the knee):

[0087]

[0088] Solve for the ordinate S of the knee point KP (Knee pixel value):

[0089]

[0090] In one embodiment, when storing the calibrated row FPN calibration parameters and column FPN calibration parameters, the parameter storage module is further configured to:

[0091] The line FPN calibration parameters obtained by the image sensor through multiple HDR modes are acquired and stored in the calibration parameter RAM in the parameter storage module.

[0092] Obtain the column FPN calibration parameters obtained from the black and white images, and store the column FPN calibration parameters in the calibration parameter RAM.

[0093] Multiple random images with uniform gray levels acquired by an image sensor are used to form a random image test set. The mean square error value of each image in the random image test set is calculated, and the evaluation index for evaluating the uniformity of the image sensor is determined based on the mean square error value of each image.

[0094] Optionally, the row-column joint correction effect is evaluated using a random image test set. Images with random uniform gray levels are acquired using an image sensor; in this embodiment, eight images are acquired as the test set. The mean MSE of these eight images is used to evaluate the uniformity of the calibrated image sensor. In this embodiment, the mean MSE is saved to... Figure 1 The “non-volatile memory” in the context.

[0095] The evaluation metrics and calibration parameters are stored in RAM to non-volatile memory; the evaluation metrics are used to verify whether the correction effect of the row FPN calibration parameters and column FPN calibration parameters meets the standard.

[0096] Optionally, the calibration parameters can be stored in FLASH memory for automatic loading upon FPGA power-on. In this embodiment, since the FPGA's RAM is used as the carrier of the lookup table, power-off saving is not supported. After calibration, a command needs to be sent to the FPGA to load the data. Figure 1 The inputs in the "calibration parameter RAM" are transferred to the "non-volatile memory". Each time the FPGA starts, it moves data from the "non-volatile memory" to the "calibration parameter RAM".

[0097] By storing row parameters calibrated for various HDR modes and column parameters calibrated based on black and white points, the adaptability of the calibration model to different imaging conditions is ensured. Furthermore, the mean MSE value is calculated using multi-grayscale test images acquired by the image sensor as a quantitative evaluation index, realizing an objective and quantitative assessment of the joint row and column correction effect, thereby verifying whether the calibration effect meets the standard. Finally, the evaluation index and parameters are stored together in non-volatile memory (such as FLASH), so that the calibration parameters can be saved even when power is off and automatically loaded into the calibration parameter RAM of the FPGA every time the system is powered on, realizing the solidified management of calibration parameters and the system's ready-to-use capability, effectively ensuring the consistency and durability of the calibration effect.

[0098] In one embodiment, when the parameter storage module acquires the row FPN calibration parameters obtained by calibrating the image sensor through multiple HDR modes, it is further configured to:

[0099] Control the image sensor to acquire at least four levels of uniform grayscale images in each HDR mode;

[0100] The slope of the high-gain dark area, the intercept of the high-gain dark area, the slope of the low-gain bright area, and the intercept of the low-gain bright area are obtained by fitting each row of pixels in a uniform grayscale image.

[0101] Based on the slope of the high-gain dark area, the intercept of the high-gain dark area, the slope of the low-gain bright area, and the intercept of the low-gain bright area, the knee pixel value and the scaling factor are calculated for each row of pixels.

[0102] Use the knee pixel value and the scaling factor as the row FPN calibration parameters.

[0103] Optionally, multi-level uniform grayscale color fields are collected, and the FPN model parameters are calculated and loaded onto the FPGA. This embodiment supports three HDR modes, each requiring four images for calibration, for a total of 12 images. Each HDR mode has a different knee point, so individual calibration is necessary. At least four images must be sampled in this embodiment, as at least two points are needed to fit one slope. Since the HDR response curve consists of two slopes, at least four points are required. These four points on the response curve require four levels of grayscale images. After sampling, a PC is used to fit each column of model parameters. , , , The derivation and calculation show that: , These two parameters need to be loaded into the FPGA. Specifically, the parameters need to be loaded into... Figure 1 In the "Calibration Parameter RAM" section, the above-mentioned grayscale images require the image sensor to be pointed at a uniformly illuminated reference light source for capture.

[0104] By acquiring at least four levels of uniform grayscale images separately for each HDR mode, the two-segment response curves of the sensor under different exposure strategies (i.e., the slope and intercept of the high-gain region and the low-gain region) can be accurately fitted, thereby calculating the knee pixel value Skp and the scaling factor R12 that reflect its specific switching characteristics. This method compresses the calibration parameters required for each row from the complete response curve (A1, B1, A2, B2) to only two core parameters (Skp, R12), achieving significant reduction in storage without sacrificing the accuracy of the physical model. Simultaneously, separate calibration for different HDR modes ensures that the calibration parameters accurately match the dynamic response characteristics of each mode, thus solving the problem of FPN dynamically changing with exposure time and the poor calibration effect of traditional single parameter sets. Ultimately, while ensuring high-precision calibration results, it greatly reduces the demand for on-chip storage resources on the FPGA.

[0105] In one embodiment, when the parameter storage module obtains the column FPN calibration parameters calibrated based on the black and white images, it is further configured to:

[0106] Control the image sensor to acquire black and white images;

[0107] Gain coefficients and offsets are obtained by fitting each column of pixels based on black and white images;

[0108] Use the gain factor and offset as column FPN calibration parameters.

[0109] Optionally, acquire black and white images, calculate column FPN mode parameters, and load them onto the FPGA. Cover the image sensor with a lens cap and acquire black images. Use a uniform white reference light source on the image sensor and acquire white images. Fit the model parameters for each column: Gain and Offset. After calculation, load the two parameters: Gain and 1 / Offset onto the FPGA. It is necessary to load the above parameters into... Figure 1 In the "Calibration Parameters RAM".

[0110] By analyzing the black field image, the fixed offset introduced by the readout circuit for each column of pixels can be obtained. By analyzing the white field image and eliminating the offset, the gain coefficient for each column of pixels can be accurately calculated. Ultimately, only these two core parameters need to be stored per column, drastically reducing the storage requirement for column FPN calibration parameters from being proportional to the total number of pixels to being proportional only to the number of sensor columns. This compresses the massive pixel-level parameter table into a lightweight column-level parameter table. This fundamental compression enables high-precision real-time column FPN correction on low-end FPGA or embedded platforms with extremely limited storage resources and cost sensitivity, resolving the core contradiction of traditional solutions being difficult to deploy in low-cost hardware due to their huge parameter storage requirements.

[0111] In one embodiment, when the line calibration module receives image data from the image sensor, performs calibration based on the line FPN calibration parameters, and obtains the line calibration result parameters, it is further configured to:

[0112] Receive image data from the image sensor, and obtain the corresponding row FPN calibration parameters from the parameter storage module based on the row index of the image data;

[0113] Compare the raw pixel values ​​in the image data with the knee pixel values ​​of the row FPN calibration parameters;

[0114] If the original pixel value is less than the knee pixel value, the original pixel value S will be output as the row calibration result parameter.

[0115] If the original pixel value is greater than or equal to the knee pixel value, the difference between the original pixel value and the knee pixel value is scaled based on the scaling factor in the row FPN calibration parameters, and the scaling result is added to the knee pixel value to obtain the row calibration result parameters.

[0116] Optionally, each row of images needs to have four parameters, A1, B1, A2, and B2, fitted. If FPN calibration is required, only the inverse step needs to be performed. The purpose of the inverse step is to pull the second segment back to the scale of the first segment. The row FPN calibration formula is:

[0117]

[0118] In the formula, These are the pixel values ​​after row calibration. These are the original pixel values.

[0119] The row calibration module receives raw image data streams in AXI4-Stream format from the image sensor. For each pixel in the stream, the module first reads the corresponding row FPN calibration parameters from the calibration parameter RAM of the parameter storage module in real time, based on its row number (row index). These parameters specifically include the knee pixel value S. KP And the scaling factor R12. Then, the module compares the original pixel value S of the current pixel with S... KP Compare them. If S < S KP This indicates that the pixel is in the high-gain response region of the sensor, and the module directly outputs the original value S as the row calibration result without correction. If S ≥ S KP This indicates that the pixel has entered the low-gain response region of the sensor, and the module performs calibration calculations: first, it calculates the difference (S - S). KP Then multiply it by the scaling factor R12 to scale it, and finally combine the scaled result with S. KP The sums are then used to obtain the row calibration result parameters. The entire process is implemented in a pipelined manner, ensuring that each pixel is processed in real time as it flows through the module.

[0120] Through the segmented calibration operation based on knee point judgment, the sensor's response curve in the low-gain region is linearly mapped back to the numerical scale of the high-gain region, thereby eliminating the "knee point" jump in the response curve caused by dual-gain switching and ensuring the consistency of the sensor's response across the entire dynamic range. This not only effectively compensates for the fixed horizontal noise caused by differences in row-level circuitry, but also significantly improves the uniformity between image rows and enhances the image quality uniformity of HDR images.

[0121] In one embodiment, when the column calibration module receives the row calibration result parameters and performs calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode, it is further configured to:

[0122] Based on the column index of the image data, the corresponding column FPN calibration parameters are obtained from the parameter storage module;

[0123] Receive the row calibration result parameters and subtract the offset from the column FPN calibration parameters to obtain the offset correction value;

[0124] Divide the offset correction value by the gain coefficient in the FPN calibration parameters to obtain the calibrated image data.

[0125] Optionally, the mathematical model for FPN is as follows:

[0126]

[0127] The FPN calibration formula is as follows:

[0128]

[0129] In the formula, These are the calibrated pixel values. These are the pixel values ​​after row calibration.

[0130] The column calibration module receives the row calibration result parameter data stream (typically in AXI4-Stream format) from the row calibration module. For each pixel in the data stream, the module first reads the corresponding column FPN calibration parameters (offset and gain coefficient) from the calibration parameter RAM of the parameter storage module in real time, based on its column number (column index). Then, the module performs a two-step calibration calculation: First, it subtracts the column offset from the received row calibration result parameter S_row to obtain an intermediate value that eliminates the fixed offset deviation; second, it divides this intermediate value by the column gain coefficient Gain to correct the response amplitude difference caused by inconsistent column amplifier gain, finally outputting the calibrated pixel data. The entire process, similar to row calibration, is implemented in an efficient pipeline to ensure real-time continuous processing of the pixel stream.

[0131] Through the aforementioned calibration calculations that first subtract the offset and then divide the gain, the fixed offset and gain errors caused by the process deviation of the column-level readout circuit for each column of pixels are accurately compensated, thereby effectively eliminating the fixed stripe noise (column FPN) in the vertical direction of the image. Ultimately, this significantly improves the inter-column uniformity and overall visual quality of the image.

[0132] In one embodiment, the image data output by the image sensor, the line calibration result parameters output by the line calibration module, and the image data output by the line calibration module are all transmitted within the system via an AXI4-Stream signal. By employing a unified AXI4-Stream protocol to transmit all image data streams within the system, a standardized high-speed data processing pipeline is constructed.

[0133] In one embodiment, the system further includes a RISC-V soft core mounted on the AXI bus; the RISC-V soft core is used to control the row FPN calibration parameters and column FPN calibration parameters in the non-volatile memory to be loaded into the calibration parameter RAM of the parameter storage module when the system is powered on.

[0134] In one embodiment, the system further includes an image caching module and an Ethernet communication module;

[0135] The image caching module is connected to the column calibration module and is used to receive and cache the calibrated image data output by the column calibration module;

[0136] The Ethernet communication module is connected to the image cache module and is used to package the image data cached in the image cache module and send it to an external host.

[0137] The Ethernet communication module is also used to receive control commands and calibration parameter files from an external host, and to store the calibration parameters in RAM to non-volatile memory or write them to the parameter storage module according to the control commands.

[0138] Optionally, the system includes an image sensor, non-volatile memory, a host computer, and an FPGA; wherein the FPGA includes: a row FPN calibration module, a column FPN calibration module, a calibration parameter RAM, an image buffer module, and an Ethernet UDP wrapper module (Ethernet communication module).

[0139] The "calibration parameter RAM" is copied to the corresponding offset address in the "non-volatile memory" when the system starts / powers on / reboots. In this embodiment, the "non-volatile memory" is 32MB of NorFlash. In this embodiment, the above-mentioned "calibration parameter RAM" and "Flash controller" are mounted on the AXI bus and read / write control is performed by a RISC-V soft core.

[0140] Figure 1 The image sensor has an FPGA for multi-slope HDR exposure control and image decoding. In this embodiment, the above image is decoded into an AXI4-Stream signal. The multi-slope HDR exposure in this embodiment consists of two slopes. The multi-slope HDR sensor switches the conversion gain (slope) at the moment when the pixel charge is close to the "knee point" of overexposure, thereby capturing signals of different brightness in segments in one exposure and finally synthesizing a high dynamic range image. Figure 3 This is a schematic diagram illustrating the relationship between the knee point and the slope according to an exemplary embodiment, such as... Figure 3 As shown, Figure 3 The horizontal axis I represents Light Intensity, measured in lux, and the vertical axis S represents Raw Signal (the raw image signal, i.e., the raw pixel value), measured in digital form.

[0141] Figure 1 The input to the "line FPN calibration module" is the AXI4-Stream signal obtained by decoding the "image sensor"; the output of the "line FPN calibration module" is the AXI4-Stream signal. Figure 1 The input to the "column FPN calibration module" is as follows: The "row FPN calibration module" needs to read the corresponding calibration parameters from the "calibration parameter RAM" and use the calibration parameters to perform pipelined addition and pipelined multiplication on the input AXI4-Stream signal.

[0142] Figure 1The input of the "column FPN calibration module" is the AXI4-Stream signal output by the "row FPN calibration module"; the AXI4-Stream signal output by the "column FPN calibration module" is the input of the "image buffer module"; the "column FPN calibration module" needs to read the corresponding calibration parameters from the "calibration parameter RAM" and use the calibration parameters to perform pipelined addition and pipelined multiplication on the input AXI4-Stream signal.

[0143] Figure 1 The input of the "image cache module" is the AXI4-Stream signal output by the "column FPN calibration module"; the AXI4-Stream signal output by the "image cache module" serves as the input of the "Ethernet UDP packet encapsulation module"; in this embodiment, the "image cache module" mainly consists of a DDR3 controller with an AXI4 interface, and the image cache module also includes some glue modules that can use the DDR3 memory as a large-capacity FIFO;

[0144] Figure 1 The input of the "Ethernet UDP Wrapper Module" is the AXI4-Stream signal input of the "Image Buffer Module"; the output of the "Ethernet UDP Wrapper Module" is Ethernet; the "host" can receive image data processed by the FPGA via Ethernet; in addition to receiving images, the "host" is also responsible for sending control signals, such as sending the "calibration parameter file" to the FPGA, and also has a command to save the "calibration parameter file" to the "non-volatile memory".

[0145] In summary, this application provides an FPGA-based image sensor fixed pattern noise calibration system. A parameter storage module stores calibrated row FPN calibration parameters and column FPN calibration parameters. The row FPN calibration parameters are stored according to the number of rows in the image sensor, and the column FPN calibration parameters are stored according to the number of columns. By compressing the parameter storage from the total number of pixels to the sum of the number of rows and columns, the storage resource overhead is significantly reduced, enabling the system to be implemented on a low-cost FPGA platform with limited storage resources. This effectively solves the problem of large-capacity parameter storage in embedded scenarios. The row calibration module receives image data from the image sensor and performs calibration based on the row FPN calibration parameters to obtain row calibration result parameters. By applying targeted corrections row by row, it can accurately compensate for horizontal fixed noise caused by factors such as row readout path deviation and row buffer inconsistency, significantly improving the uniformity of the image in the row direction. The column calibration module, connected to the row calibration module, receives the row calibration result parameters and performs calibration based on the row calibration result parameters and column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode. Building upon row calibration, further column-based correction is applied to address vertical stripe noise caused by inconsistencies in column amplifier gain / offset. This cascaded row-column correction achieves synergistic suppression of bidirectional fixed-pattern noise. Particularly in multi-slope HDR mode, it adapts to the dynamic changes in column response curves under different exposure times, effectively suppressing FPN exacerbated by exposure time switching, and significantly improving the image quality uniformity and overall visual effect of high dynamic range images. This addresses the challenge in related technologies of effectively correcting fixed-pattern noise in multi-slope HDR modes under limited storage resource constraints.

[0146] This application also includes the following beneficial effects:

[0147] Extremely low storage requirements: a single calibration file requires only 1216+2048 coefficients (<10KB), and the total storage for more than 5 exposure modes is still <50KB, which is more than 500 times more compressed than the traditional pixel-by-pixel solution. It can be completely placed in the FPGA on-chip BRAM without the need for external DDR.

[0148] Computational resources are close to zero: each pixel requires only 2 table lookups + 1 addition / subtraction, DSP hard core usage is 0, LUT / FF resources are less than 1000, and it supports full pipeline real-time processing of 2048×1216@250fps and above;

[0149] High correction accuracy: dark field RMS noise <0.08DN (12bit), bright field non-uniformity <0.3%, comparable to 16-bit pixel-by-pixel LUT, PSNR difference <0.2dB;

[0150] Zero-delay switching for multiple exposures: Seamless switching between HDR short / medium / long exposure modes can be completed by simply switching two LUT pointers, without the need to reload the large table;

[0151] Supports one-click on-site recalibration in 2 seconds: without stopping the machine or using an external computer, new calibration files can be quickly generated and loaded in the actual use environment, solving problems such as temperature drift, aging, and optical differences;

[0152] Significantly improves HDR image quality: After complete removal of line noise, subsequent HDR fusion ghosting is reduced by 0.8 to 1.5 stops.

[0153] Secondly, embodiments of this application provide a fixed-pattern noise calibration method for image sensors based on FPGA. Figure 4 This is a flowchart of a fixed-pattern noise calibration method for image sensors based on FPGA, such as... Figure 4 As shown, the method includes:

[0154] Step S101: Store the calibrated row FPN calibration parameters and column FPN calibration parameters, wherein the row FPN calibration parameters are stored according to the number of rows of the image sensor, and the column FPN calibration parameters are stored according to the number of columns of the image sensor.

[0155] Step S102: Receive image data from the image sensor and perform calibration based on the row FPN calibration parameters to obtain the row calibration result parameters.

[0156] Step S103: Receive row calibration result parameters, and perform calibration based on row calibration result parameters and column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode.

[0157] In summary, this application provides an FPGA-based image sensor fixed pattern noise calibration method. A parameter storage module stores calibrated row FPN calibration parameters and column FPN calibration parameters. The row FPN calibration parameters are stored according to the number of rows in the image sensor, and the column FPN calibration parameters are stored according to the number of columns. By compressing the parameter storage from the total number of pixels to the sum of the number of rows and columns, the storage resource overhead is significantly reduced, enabling the system to be implemented on a low-cost FPGA platform with limited storage resources. This effectively solves the problem of large-capacity parameter storage in embedded scenarios. The row calibration module receives image data from the image sensor and performs calibration based on the row FPN calibration parameters to obtain row calibration result parameters. By applying targeted corrections row by row, it can accurately compensate for horizontal fixed noise caused by factors such as row readout path deviation and row buffer inconsistency, significantly improving the uniformity of the image in the row direction. The column calibration module, connected to the row calibration module, receives the row calibration result parameters and performs calibration based on the row calibration result parameters and column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode. Building upon row calibration, further column-based correction is applied to address vertical stripe noise caused by inconsistencies in column amplifier gain / offset. This cascaded row-column correction achieves synergistic suppression of bidirectional fixed-pattern noise. Particularly in multi-slope HDR mode, it adapts to the dynamic changes in column response curves under different exposure times, effectively suppressing FPN exacerbated by exposure time switching, and significantly improving the image quality uniformity and overall visual effect of high dynamic range images. This addresses the challenge in related technologies of effectively correcting fixed-pattern noise in multi-slope HDR modes under limited storage resource constraints.

[0158] It should be noted that the FPGA-based image sensor fixed pattern noise calibration system provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0159] Thirdly, embodiments of this application provide an electronic device, Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 5 As shown, the electronic device may include a processor 51 and a memory 52 storing computer program instructions.

[0160] Specifically, the processor 51 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0161] The memory 52 may include a large-capacity memory for data or instructions. For example, and not limitingly, the memory 52 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 52 may include removable or non-removable (or fixed) media. Where appropriate, the memory 52 may be internal or external to a data processing device. In a particular embodiment, the memory 52 is non-volatile memory. In a particular embodiment, the memory 52 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0162] The memory 52 can be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 51.

[0163] The processor 51 reads and executes the computer program instructions stored in the memory 52 to implement any of the FPGA-based image sensor fixed pattern noise calibration methods in the above embodiments.

[0164] In one embodiment, an FPGA-based image sensor fixed pattern noise calibration device may further include a communication interface 53 and a bus 50. Wherein, as Figure 5 As shown, the processor 51, memory 52, and communication interface 53 are connected through bus 50 and complete communication with each other.

[0165] The communication interface 53 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 53 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0166] Bus 50 includes hardware, software, or both, that couples together components of an FPGA-based image sensor fixed-pattern noise calibration device. Bus 50 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 50 may include Accelerated GraphicsPort (AGP) or other graphics buses, Extended Industry Standard Architecture (EISA) buses, Front Side Bus (FSB), HyperTransport (HT) interconnects, Industry Standard Architecture (ISA) buses, InfiniBand interconnects, Low Pin Count (LPC) buses, memory buses, Micro Channel Architecture (MCA) buses, Peripheral Component Interconnect (PCI) buses, PCI-Express (PCI-X) buses, Serial Advanced Technology Attachment (SATA) buses, Video Electronics Standards Association Local Bus (VLB) buses, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 50 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0167] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the FPGA-based image sensor fixed pattern noise calibration method provided in the first aspect.

[0168] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0169] In a possible implementation, the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform steps implementing the FPGA-based image sensor fixed pattern noise calibration method provided in the first aspect.

[0170] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A fixed-pattern noise calibration system for an image sensor based on FPGA, used to correct fixed-pattern noise in multi-slope HDR mode, characterized in that, The system includes: The parameter storage module is used to store the calibrated row FPN calibration parameters and column FPN calibration parameters, wherein the row FPN calibration parameters are stored according to the number of rows of the image sensor, and the column FPN calibration parameters are stored according to the number of columns of the image sensor. The row calibration module is used to receive image data from the image sensor, obtain the corresponding row FPN calibration parameters from the parameter storage module based on the row index of the image data, compare the original pixel value in the image data with the knee pixel value of the row FPN calibration parameters, if the original pixel value is less than the knee pixel value, output the original pixel value as the row calibration result parameter; if the original pixel value is greater than or equal to the knee pixel value, scale the difference between the original pixel value and the knee pixel value based on the scaling factor in the row FPN calibration parameters, and add the scaling result to the knee pixel value to obtain the row calibration result parameter. The column calibration module, connected to the row calibration module, is used to receive the row calibration result parameters and perform calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode.

2. The fixed pattern noise calibration system for an image sensor based on FPGA according to claim 1, characterized in that, When storing the calibrated row FPN calibration parameters and column FPN calibration parameters, the parameter storage module is also used for: The row FPN calibration parameters obtained by the image sensor through multiple HDR modes are acquired, and the row FPN calibration parameters are stored in the calibration parameter RAM in the parameter storage module. Obtain the column FPN calibration parameters obtained from the calibration based on the black point image and the white point image, and store the column FPN calibration parameters in the calibration parameter RAM; Multiple random images with uniform gray levels acquired by the image sensor constitute a random image test set. The mean square error value of each image in the random image test set is calculated, and an evaluation index for evaluating the uniformity of the image sensor is determined based on the mean square error value of each image. The evaluation index and the calibration parameters are stored in RAM in a non-volatile memory; wherein the evaluation index is used to verify whether the correction effect of the row FPN calibration parameters and the column FPN calibration parameters meets the standard.

3. The fixed pattern noise calibration system for an image sensor based on FPGA according to claim 2, characterized in that, When acquiring the row FPN calibration parameters obtained by the image sensor through multiple HDR mode calibrations, the parameter storage module is also used for: The image sensor is controlled to acquire at least four levels of uniform grayscale images in each HDR mode; For each row of pixels in the uniform grayscale image, the slope of the high-gain dark area, the intercept of the high-gain dark area, the slope of the low-gain bright area, and the intercept of the low-gain bright area are obtained by fitting. Based on the high-gain dark area slope, the high-gain dark area intercept, the low-gain bright area slope, and the low-gain bright area intercept, calculate the knee pixel value and scaling factor for each row of pixels; The knee pixel value and the scaling factor are used as the row FPN calibration parameters.

4. The FPGA-based image sensor fixed pattern noise calibration system according to claim 2, characterized in that, When the parameter storage module acquires the column FPN calibration parameters obtained from the black and white images, it is also used for: The image sensor is controlled to acquire black and white images; Based on the black field image and the white field image, the gain coefficient and offset are fitted to each column of pixels; The gain coefficient and the offset are used as the column FPN calibration parameters.

5. The fixed pattern noise calibration system for an image sensor based on FPGA according to claim 1, characterized in that, When the column calibration module receives the row calibration result parameters and performs calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode, it is also used to: Based on the column index of the image data, the corresponding column FPN calibration parameters are obtained from the parameter storage module; Receive the row calibration result parameters, and subtract the offset from the column FPN calibration parameters to obtain the offset correction value; The offset correction value is divided by the gain coefficient in the column FPN calibration parameters to obtain the calibrated image data.

6. The fixed pattern noise calibration system for an image sensor based on FPGA according to claim 1, characterized in that, The image data output by the image sensor, the line calibration result parameters output by the line calibration module, and the image data output by the line calibration module are all transmitted within the system via the AXI4-Stream signal.

7. The fixed pattern noise calibration system for an image sensor based on FPGA according to claim 1, characterized in that, The system also includes a RISC-V soft core mounted on the AXI bus; the RISC-V soft core is used to control the row FPN calibration parameters and column FPN calibration parameters in the non-volatile memory to be loaded into the calibration parameter RAM of the parameter storage module when the system is powered on.

8. The fixed-pattern noise calibration system for an image sensor based on FPGA according to claim 2, characterized in that, The system also includes an image caching module and an Ethernet communication module; The image caching module is connected to the column calibration module and is used to receive and cache the calibrated image data output by the column calibration module. The Ethernet communication module is connected to the image cache module and is used to package the image data cached in the image cache module and send it to an external host. The Ethernet communication module is also used to receive control commands and calibration parameter files from the external host, and to store the calibration parameters RAM in non-volatile memory or write them into the parameter storage module according to the control commands.

9. A fixed-pattern noise calibration method for an image sensor based on FPGA, characterized in that, The method is applied to the system according to any one of claims 1 to 8, and the method includes: The calibrated row FPN calibration parameters and column FPN calibration parameters are stored, wherein the row FPN calibration parameters are stored according to the number of rows of the image sensor, and the column FPN calibration parameters are stored according to the number of columns of the image sensor. Receive image data from the image sensor and perform calibration based on the row FPN calibration parameters to obtain row calibration result parameters; The system receives the row calibration result parameters and performs calibration based on the row calibration result parameters and the column FPN calibration parameters to obtain calibrated image data in multi-slope HDR mode.

Images