Real-time visual imaging device and method based on FPGA
By constructing a co-current clock system and using CDR clock recovery technology in the FPGA real-time imaging system, the problems of cross-clock domain processing complexity and frame buffer latency were solved, achieving stable, low-latency image data transmission and display.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JILIN UNIVERSITY
- Filing Date
- 2026-02-06
- Publication Date
- 2026-05-15
AI Technical Summary
In FPGA-based real-time imaging systems, the timing references of the image acquisition end, FPGA processing end, and display output end are different, which leads to increased complexity of cross-clock domain processing, unstable line/frame boundary correspondence, and external frame buffering causes latency and resource consumption.
By constructing a co-source clock system, multiple synchronous clocks are generated using a reference clock source to ensure that the CMOS image sensor, FPGA processing, and HDMI display output operate in the same clock domain. Line buffers and small-depth FIFOs are used for clock matching to reduce external DDR frame-level synchronization buffers, and CDR clock recovery technology is used to stabilize data transmission.
It significantly reduces the complexity of cross-clock domain processing, reduces frame-level caching requirements, lowers system latency and storage resource consumption, improves real-time performance and stability, and avoids frame duplication and frame errors.
Smart Images
Figure CN121665131B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of electronic information and machine vision technology, specifically to a real-time visual imaging device and method based on FPGA. Background Technology
[0002] With the development of applications such as machine vision, industrial inspection, and real-time display preview, the resolution and frame rate of image sensors are constantly improving. Raw image data is usually output to the processing end via a high-speed serial link, where the image signal is processed and then output to the display device. In a real-time imaging system based on FPGA (Field Programmable Gate Array), a common system link includes: image sensor outputting raw data, FPGA receiving and parsing the data and performing ISP processing, and then generating the display interface signal output.
[0003] However, in engineering implementation, the image acquisition end, FPGA processing end, and display output end often involve different timing references or cross-clock domain interactions. On the one hand, the high-speed serial receiver usually relies on a reference clock and needs to perform clock data recovery to meet the requirements of SerDes (serializer / deserializer) and phase-locked loop operation. On the other hand, the subsequent ISP (Image Signal Processor) processing and display output link require a stable, continuous, and consistent pixel stream rhythm. If the system timing is not properly organized, it may lead to increased complexity of cross-clock domain processing, unstable line / frame boundary correspondence, and latency and resource consumption caused by introducing an external frame buffer to achieve frame synchronization. Therefore, a real-time visual imaging device and its implementation method are needed that can achieve stable operation in the "high-speed acquisition-FPGA streaming processing-display output" link, reduce the complexity of cross-clock domain organization, and reduce dependence on external frame buffers. Summary of the Invention
[0004] This application provides an FPGA-based real-time visual imaging device and its implementation method, enabling image data to operate stably for a long time in the link of high-speed serial acquisition, FPGA streaming ISP processing, and HDMI display output. By constructing a common clock system for acquisition, processing, and output, the implementation complexity and potential risks caused by cross-clock domain processing are significantly reduced. In most display passthrough or lightweight processing scenarios, row buffer / small-depth FIFO (first-in, first-out) is used to achieve clock matching, reducing or avoiding the use of external DDR (double data rate) for frame-level synchronization buffering, thereby reducing end-to-end display latency and reducing storage resource consumption.
[0005] A real-time visual imaging device based on FPGA according to an embodiment of the first aspect of this application includes:
[0006] A CMOS image sensor is used to output raw image data to the FPGA main controller via a high-speed serial link.
[0007] The FPGA main controller is used to perform receiving, parsing and pixel reassembly on the original image data to form an internal pixel stream and send it to the ISP image processing pipeline for streaming processing to obtain a display pixel stream.
[0008] The display output interface circuit is used to generate an HDMI display signal based on the display pixel stream and output it to the display device.
[0009] The clock synchronization circuit includes a reference clock source and a multi-clock fan-out unit. The multi-clock fan-out unit includes at least the following outputs: an input clock for driving the CMOS image sensor, a main clock for driving the image acquisition and ISP image processing pipeline inside the FPGA main controller, and a reference clock for the high-speed serial transmitter on the HDMI output side, so that acquisition, processing and output are under the same clock system.
[0010] Furthermore, the FPGA main controller includes a high-speed serial receiver. The SerDes reference clock of the high-speed serial receiver and the input clock of the CMOS image sensor belong to different clock sources. The high-speed serial receiver tracks the data edge information in the serial data stream, locks the clock phase of the high-speed serial transmitter in real time, and recovers a recovery clock synchronized with the high-speed serial transmitter from the serial data stream without an accompanying clock.
[0011] Furthermore, the raw image data reception of the FPGA main controller includes: using a high-speed serial receiver to enable CDR clock recovery of the high-speed serial code stream to obtain a recovery clock, and completing differential serial data reception and boundary positioning through predefined control codes under CDR clock recovery to obtain a parallel domain 10-bit code group data stream, and performing 8B / 10B decoding on the parallel domain 10-bit code group to obtain a parallel domain byte stream.
[0012] Furthermore, the pixel reassembly includes: unpacking the parallel domain 10-bit code group data stream through a high-speed serial receiver to extract the payload, and reassembling the byte stream into a Bayer RAW12 pixel stream according to the RAW12 data packing rules.
[0013] Furthermore, the FPGA main controller encapsulates the Bayer RAW12 pixel stream into an internal pixel stream interface, which includes at least a pixel data port and data valid, line valid and frame valid flag signals, so that each stage of the ISP image processing pipeline is aligned and propagated in a pipeline with a fixed delay.
[0014] Furthermore, the FPGA main controller includes a sensor register configuration interface, which includes an SPI configuration interface. The FPGA main controller generates exposure time and gain update parameters based on image statistics and writes them into the registers of the CMOS image sensor through the SPI configuration interface to achieve dynamic exposure and dynamic gain control.
[0015] Furthermore, the device also includes a display information interaction interface, which reads the display device EDID information through the DDC channel and configures HDMI output timing parameters based on the EDID information.
[0016] Furthermore, the FPGA main controller maps the output of the ISP module to RGB888 format and aligns the display synchronization flag with the same delay to ensure that the output pixels and the line / frame boundary flags are consistent in timing.
[0017] According to the second aspect of the present application, a real-time visual imaging method based on FPGA is provided, comprising: S1 generating a synchronous clock from a reference clock source via a one-to-multi-clock fan-out unit for input to a CMOS image sensor, processing of the FPGA internal acquisition and image processing pipeline, and output of a high-speed serial transmitter of an HDMI driver unit.
[0018] S2 initializes and configures the CMOS image sensor through the sensor register configuration interface;
[0019] S3 receives raw image data from the CMOS image sensor via a high-speed serial link, obtains the recovered clock through CDR clock recovery, and completes differential data reception, boundary alignment, and data decoding to obtain a parallel domain byte stream.
[0020] S4 unpacks the parallel domain byte stream to extract the payload, and completes the pixel reassembly of the RAW12 data to form an internal pixel stream interface and sends it to the ISP image processing pipeline for streaming processing.
[0021] The S5 generates an HDMI display signal from the display pixel stream obtained after streaming processing and outputs it to the display device.
[0022] Furthermore, before or during the process of generating an HDMI display signal from the processed display pixel stream and outputting it to the display device, the EDID information of the display device is read, and the output timing parameters of the HDMI display signal are configured based on the EDID information.
[0023] Furthermore, the ISP image processing pipeline performs image statistics and generates exposure time and gain update parameters, which are then written to the registers of the CMOS image sensor via the SPI configuration interface to dynamically update the imaging parameters.
[0024] Compared with the prior art, this application has at least the following beneficial effects:
[0025] 1) Significantly reduce cross-clock domain processing complexity: Acquisition, ISP processing and output are all under the same clock system, which can significantly reduce cross-clock domain complexity caused by frequency difference of independent oscillators; at the same time, the internal reference clock difference of the receiver can be absorbed by CDR and elastic buffer, thereby reducing the number and complexity of asynchronous FIFO, cross-domain synchronous trigger chain and other circuits, reducing the risk of implicit line / frame loss caused by metastability and frequency difference accumulation, and making system timing convergence easier.
[0026] 2) Replace "frame-level DDR synchronization buffer" with "line buffer / small-depth FIFO" to reduce system cost and bandwidth pressure: Same source clock means that the acquisition clock and the output reference clock have the same frequency reference and there will be no long-term cumulative drift; therefore, in most display pass-through or lightweight processing scenarios, line-level buffer, small-depth FIFO plus handshake flow control can be used to complete the clock matching, thereby achieving stable correspondence between frame / line boundaries, without the need for external DDR to perform full frame buffering to complete "frame synchronization", and reducing memory bandwidth and cost.
[0027] 3) Significantly reduced and more deterministic end-to-end latency: Since there is no waiting time introduced by frame-level DDR memory caching, the system end-to-end latency is mainly determined by "limited-depth FIFO + line buffer + fixed latency of ISP image processing pipeline". In engineering implementation, the buffer depth can be controlled at a small number of lines (for example, N-line buffer introduces a buffer latency of about N lines), thereby improving the latency from acquisition to display and improving real-time determinism.
[0028] 4) Avoid display anomalies that may be introduced by frame-level caching under asynchronous read and write conditions: When using external DDR memory for frame-level synchronization, it is usually necessary to deal with cross-domain synchronization between the write clock domain and the read clock domain and the consistency of read and write pointers; This application reduces such cross-domain synchronization requirements through the same source clock and row-level buffer strategy, and reduces / suppresses the risk of frame duplication, frame errors or frame boundary jitter caused by inconsistent read and write clocks at the system structure level.
[0029] 5) By aligning the internal pixel stream interface with the fixed delay of the ISP image processing pipeline, pixel data and line / frame boundary markers are propagated synchronously, facilitating stable display output. Attached Figure Description
[0030] Figure 1 This is a hardware circuit block diagram of an embodiment of this application; Figure 2 This is a schematic diagram of the various modules contained in the internal ISP image processing pipeline of the FPGA main controller according to an embodiment of this application; Figure 3This is a flowchart illustrating the operation of a high-speed serial receiver for an FPGA master controller provided in one embodiment of this application. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0032] See Figure 1 As shown, in this embodiment, a real-time visual imaging device based on FPGA is provided, including: a CMOS image sensor, an FPGA main controller, a clock synchronization circuit, a control and configuration interface circuit, a display output interface circuit, and a configuration and debugging interface circuit.
[0033] The CMOS image sensor is used to sense the target scene image from the optical imaging lens, complete photoelectric conversion and analog-to-digital conversion, and output raw image data. The optical imaging lens is used to image the target scene onto the photosensitive surface of the CMOS image sensor. The FPGA main controller is used to receive the raw image data and execute the ISP image processing pipeline inside the FPGA main controller, and output display data after processing. The display output interface circuit is used to form the display data into an HDMI display signal and output it to the display device. The display output interface circuit for forming the display data into an HDMI display signal and outputting it to the display device includes an HDMI driver unit and a high-speed serial transmitter.
[0034] In this embodiment, the CMOS image sensor outputs raw image data to the FPGA main controller through a high-speed serial receiver, and acquires the raw image data under the drive of the CMOS driving unit.
[0035] In this embodiment, to ensure that the CMOS image sensor data output, image acquisition, FPGA main controller internal processing, and HDMI driver unit output operate under a unified timing reference, a common-source clock synchronization scheme is adopted. This scheme is implemented through a clock synchronization circuit, which includes a reference crystal oscillator and a multi-clock fan-out unit. The reference crystal oscillator outputs a reference clock signal, which is then used by the multi-clock fan-out unit to generate at least three synchronized clocks, namely:
[0036] 1) The input clock of the CMOS image sensor is used to drive the operating clock input of the CMOS image sensor;
[0037] 2) FPGA master clock, used to output at least three clock signals through the timing control unit to drive the CMOS driver unit inside the FPGA master controller to perform CMOS image acquisition logic, ISP image processing pipeline logic, HDMI driver unit encoding logic and control logic;
[0038] 3) FPGA high-speed transceiver reference clock, used as the reference clock for the high-speed serial transmitter on the output side of the HDMI driver unit.
[0039] Through the aforementioned shared clock allocation, the input clock of the CMOS image sensor, the FPGA master clock, and the reference clock on the HDMI output side are generated from the same reference clock, ensuring that acquisition, processing, and output are all within the same clock system. Since there is no long-term cumulative phase shift caused by independent clock drift / frequency difference, the device can use a pixel-level pipeline with finite depth for simultaneous input and output to achieve data clock synchronization, without relying on external DDR memory for frame buffering. This reduces system latency and storage resource consumption while ensuring real-time performance.
[0040] It should be noted that the "same-source clock system" in this article mainly refers to the fact that the clocks used to define the image data generation cycle and the system processing / display cycle (including the input clock of the CMOS image sensor, the FPGA master clock, and the reference clock on the output side of the HDMI driver unit) are derived from the same reference clock. This ensures that the frequency references for acquisition, processing, and output are consistent, thereby avoiding long-term cumulative drift caused by the frequency difference of independent oscillation sources. Simultaneously, to meet the operational requirements of the SerDes (serializer) analog front-end and Clock Data Recovery (CDR), the high-speed serial receiver can use the SerDes reference clock as the working reference for phase-locked loop / frequency multiplication; this SerDes reference clock does not directly define the CMOS image sensor transmitter cycle. After CDR locking, the high-speed serial receiver can recover a recovery clock synchronized with the transmitter from the serial data stream to ensure the stability of deserialization and subsequent decoding processes.
[0041] In this embodiment, the control and configuration interface circuit is used to complete the initialization of the CMOS image sensor and the dynamic parameter update during operation, as well as the acquisition of display capabilities and output timing configuration of the HDMI driver unit display output side. The control and configuration interface circuit includes a sensor register configuration interface, a sensor mode selection interface, a display information interaction interface, and a configuration and debugging interface.
[0042] The sensor register configuration interface refers to the FPGA main controller connecting to the CMOS image sensor via the SPI configuration interface (Standard Serial Configuration Interface (SPI)). This interface is used for initializing and writing to the CMOS image sensor's registers and updating dynamic parameters during operation. The sensor register configuration content includes at least the following:
[0043] Imaging parameters include: exposure time, analog gain, digital gain, black level / bias, etc.
[0044] Output and window parameters include: image window size, start / end position of valid data, and output resolution settings;
[0045] The operating mode parameters include: PLL frequency multiplication and division parameters inside the CMOS image sensor, frame rate related parameters, and link configuration parameters (line rate / link control code / synchronization code) related to high-speed serial output.
[0046] In one implementation, the FPGA main controller completes power-on initialization by configuring a state machine according to a preset register sequence. During operation, the FPGA main controller can update parameters such as exposure time and gain online based on image statistics, thereby dynamically adjusting the imaging parameters of the CMOS image sensor without interrupting the video stream.
[0047] Sensor mode selection interface (GPIO):
[0048] In addition to the sensor register configuration interface configuration, some operating modes of the CMOS image sensor are selected by the FPGA main controller through the sensor mode selection interface pins and controlled by the sensor mode selection interface. The content of the sensor mode selection interface control includes at least the following:
[0049] Output interface mode selection: Used to select whether the CMOS image sensor output is SLVS (Serial Links Verification Standard) or SLVS-EC (Serial Links Verification Standard - Electrical Compliance).
[0050] Master / Slave Mode Selection: Used to select whether the CMOS image sensor operates in master or slave mode;
[0051] Reset / enable control signals: used to achieve power-on reset, abnormal recovery, and switching of working states.
[0052] By combining SPI register configuration with GPIO mode selection, the power-on startup process, initialization configuration, and dynamic parameter updates of the CMOS image sensor can be realized.
[0053] Display information interaction interface:
[0054] In one implementation, the FPGA main controller interacts with the display device via the DDC channel (I²C) of the HDMI interface to read the EDID information (Extended Display Identification Data, a standard published by the Video Electronics Standards Association (VESA) for displays to describe their capabilities to video sources), obtaining the resolution, frame rate, and key timing parameters supported by the display device. The FPGA main controller configures the output timing parameters of the HDMI display signal output by the HDMI driver unit based on the EDID information to match the display device's capabilities. In implementations requiring self-adaptation, the FPGA main controller can also adjust the CMOS image sensor output resolution and the ISP image processing pipeline's cropping / scaling parameters based on the EDID information, achieving coordinated configuration of acquisition specifications, processing specifications, and display specifications. When the output mode selected based on the EDID information is inconsistent with the CMOS image sensor output resolution, the system can adapt by cropping / expanding the image sensor window.
[0055] Configuration and debugging interface:
[0056] The device is configured with a debugging interface circuit, including a JTAG interface, for downloading FPGA main controller logic configuration files, online debugging, and development testing. In mass production implementation, it can also be used with external non-volatile memory to enable FPGA main controller configuration hardening and power-on self-starting. The configuration and debugging interface can be connected to external configuration and debugging equipment, as well as the human-machine interaction module.
[0057] In this embodiment, the raw image data output by the CMOS image sensor is in Bayer array format (preferably RGGB arrangement) with a pixel bit depth of RAW12 (image data stored in 12-bit RAW format). The raw image data is output to the FPGA main controller via at least one differential transmission through the SLVS-EC high-speed serial link. The FPGA main controller calls the on-chip high-speed serial receiver and high-speed serial transmitter to construct the SLVS-EC (a high-speed image sensor interface standard developed by Sony Semiconductor Solutions, mainly used for high-speed data transmission between high-resolution image sensors and processing units) receiving link, which is used to complete physical layer reception, link layer unpacking, and pixel reassembly to provide a continuous pixel stream to the ISP image processing pipeline.
[0058] In this embodiment, the CMOS image sensor outputs image data to the FPGA main controller via the SLVS-EC high-speed serial link through the high-speed serial receiver.
[0059] like Figure 3As shown, the FPGA main controller is used for at least: clock data recovery (CDR), differential serial data reception, byte alignment, 8B / 10B decoding, multi-lane de-skew / alignment, unpacking, pixel reconstruction, and feeding into the ISP image processing pipeline for streaming processing.
[0060] In one implementation, the serial output line rate on the CMOS image sensor side is generated by its input reference clock through an internal frequency multiplier. For example, the CMOS image sensor side line rate is generated by internal frequency multiplication of the input reference clock; the SerDes reference clock on the FPGA main controller side is internally multiplied to form the operating rate of the matching high-speed serial receiver. It should be understood that this is not strictly a homogeneous source, meaning that the input reference clock on the CMOS image sensor side and the SerDes reference clock provided by the high-speed serial receiver of the FPGA main controller (used for its internal frequency multiplication / phase-locked loop) may not be the same clock source. There may be a PPM-level frequency error between the two, and phase drift may occur with factors such as temperature and power supply. However, in this embodiment, the high-speed serial receiver enables the CDR clock recovery mechanism. After CDR locking, a recovered clock synchronized with the transmitter can be recovered from the serial data stream, thereby ensuring long-term stable sampling and reliable deserialization.
[0061] In this embodiment, the high-speed serial receiver of the FPGA main controller enables the CDR clock recovery mechanism. The CDR is a data-driven adaptive phase-locked loop structure. By tracking the data edge information in the serial data stream, it locks the clock phase of the high-speed serial transmitter in real time and recovers a recovered clock synchronized with the high-speed serial transmitter from the serial data stream without an accompanying clock. This dynamically determines the optimal data sampling point and ensures stable data reception even under conditions of PPM-level frequency difference and phase drift.
[0062] After CDR locking, the high-speed serial receiver further utilizes predefined control codes (Comma) in the bitstream to achieve byte boundary positioning and byte alignment: when a control code offset is detected, the high-speed serial receiver relocks to the correct byte boundary by adjusting the decoded word boundary (bit-slip or equivalent mechanism), thereby ensuring that subsequent data is output to the parallel domain in a stable byte alignment manner. Afterwards, the high-speed serial receiver performs 8B / 10B decoding on the serial bitstream, restoring the 10-bit code group to 8-bit symbols and distinguishing between data symbols and control symbols.
[0063] In addition, control codes / training sequences for link maintenance are periodically inserted into the transmitted data stream to verify and correct the alignment status. The high-speed serial receiver continuously detects the position and arrival timing of the control codes; when the boundary or position offset of the control codes is found to exceed a preset threshold, a realignment process is triggered (re-aligning bytes, re-de-skewing the lane (transmission line), and triggering link retraining / reset recovery if necessary) to maintain the reliability of the link during long-term operation.
[0064] When using multi-transmission-line output, in addition to byte alignment on a single transmission line, the high-speed serial receiver also performs inter-transmission-line alignment / de-skew: by identifying alignment markers or synchronization sequences across transmission lines and using a de-skew FIFO to compensate for the delay of the parallel output of each transmission line, the payload of the same packet / line is aligned to the same output time across transmission lines; in a single-transmission-line scenario, this de-skew processing can be turned off to save resources.
[0065] To further absorb transient differences between the recovery clock and the subsequent local processing clock, the high-speed parallel receiver can be configured with a receive FIFO buffer for rate differential compensation and burst congestion absorption, avoiding overflow caused by short-term jitter or subsequent flow control. This provides continuous and stable data input for subsequent unpacking, pixel reconstruction, and ISP image processing pipelines. The elastic buffer depth can be appropriately reduced or some deskipation / compensation functions can be disabled to save resources, depending on system throughput and timing constraints.
[0066] In one implementation, the SLVS-EC (a high-speed serial interface for receiving image data) interface of the CMOS image sensor employs multi-transmission-line parallel output. Since propagation delay differences may exist between the transmission lines within the device, in the package interconnects, and in the board-level traces, the high-speed serial receiver, after completing CDR locking (carrier lock) and byte alignment for a single transmission line, further performs multi-transmission-line deskewing and transmission-line alignment processing. Specifically, the high-speed serial receiver establishes the relative time delay relationship of each transmission line by detecting predefined cross-transmission-line alignment markers, and performs delay compensation for the parallel output of each transmission line through a deskew FIFO, ensuring that valid data of the same packet / line is aligned to a unified output time across all transmission lines. This guarantees that the subsequent unpacking and pixel reassembly logic obtains consistent byte boundaries and relative data order.
[0067] In addition, the high-speed serial receiver can monitor state variables such as transmission line alignment status, elastic buffer status (overflow, high bit, medium bit, low bit, read empty), and control code detection results; when continuous abnormalities occur or alignment cannot be maintained, the link recovery process (re-deskipation, retraining, or reset recovery) is triggered to improve the stability of long-term operation.
[0068] In one embodiment, valid image data from the SLVS-EC interface is transmitted in the form of "packets". The high-speed serial receiver performs packet unpacking operations: identifying the packet header from the serial data stream, extracting the payload, and, in optional configuration, parsing the packet footer. The packet format includes at least "packet header + payload"; optionally, it also includes "packet header + payload + packet footer (CRC)," or "packet header + payload containing ECC (encoded for error detection and correction)." The CRC option is used for error detection; the ECC option is used for payload error correction; the two are mutually exclusive and cannot be enabled simultaneously.
[0069] In one embodiment, after obtaining the payload byte stream, the high-speed serial receiver reassembles the pixels according to the RAW12 data packing rules, converting the byte stream into a Bayer RAW12 pixel stream. Preferably, a mapping method of "3 bytes to 2 pixels" is used, where the first two bytes are used as the high 8 bits of the pixel, and the third byte is split into 4 bits + 4 bits, which are used as the low 4 bits of the pixel. To meet the link transmission alignment requirements, the payload length must be aligned to 16 bytes. When the number of bytes after pixel conversion in a row is insufficient to meet the alignment conditions, payload padding is added to the end of the row. In the RAW12 data example, the padding can be up to 14 bytes to achieve length alignment.
[0070] In one embodiment, after unpacking and pixel reassembly are completed, the FPGA main controller encapsulates the Bayer RAW12 pixel stream into an internal pixel stream interface and sends it to the ISP image processing pipeline. The internal pixel stream interface includes at least the following ports / signals:
[0071] Raw image data port: Represented as raw_data[11:0], used to carry RAW12 data;
[0072] Data valid signal: Represented as data_valid, indicating that the raw image data (raw_data) is valid in the current clock cycle;
[0073] Line / frame boundary signals: Represented as line_valid (line boundary signal) and frame_valid (frame boundary signal), used to identify the valid intervals of lines and frames and to align and propagate them in each stage of the pipeline;
[0074] The aforementioned pixel stream interface transmits RAW12 data within the FPGA main controller in a pixel-by-pixel streaming manner, ensuring that line / frame boundary signals propagate synchronously with pixel data. This enables subsequent ISP image processing pipelines, such as bad pixel correction, white balance, histogram analysis (used for linkage with automatic exposure / gain), noise reduction, color correction, and de-mosaicing, to operate stably on a pipeline with a fixed delay.
[0075] In one embodiment, the input clock of the CMOS image sensor, the FPGA master clock, and the reference clock of the high-speed serial transmitter on the HDMI output side are obtained from the same reference clock via fan-out conversion, which brings the following verifiable engineering advantages in terms of system timing organization:
[0076] 1) Reduce or avoid asynchronous cross-clock domain switching: When the acquisition, ISP image processing pipeline's processing and output are under the same clock system, the cross-clock domain complexity caused by the frequency difference of independent oscillators in the data path can be significantly reduced; the internal reference clock difference of the high-speed serial receiver can be absorbed by CDR and elastic buffer, making it easier for subsequent processing and output to maintain a stable frame / line correspondence. This can reduce the number and complexity of asynchronous FIFOs, cross-domain synchronous trigger chains, etc., reduce the risk of implicit line / frame loss caused by metastability and frequency difference accumulation, and make system timing convergence easier.
[0077] 2) Replace "Frame-Level DDR Synchronization Buffer" with "Line Buffer / Small-Depth FIFO": A shared clock source means that the acquisition clock and the output reference clock have a consistent frequency reference, preventing long-term cumulative drift caused by the frequency difference between two independent oscillation sources. Therefore, in most display passthrough or lightweight processing scenarios, the system can use a line buffer + small-depth FIFO + handshake flow control approach. This achieves stable frame / line boundary correspondence without requiring external DDR for full-frame buffering to complete so-called "frame synchronization." This not only reduces memory bandwidth and cost but also reduces the latency introduced by the frame buffer, improving the latency from acquisition to the display.
[0078] like Figure 2As shown, in this embodiment, the FPGA main controller internally implements a hardware pipelined ISP image processing pipeline. This ISP image processing pipeline processes the input data in a pixel-by-pixel stream manner and outputs target pixel format data for display. The ISP image processing pipeline preferably includes the following functional modules: a bad pixel correction module, a white balance module, a histogram analysis module, an automatic exposure / automatic gain control module, a noise reduction module, a color correction module, and a de-mosaic module. To meet the real-time and FPGA main controller implementation requirements, the above modules are preferably implemented using fixed-point arithmetic, line buffering and sliding window structures, and a pipelined parallel structure. Furthermore, the line / frame flag signal is delayed and aligned at each stage of the pipeline before being output synchronously with the output pixels. Each ISP image processing pipeline module uses line buffers / shift registers to construct neighborhood windows and performs fixed-point calculations; the delay modules in each ISP image processing pipeline synchronize data_valid / line_valid / frame_valid with pixels through a unified delay alignment unit; the module parameters in each ISP image processing pipeline are configured through registers and support bypassing for different cropping modes. Specific details are as follows:
[0079] Dead pixel correction module:
[0080] The defective pixel correction module is used to detect and repair defective pixels in the input RAW12 data. Preferably, this module identifies defective pixels based on neighborhood anomaly detection criteria and generates repaired pixel values using a neighborhood statistical value replacement method, thereby outputting a data stream with repaired defective pixels.
[0081] White balance module:
[0082] The white balance module is used to perform channel gain compensation on the Bayer RAW12 pixel stream. Configurable gain coefficients can be applied to the R, G, and B channels. These gain coefficients can be calculated from image statistics and / or written by external configuration to achieve color temperature compensation under different lighting conditions.
[0083] Histogram analysis module and automatic exposure / gain control module:
[0084] The histogram analysis module is used to statistically analyze the brightness distribution of one or more frames of image data and generate statistical results. The automatic exposure / automatic gain control module generates exposure time and gain update parameters based on the statistical results and writes them into the register of the CMOS image sensor through the SPI configuration interface to update the imaging parameters, thereby realizing dynamic exposure and dynamic gain control in the running state.
[0085] Noise reduction module:
[0086] The noise reduction module is used to suppress noise in the input pixel stream, and it can be implemented using bilateral filtering. The noise reduction module performs weighted fusion of neighboring pixels based on pixel spatial distance weights and pixel value similarity weights, outputting the denoised pixel values. On the FPGA main controller, a neighboring window is preferably constructed using a window generation unit (row buffer + shift register). Weight calculation is performed through table lookup or fixed-point approximation, and weighted summation and normalization are achieved through pipelined accumulation.
[0087] Color correction module:
[0088] The color correction module is used to perform color correction processing on color pixel data. It can use matrix transformation or segmented mapping to adjust the color response to a predetermined color space or correction standard and output the corrected color pixel data.
[0089] De-mosaic module:
[0090] The demosaic module is used to interpolate and reconstruct the corrected color pixel data to obtain three-channel color pixel data. The demosaic module preferably uses a 5×5 neighborhood window to enhance edge texture, specifically including:
[0091] At least 5 rows of buffers are constructed from the input RAW12 pixel stream by the row buffer unit, and a 5×5 sliding window is formed by combining the shift register;
[0092] Interpolation calculations are performed on the missing color components based on the 5×5 sliding window; in a preferred implementation, edge direction determination or texture intensity determination can be introduced, and interpolation direction or interpolation weight can be selected in the edge region to improve the reconstruction effect of edge and texture details.
[0093] The reconstructed RGB three-channel data is output to the subsequent color correction module or output format conversion module.
[0094] To accommodate the resource and throughput requirements of the FPGA main controller, the interpolation and decision operations are preferably implemented using fixed-point computation and a pipelined structure, and the line / frame flag signal is output synchronously with the output pixel after being delayed and aligned at each stage of the pipeline.
[0095] In one embodiment, the HDMI output adopts an RGB888 pixel format. The pixel bit width and format adaptation module converts the internal high-bit width R / G / B data into 8-bit output by extracting the high bits, and merges them into an rgb_data[23:0] array. At the same time, the control signals such as data_valid, line_valid, and frame_valid are aligned with the same delay to ensure that the output pixels and line / frame boundary markers are strictly consistent in timing, avoiding misalignment between HDMI timing generation and serial output.
[0096] In this embodiment, the ISP image processing pipeline sends the RGB888 pixel format data rgb_data[23:0] and control signals to the HDMI display output module. The FPGA main controller implements the high-speed serial transmission end on the output side of the HDMI driver unit based on its high-speed transceiver resources, and generates the logic output HDMI display signal in combination with the display timing. The reference clock of the high-speed serial output on the output side of the HDMI driver unit is taken from the third synchronous clock of the same source clock synchronization circuit, so that the reference clock of the output link and the acquisition / processing clock maintain the same source reference relationship.
[0097] In this embodiment, after the system is powered on, the FPGA main controller completes the configuration startup and initializes the display output timing (in the EDID-enabled implementation, the EDID information is read first and the output mode is configured accordingly); then, the CMOS image sensor is initialized and configured through the image processing pipeline and the output is started; the raw image data enters the FPGA main controller through the SLVS-EC link, and after being received and aligned, it forms the internal pixel stream of the RAW12 database and enters the ISP image processing pipeline; the ISP image processing pipeline sequentially performs bad pixel correction, white balance, noise reduction, de-mosaic, color correction and other processing and outputs the RGB888 pixel stream to the HDMI driver unit to complete the display output; during operation, the histogram analysis module continuously generates statistical results, and the AE / AGC (automatic exposure / automatic gain control) control module updates the exposure and gain parameters online according to the statistical results, so that the system maintains stable imaging under different lighting conditions.
[0098] On the other hand, this application provides a real-time visual imaging method based on FPGA, employing the real-time visual imaging device based on FPGA provided in the above embodiments, including:
[0099] S1 is generated by a reference clock source via a multi-clock fan-out unit for the synchronization clock used in the CMOS image sensor input, the FPGA internal acquisition and image processing pipeline, and the output high-speed serial transmitter of the HDMI driver unit.
[0100] S2 initializes and configures the CMOS image sensor through the sensor register configuration interface;
[0101] S3 receives raw image data from the CMOS image sensor via a high-speed serial link, obtains the recovered clock through CDR clock recovery, and completes differential data reception, boundary alignment, and data decoding to obtain a parallel domain byte stream.
[0102] S4 unpacks the parallel domain byte stream to extract the payload, and performs pixel reassembly of RAW12 format data to form an internal pixel stream interface and sends it to the ISP image processing pipeline for streaming processing.
[0103] S5 generates an HDMI display signal from the display pixel stream obtained after streaming processing and outputs it to the display device;
[0104] Before or during the process of generating an HDMI display signal from the processed display pixel stream and outputting it to the display device, the EDID information of the display device is read, and the output timing parameters of the HDMI display signal are configured based on the EDID information.
[0105] The ISP image processing pipeline performs image statistics and generates exposure time and gain update parameters, which are then written to the registers of the CMOS image sensor via the SPI configuration interface to dynamically update the imaging parameters.
[0106] This enables image data to operate stably for a long time in the link of high-speed serial acquisition, FPGA streaming ISP processing and HDMI display output; by constructing a common clock system for acquisition, processing and output, the implementation complexity and potential risks caused by cross-clock domain processing are significantly reduced.
[0107] At least three synchronous clocks are generated from a reference clock source via a multi-fan-out unit. These are used for the image sensor's input clock, the FPGA's internal image acquisition and ISP image processing pipeline master clock, and the high-speed serial transmitter reference clock on the HDMI output side. This ensures that acquisition, processing, and output operate within a common clock system. The common clock system is defined as clocks derived from the same source to define the image data generation clock and the system's processing and display clock. The high-speed serial receiver's SerDes reference clock serves only as a reference for CDR (Clock Data Recovery) phase-locked loop operation and does not affect the common clock derivation for acquisition, processing, and output. After CDR locking, a restored clock synchronized with the transmitter is restored to ensure stable sampling and reliable deserialization. After CDR locking, byte boundary positioning and alignment are achieved through predefined control codes, and 8B / 10B decoding is performed on the bitstream to restore the 10-bit code group to an 8-bit byte stream, providing stable input for subsequent unpacking and pixel reconstruction.
[0108] In multi-transmission-line scenarios, de-skew FIFO is used to compensate for delay differences between transmission lines in order to maintain the reliability of the link during long-term operation.
[0109] The byte stream is unpacked to extract the payload and reassembled into pixels according to the RAW12 pixel packing rules to form an internal pixel stream interface, which includes at least pixel data, data validity, line validity, and frame validity signals. This interface is then input into the ISP image processing pipeline for pixel-by-pixel streaming processing.
[0110] The ISP image processing pipeline converts the internal high-bit-width R / G / B data into RGB888 format output via the pixel bit width and format adaptation module, and outputs the pixel data and line / frame boundary markers with the same delay; it can also read the display device EDID information through DDC to configure HDMI output timing parameters.
[0111] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A real-time visual imaging device based on FPGA, characterized in that, include: A CMOS image sensor is used to output raw image data to the FPGA main controller via a high-speed serial link. The FPGA main controller is used to perform receiving, parsing and pixel reassembly on the original image data to form an internal pixel stream and send it to the ISP image processing pipeline for streaming processing to obtain the display pixel stream. The display output interface circuit is used to generate an HDMI display signal based on the display pixel stream and output it to the display device. The clock synchronization circuit includes a reference clock source and a multi-clock fan-out unit. The multi-clock fan-out unit includes at least the following outputs: an input clock for driving the CMOS image sensor, a main clock for driving the image acquisition and ISP image processing pipeline inside the FPGA main controller, and a reference clock for the high-speed serial transmitter on the HDMI output side, so that acquisition, processing and output are under the same clock system.
2. The FPGA-based real-time visual imaging device according to claim 1, characterized in that, The FPGA main controller includes a high-speed serial receiver. The SerDes reference clock of the high-speed serial receiver and the input clock of the CMOS image sensor belong to different clock sources. The high-speed serial receiver tracks the data edge information in the serial data stream, locks the clock phase of the high-speed serial transmitter in real time, and recovers a recovery clock synchronized with the high-speed serial transmitter from the serial data stream without an accompanying clock.
3. The FPGA-based real-time visual imaging device according to claim 2, characterized in that, The raw image data reception of the FPGA main controller includes: using a high-speed serial receiver to enable CDR clock recovery of the high-speed serial code stream to obtain a recovery clock, and completing differential serial data reception and boundary positioning through predefined control codes under CDR clock recovery to obtain a parallel domain 10-bit code group data stream, and performing 8B / 10B decoding on the parallel domain 10-bit code group to obtain a parallel domain byte stream.
4. The FPGA-based real-time visual imaging device according to claim 3, characterized in that, The pixel reassembly includes: unpacking the parallel domain 10-bit code group data stream through a high-speed serial receiver to extract the payload, and reassembling the byte stream into a Bayer RAW12 pixel stream according to the RAW12 data packing rules.
5. The FPGA-based real-time visual imaging device according to claim 4, characterized in that, The FPGA main controller encapsulates the Bayer RAW12 pixel stream into an internal pixel stream interface. The internal pixel stream interface includes at least a pixel data port and data valid, line valid, and frame valid flag signals to ensure that each stage of the ISP image processing pipeline propagates in alignment within a pipeline with a fixed delay.
6. The FPGA-based real-time visual imaging device according to claim 1, characterized in that, The FPGA main controller includes a sensor register configuration interface, which includes an SPI configuration interface. The FPGA main controller generates exposure time and gain update parameters based on image statistics and writes them into the registers of the CMOS image sensor through the SPI configuration interface to achieve dynamic exposure and dynamic gain control.
7. The FPGA-based real-time visual imaging device according to claim 1, characterized in that, The device also includes a display information interaction interface, which reads the display device EDID information through the DDC channel and configures HDMI output timing parameters based on the EDID information.
8. The FPGA-based real-time visual imaging device according to claim 1, wherein the FPGA main controller maps the output of the ISP module to RGB888 format and performs same-delay alignment on the display synchronization flag to ensure that the output pixels and the line / frame boundary flags are consistent in timing.
9. A real-time visual imaging method based on FPGA, applied to the apparatus of any one of claims 1 to 8, characterized in that, include: S1 is generated by a reference clock source via a multi-clock fan-out unit for the synchronization clock used in the CMOS image sensor input, the FPGA internal acquisition and image processing pipeline, and the output high-speed serial transmitter of the HDMI driver unit. S2 initializes and configures the CMOS image sensor through the sensor register configuration interface; S3 receives raw image data from the CMOS image sensor via a high-speed serial link, obtains the recovered clock through CDR clock recovery, and completes differential data reception, boundary alignment, and data decoding to obtain a parallel domain byte stream. S4 unpacks the parallel domain byte stream to extract the payload, and completes the pixel reassembly of the RAW12 data to form an internal pixel stream interface and sends it to the ISP image processing pipeline for streaming processing. The S5 generates an HDMI display signal from the display pixel stream obtained after streaming processing and outputs it to the display device.
10. The FPGA-based real-time visual imaging method according to claim 9, characterized in that, Before or during the process of generating an HDMI display signal from the processed display pixel stream and outputting it to the display device, the EDID information of the display device is read, and the output timing parameters of the HDMI display signal are configured based on the EDID information. The ISP image processing pipeline performs image statistics and generates exposure time and gain update parameters, which are then written to the registers of the CMOS image sensor via the SPI configuration interface to dynamically update the imaging parameters.