A low-power consumption panoramic video splicing FPGA processor
By using a low-power FPGA processor to achieve rapid fusion and encoding of multiple video streams, the problems of high power consumption and slow speed in existing technologies are solved, thus meeting the real-time display requirements of in-vehicle panoramic systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Utility models(China)
- Current Assignee / Owner
- NORTHERN INST OF AUTOMATIC CONTROL TECH
- Filing Date
- 2025-07-24
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies for panoramic video stitching rely on local and cloud computing power, resulting in high power consumption and slow image fusion speed.
Employing a low-power FPGA processor, it integrates multiple video input interfaces, a parallel processing core, an encoder, and a video output interface. It utilizes the MIPI-CSI2 bus, AXI-Stream bus, and HDMI transmitter to achieve rapid fusion and encoding of multiple video streams, and optimizes power consumption by combining a dynamic power management module.
It achieves low-latency, high-definition panoramic video display, meeting the real-time display requirements of in-vehicle systems and reducing system power consumption.
Smart Images
Figure CN224503442U_ABST
Abstract
Description
Technical Field
[0001] This utility model relates to the field of image processing technology, and in particular to a low-power panoramic video stitching FPGA processor. Background Technology
[0002] Panoramic video refers to wide-angle video content that exceeds the normal field of view. Commonly seen 360-degree circular / cylindrical videos provide a 360-degree horizontal view, while spherical panoramas can reach 180 degrees vertically, thus covering the user's entire three-dimensional space. Panoramic video offers users a more immersive experience and has applications in many fields, such as virtual reality, security, surveillance, film, video communication, education, and tourism.
[0003] Existing technologies mainly use software processing to map video images from multiple real-time video sequences onto a panoramic projection surface. However, the required computing power depends not only on local processing but also on cloud support, resulting in high power consumption and slow image fusion and stitching speed. Utility Model Content
[0004] This application proposes a low-power panoramic video stitching FPGA processor for fusing images from multiple cameras or other image acquisition devices to quickly generate panoramic videos.
[0005] In one aspect, a low-power panoramic video stitching FPGA processor includes: a multi-channel video input interface, a parallel processing core, an encoder, and a video output interface integrated into the top circuit layer of the board.
[0006] Multiple video input interfaces are used to connect to external image terminals;
[0007] The input of the parallel processing core is electrically connected to the output of the multi-channel video input interface via the MIPI-CSI2 bus; the parallel processing core consists of multiple CLBs that communicate with the DSP chip via the AXI-Stream bus.
[0008] The output of the parallel processing core is electrically connected to the input of the encoder, and the output of the encoder is connected to the display via a video output interface; the parallel processing core is electrically connected to the BRAM.
[0009] In conjunction with the first aspect, the multi-channel video input interface includes an AHD port for multiple camera inputs; wherein the AHD port integrates signal conditioning circuitry and adaptive power gating circuitry.
[0010] In conjunction with the first aspect, the TVALID signal output terminal and TREADY signal input terminal of each CLB in the parallel processing core are electrically connected to the TREADY signal input terminal and TVALID signal output terminal of the corresponding DSP chip, respectively, and the TDATA output terminal of the CLB is electrically connected to the TDATA input terminal of the DSP chip, thus forming an AXI-Stream bus communication link.
[0011] In conjunction with the first aspect, the BRAM of the parallel processing core is electrically connected to the cache data output terminals of each group of CLBs and the intermediate result input terminals of the DSP chip via the high-speed interconnect bus inside the FPGA.
[0012] In conjunction with the first aspect, the output terminal of the parallel processing core is electrically connected to the input terminal of the encoder via an AXI-Stream bus, the TVALID signal output terminal of the parallel processing core is electrically connected to the TREADY signal input terminal of the encoder, and the TDATA output terminal of the parallel processing core is electrically connected to the TDATA input terminal of the encoder.
[0013] In conjunction with the first aspect, the output terminal of the encoder is electrically connected to the video output interface via an HDMI transmitter, and the HDMI signal output terminal of the video output interface is electrically connected to the HDMI signal input terminal of the display.
[0014] In conjunction with the first aspect, the parallel processing core also includes a dynamic power management module, wherein the clock enable signal output terminal of the dynamic power management module is electrically connected to the clock enable pin of the CLB, and the power gating signal output terminal of the dynamic power management module is electrically connected to the power control pin of the DSP chip.
[0015] In conjunction with the first aspect, the load monitoring signal input terminal of the dynamic power management module is electrically connected to the performance counter output terminal of the CLB and the operation status register output terminal of the DSP chip.
[0016] In conjunction with the first aspect, the write enable terminal of the BRAM is electrically connected to the frame valid signal output terminal of the CLB, and the read enable terminal of the BRAM is electrically connected to the data request signal output terminal of the DSP chip.
[0017] In conjunction with the first aspect, the multi-channel video input interface includes at least two independent input channels, the output of each input channel is electrically connected to different data receiving ports of the parallel processing core, and the input of each input channel is connected to the video output of different external image terminals.
[0018] The technical advantages of this application are as follows:
[0019] In this application, during video stitching, the multi-channel video input interface integrates AHD ports and signal conditioning circuits, supporting anti-interference filtering, level amplification, and impedance matching of analog signals from external cameras to ensure stable video signal transmission. The multi-channel video input interface includes at least two independent channels, avoiding single-channel data congestion, reducing system input latency, and meeting the real-time display requirements of in-vehicle panoramic systems. A dedicated computing unit in the DSP chip performs color space conversion and feature extraction, while the CLB completes video preprocessing. Together, they achieve rapid fusion of multiple video streams without relying on cloud computing power. Multiple CLBs and the DSP form a parallel processing link via the AXI-Stream bus, enabling parallel execution of video preprocessing (distortion correction, downsampling) and feature matching. The encoder, in conjunction with the HDMI transmitter, encodes the stitched panoramic video data into a TMDS differential signal, which is transmitted to the display in real-time through the video output interface, achieving low-latency, high-definition panoramic image display.
[0020] Other features and advantages of this invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of this invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.
[0021] The technical solution of this utility model will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0022] The accompanying drawings are provided to further illustrate the present invention and form part of the specification. They are used together with the embodiments of the present invention to explain the present invention, but do not constitute a limitation thereof. In the drawings:
[0023] Figure 1 This is a hardware architecture diagram of an FPGA processor for low-power panoramic video stitching in an embodiment of this utility model.
[0024] Figure 2 This is a hardware architecture diagram of the parallel processing core in an embodiment of this utility model;
[0025] Figure 3 This is a first hardware illustration of the multi-channel video input interface in this embodiment of the present utility model;
[0026] Figure 4 This is a diagram illustrating the second hardware architecture of the parallel processing core in this embodiment of the present invention;
[0027] Figure 5 This is a diagram illustrating the third hardware architecture of the parallel processing core in this embodiment of the present invention;
[0028] Figure 6This is a diagram illustrating the fourth hardware architecture of the parallel processing core in this embodiment of the present invention;
[0029] Figure 7 This is a diagram illustrating the first hardware connection for output display in an embodiment of this utility model. Detailed Implementation
[0030] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0031] Example 1:
[0032] See Figure 1 and Figure 2 A low-power panoramic video stitching FPGA processor includes: a multi-channel video input interface 10, a parallel processing core 20, an encoder 40, and a video output interface 30 integrated in the top circuit layer of board 1.
[0033] The multi-channel video input interface 10 is used to connect to an external image terminal 60;
[0034] In the specific implementation process, the external image terminal is a video stream input device, mainly a camera. The video stream is transmitted to the MAX96712 serializer through the MIPI-CSI2 interface, and then transmitted to the MAX9295 deserializer on the FPGA side via a coaxial cable. The deserialized LVDS signal is converted into parallel data by the MIPI-CSI2 receiver IP core of the FPGA, and finally input into the parallel processing core.
[0035] The input of the parallel processing core is electrically connected to the output of the multi-channel video input interface 10 via the MIPI-CSI2 bus; the parallel processing core communicates with the DSP chip 202 via multiple sets of CLB201 through the AXI-Stream bus.
[0036] Multiple CLB201 units are used to perform basic correction and dynamic sampling on the input parallel data, and send it to the corresponding DSP communication chip via the AXI-Stream bus. The DSP chip 202 performs color space conversion and feature matching to fuse multiple video streams and output panoramic RGB video data.
[0037] The output of the parallel processing core 20 is electrically connected to the input of the encoder 40, and the output of the encoder 40 is connected to the display through the video output interface 30; wherein, the parallel processing core 20 is electrically connected to the BRAM.
[0038] The encoder 40 is used to digitally encode the panoramic RGB video data, which is then converted into a TMDS differential signal by the HDMI transmitter and transmitted to the display through the video output interface 30 to achieve real-time display of the panoramic image.
[0039] Example 2:
[0040] See Figure 3 The multi-channel video input interface 10 includes an AHD port for multiple camera inputs; wherein, the AHD port integrates a signal conditioning circuit 101 and an adaptive power gating circuit 102. The AHD port is used to connect an external AHD camera via a coaxial cable to receive its analog video signal. The signal conditioning circuit 101 is used to perform anti-interference filtering, level amplification, and impedance matching on the analog video signal; the adaptive power gating circuit 102 detects the AHD signal status and dynamically controls the camera power supply to reduce idle power consumption.
[0041] Example 3:
[0042] See Figure 1 and Figure 4 In the parallel processing core 20, the TVALID signal output terminal and TREADY signal input terminal of each group of CLB201 are electrically connected to the TREADY signal input terminal and TVALID signal output terminal of the corresponding DSP chip 202, respectively. The TDATA output terminal of CLB201 is electrically connected to the TDATA input terminal of DSP chip 202, forming an AXI-Stream bus communication link.
[0043] The CLB201 is a configurable logic block with eight configuration groups, implementing video preprocessing logic, including distortion correction, downsampling, and buffer control. The DSP chip 202 is used for dedicated computing, performing feature recognition and extraction. Each AXI-Stream bus contains four signals: TVALID, TREADY, TDATA (32-bit), and TLAST (end-of-frame marker), used for data stream transmission between the CLB201 and DSP202.
[0044] Example 4:
[0045] See Figure 1 and Figure 5The BRAM of the parallel processing core 20 is electrically connected to the buffer data output terminals of each group of CLB201 and the intermediate result input terminal of the DSP chip 202 via the FPGA's internal high-speed interconnect bus. The BRAM50 of the parallel processing core 20 is connected to the buffer output terminal of CLB201 and the intermediate result input terminal of DSP via the FPGA's high-speed interconnect bus. After CLB201 preprocesses the video data, it writes it to the BRAM (such as distortion correction pixel values + coordinates); the DSP reads the data from the BRAM to complete feature calculations (such as SAD values) and writes the results back to BRAM50; CLB201 then reads the feature results in BRAM50 to complete the panoramic stitching. The dual-port BRAM50 enables efficient data interaction between CLB201 and DSP chip 202, meeting the requirements of low latency (≤3 clock cycles), high throughput (500MB / s), and low power consumption (idle static power consumption <1mW) for real-time stitching.
[0046] Example 5:
[0047] See Figure 1 and Figure 4 The output of the parallel processing core 20 is electrically connected to the input of the encoder 40 via an AXI-Stream bus. The TVALID signal output of the parallel processing core 20 is electrically connected to the TREADY signal input of the encoder 40, and the TDATA output of the parallel processing core 20 is electrically connected to the TDATA input of the encoder 40. The parallel processing core 20 and the encoder 40 are connected via an AXI-Stream bus. The TVALID output of the parallel processing core 20 is connected to the TREADY input of the encoder 40, and the TDATA output is connected to the TDATA input of the encoder 40. When the parallel processing core 20 completes video stitching, it sets TVALID = 1 (data valid). When the encoder 40 detects TVALID = 1 and is ready, it sets TREADY = 1 (ready to receive). The parallel processing core 20 then transmits the stitched video stream to the encoder 40 for encoding via TDATA. This achieves low-latency (≤2 clock cycles) and packet-loss-free real-time video transmission, meeting the real-time display requirements of the vehicle-mounted panoramic system.
[0048] Example 6:
[0049] See Figure 7The output of encoder 40 is electrically connected to video output interface 30 via an HDMI transmitter. The HDMI signal output of video output interface 30 is electrically connected to the HDMI signal input of the display. The output of encoder 40 is connected to video output interface 30 via the HDMI transmitter, and the HDMI signal of video output interface 30 is connected to the HDMI port of display 70. After compression by encoder 40, the stitched video is converted into an HDMI protocol signal (TMDS differential signal) by the HDMI transmitter and transmitted to display 70 via video output interface 30. Display 70 receives and decodes the HDMI signal, displaying the panoramic image in real time, achieving low-latency, high-definition in-vehicle panoramic video output.
[0050] Example 7:
[0051] See Figure 4 The parallel processing core 20 also includes a dynamic power management module 203. The clock enable signal output of the dynamic power management module 203 is electrically connected to the clock enable pin of the CLB 201, and the power gating signal output of the dynamic power management module 203 is electrically connected to the power control pin of the DSP chip 202. The dynamic power management module 203 of the parallel processing core 20 controls the power consumption of the CLB 201 through the clock enable signal and the DSP through the power gating signal. When a decrease in video splicing load is detected, the module outputs clock enable = 0, shutting down part of the CLB clock (reducing dynamic power consumption); if the DSP completes feature calculation and has no new tasks, it outputs power gating = 0, cutting off its power supply (reducing static power consumption). When the load recovers, clock enable = 1 and power gating = 1, and the module reactivates the CLB 201 and DSP. This achieves dynamic power optimization, reducing system power consumption during idle periods and meeting the low-power requirements of automotive equipment.
[0052] Example 8:
[0053] See Figure 4 The load monitoring signal input terminal of the dynamic power management module 203 is electrically connected to the performance counter output terminal of the CLB201 and the operation status register output terminal of the DSP chip 202. The dynamic power management module 203 connects to the performance counter of the CLB201 and the DSP operation status register via the load monitoring signal terminal. The performance counter of the CLB201 records the number of pixel processing operations in real time, and the status register of the DSP chip 202 outputs "0" (idle) or "1" (busy). When the counter value is less than the threshold and the DSP status is 0, the module determines that the load is low and triggers clock enable shutdown (CLB201 frequency reduction) or power gating cutoff (DSP power-off); it automatically recovers when the load increases. This achieves dynamic adjustment of power consumption according to actual operation needs, reducing the average power consumption of the system.
[0054] Example 9:
[0055] The write enable pin of BRAM50 is electrically connected to the frame valid signal output pin of CLB 201, and the read enable pin of BRAM50 is electrically connected to the data request signal output pin of DSP chip 202.
[0056] The write enable pin of BRAM50 is connected to the frame valid signal pin of CLB 201, and the read enable pin is connected to the data request pin of DSP chip 202. After CLB201 completes single-frame stitching, it outputs a high-level frame valid signal, triggering the write enable pin of BRAM50. CLB201 then writes the YUV420 format stitched data into BRAM50. When DSP chip 202 needs to process the image, it outputs a rising edge data request signal, triggering the read enable pin of BRAM50, and transferring the stored data to DSP chip 202 for feature extraction. By synchronizing read and write operations with the frame valid and request signals, data conflicts are avoided, achieving low-latency (≤2 cycles) and low-power cache interaction between CLB201 and DSP.
[0057] Example 10:
[0058] See Figure 1 The multi-channel video input interface 10 includes at least two independent input channels. The output of each input channel is electrically connected to different data receiving ports of the parallel processing core 20, and the input of each input channel is connected to the video output of different external image terminals. The multi-channel video input interface 10 has multiple independent input channels (corresponding to the vehicle's front / rear / left / right cameras). The input of each channel is connected to a monocular camera, and the output is connected to the Port0-Port3 data receiving ports of the parallel processing core 20. The fisheye video captured by the camera is converted into LVDS signals through the channels and then transmitted in parallel to the processing core. The processing core simultaneously receives four video channels, performs correction, alignment, and fusion operations, and achieves 360° panoramic stitching, avoiding single-channel data congestion and reducing system input latency to ≤10ms.
[0059] Obviously, those skilled in the art can make various modifications and variations to this utility model without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this utility model and their equivalents, this utility model also intends to include these modifications and variations.
Claims
1. A low-power panoramic video stitching FPGA processor, characterized in that, include: The multi-channel video input interface (10), parallel processing core (20), encoder (40) and video output interface (30) are integrated into the top circuit layer of the board (1). The multi-channel video input interface (10) is used to connect to an external image terminal (60); The input of the parallel processing core (20) is electrically connected to the output of the multi-channel video input interface (10) via the MIPI-CSI2 bus; wherein, the parallel processing core (20) is composed of multiple sets of CLBs connected to the parallel processing link of the DSP communication chip via the AXI-Stream bus; The output of the parallel processing core (20) is electrically connected to the input of the encoder (40), and the output of the encoder (40) is connected to the display (70) through the video output interface (30); wherein, the parallel processing core (20) is electrically connected to the BRAM (50); The TVALID signal output terminal and TREADY signal input terminal of each CLB (201) in the parallel processing core (20) are electrically connected to the TREADY signal input terminal and TVALID signal output terminal of the corresponding DSP chip (202), respectively. The TDATA output terminal of the CLB (201) is electrically connected to the TDATA input terminal of the DSP chip (202), forming an AXI-Stream bus communication link. The parallel processing core also includes a dynamic power management module (203), the clock enable signal output terminal of the dynamic power management module (203) is electrically connected to the clock enable pin of the CLB (201), and the power gating signal output terminal of the dynamic power management module (203) is electrically connected to the power control pin of the DSP chip (202).
2. The FPGA processor for low-power panoramic video stitching as described in claim 1, characterized in that, The multi-channel video input interface (10) includes an AHD port for multiple camera inputs; wherein the AHD port integrates a signal conditioning circuit (101) and an adaptive power gating circuit (102).
3. The FPGA processor for low-power panoramic video stitching as described in claim 1, characterized in that, The BRAM (50) of the parallel processing core (20) is electrically connected to the cache data output terminal of each group of CLB (201) and the intermediate result input terminal of the DSP chip (202) through the high-speed interconnect bus inside the FPGA.
4. The FPGA processor for low-power panoramic video stitching as described in claim 1, characterized in that, The output of the parallel processing core (20) is electrically connected to the input of the encoder (40) via an AXI-Stream bus. The TVALID signal output of the parallel processing core (20) is electrically connected to the TREADY signal input of the encoder (40). The TDATA output of the parallel processing core (20) is electrically connected to the TDATA input of the encoder (40).
5. The FPGA processor for low-power panoramic video stitching as described in claim 1, characterized in that, The output end of the encoder (40) is electrically connected to the video output interface (30) via an HDMI transmitter, and the HDMI signal output end of the video output interface (30) is electrically connected to the HDMI signal input end of the display (70).
6. The FPGA processor for low-power panoramic video stitching as described in claim 1, characterized in that, The load monitoring signal input terminal of the dynamic power management module (203) is electrically connected to the performance counter output terminal of the CLB (201) and the operation status register output terminal of the DSP chip (202).
7. The FPGA processor for low-power panoramic video stitching as described in claim 1, characterized in that, The write enable terminal of the BRAM (50) is electrically connected to the frame valid signal output terminal of the CLB (201), and the read enable terminal of the BRAM (50) is electrically connected to the data request signal output terminal of the DSP chip (202).
8. The FPGA processor for low-power panoramic video stitching as described in claim 1, characterized in that, The multi-channel video input interface (10) includes at least two independent input channels. The output of each input channel is electrically connected to different data receiving ports of the parallel processing core (20), and the input of each input channel is connected to the video output of different external image terminals.