A video splicing and fusion method and system based on FPGA hard acceleration

Abstract

The application relates to a video splicing and fusion method and system based on FPGA hard acceleration, and belongs to the technical field of video splicing. The method comprises the following steps: performing pixel-by-pixel beat alignment on multi-channel synchronous video streams of each video channel; performing pixel-level photometric equalization on the aligned multi-channel synchronous video streams to generate photometric consistent video streams and write the photometric consistent video streams into an off-chip storage medium; performing inverse projection recursion on the photometric consistent video streams, predicting addressing envelopes based on a target block space to control the off-chip storage medium to read data in a continuous long burst mode, store the data into an on-chip storage matrix, and then obtain a geometric correction video stream through parallel interpolation; acquiring a binary seam mask, performing multi-scale frequency decomposition and adaptive weighted fusion on the binary seam mask and the geometric correction video stream, and outputting a panoramic splicing video stream after delay compensation and saturation clamping reconstruction. The application reduces the off-chip bandwidth requirement and on-chip resource consumption of FPGA in panoramic splicing.

Description

Technical Field

[0001] This invention belongs to the field of video stitching technology, specifically relating to a video stitching and fusion method and system based on FPGA hardware acceleration. Background Technology

[0002] With the rapid development of fields such as autonomous driving, industrial visual inspection, and panoramic security monitoring, the application demand for multi-camera panoramic video stitching technology continues to grow, placing increasingly higher demands on end-to-end latency, stitched image quality, system power consumption, and integration. Field-programmable gate arrays (FPGAs), with their inherent parallel computing characteristics and high real-time performance, have become one of the mainstream implementation carriers for video stitching hardware acceleration.

[0003] Currently, most mainstream FPGA video stitching solutions employ a heterogeneous architecture. This means the FPGA is only responsible for front-end video acquisition, interface conversion, and data transmission, while core algorithms such as geometric transformation, seam optimization, and image fusion are performed on off-chip general-purpose processors like DSPs, ARMs, or GPUs. This type of solution requires repeatedly transferring high-definition video stream data between multiple chips, resulting in end-to-end latency typically reaching tens to hundreds of milliseconds. This makes it unsuitable for low-latency applications such as autonomous driving and high-speed industrial inspection. Furthermore, multi-chip architectures have low integration and high power consumption, making them difficult to apply in miniaturized embedded devices.

[0004] Most existing pure FPGA-based stitching solutions rely on offline pre-calculated coordinate lookup tables. These tables have fixed parameters after factory installation. When cameras experience focal length changes due to physical vibration or temperature drift, or when dynamic adjustments to the number of cameras, resolution, or projection curvature are required, the static lookup table becomes ineffective, leading to stitching misalignment and insufficient flexibility and environmental adaptability. Furthermore, existing hardware solutions often employ simple weighted average fusion algorithms, which are prone to noticeable stitching seams or edge ghosting when processing complex textures at a single scale. The stitching seam positions are often fixed, easily cutting into moving targets and causing image fragmentation, affecting visual quality and subsequent intelligent recognition accuracy. Most solutions only focus on geometric alignment, lacking real-time, frame-by-frame multi-path photometric consistency compensation, which can easily lead to bright and dark color blocks and color deviations in panoramic images, failing to meet the demands of high-quality applications.

[0005] In summary, how to build a fully hardware-based deep processing pipeline within a pure FPGA chip, completely eliminating the limitations of data interaction with off-chip general-purpose processors and offline static lookup tables, while achieving high-fidelity multi-scale fusion that balances pixel-level frame synchronization, real-time photometric equalization compensation, predictive dynamic projection addressing, and dynamic moving target avoidance, is a severe engineering challenge in this field for achieving high-performance panoramic video stitching hardware acceleration. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a video stitching and fusion method and system based on FPGA hardware acceleration. The objective of this invention can be achieved through the following technical solutions: Perform pixel-by-pixel beat alignment on the multi-channel synchronous video streams of each video channel; Pixel-level photometric equalization is performed on the aligned multi-channel synchronous video streams to generate a photometric consistent video stream, which is then written to an off-chip storage medium. The photometric consistent video stream is subjected to inverse projection recursion. Based on the target block spatial prediction addressing envelope, the off-chip storage medium is controlled to perform continuous long burst reads. After being transferred to the on-chip storage matrix, the geometrically corrected video stream is obtained through parallel interpolation. Obtain the binary stitching mask, perform multi-scale frequency decomposition and adaptive weighted fusion on the binary stitching mask and the geometrically corrected video stream, and output the panoramic stitched video stream after delay compensation and saturation clamp reconstruction.

[0007] Specifically, the pixel-by-pixel beat alignment employs cross-clock domain frame synchronization processing: The received multi-channel heterogeneous video data sources are written into the cross-clock domain asynchronous ping-pong buffer units corresponding to each video channel under the drive of the corresponding in-channel recovery clock. Under the system's master clock, the frame start characteristic status of each of the cross-clock domain asynchronous ping-pong buffer units is continuously monitored; When the frame start feature is detected to have arrived in the cross-clock domain asynchronous ping-pong buffer unit of all video channels, the read enable signal of all cross-clock domain asynchronous ping-pong buffer units is pulled high synchronously, and the pixel data stream is read synchronously to complete the cross-clock domain alignment.

[0008] Specifically, the pixel-level photometric equalization includes: Global photometric statistical features of multiple synchronous video streams are extracted; photometric gain compensation coefficients are calculated by combining historical frame information; pixel scaling and constant offset operations are performed on the multiple synchronous video streams based on the photometric gain compensation coefficients, and a photometric consistent video stream is output and written to an off-chip storage medium; wherein, the global photometric statistical features include the total brightness value and average absolute deviation value of each video channel, and the historical frame information includes the reference mean and reference deviation of the reference video channel.

[0009] Specifically, the extraction process of the global photometric statistical features includes: The multi-channel synchronous video streams are processed through a hardware sliding window to perform de-mosaic processing, reconstruct red, green, and blue color images, and convert them into a luminance-chrominance separation format through color space transformation; within the current frame, the pixel luminance components of each video channel are dynamically calculated pixel by pixel to obtain the total luminance value of the current frame; the absolute value of the difference between the pixel luminance components and the average value of the latched previous frame is calculated synchronously to generate an average absolute deviation value.

[0010] Specifically, the calculation process of the photometric gain compensation coefficient includes: Using the pre-configured reference video channel in the multi-channel synchronous video stream as the global reference channel, the reference mean and reference deviation of the global reference channel are extracted as the historical frame information; during the frame blanking period, the target parameters of each video channel are subjected to time-axis smoothing iteration through a first-order recursive smoothing filter to calculate the photometric gain compensation coefficient of the corresponding video channel.

[0011] Specifically, the pixel calculation steps for pixel-level luminance equalization include: During the frame blanking period, the constant term offset is calculated using the reference mean and the photometric gain compensation coefficient. During real-time pixel stream transmission, each pixel is scaled based on the photometric gain compensation coefficient by a hardware arithmetic unit, and the constant term offset is incorporated into the scaling result to output the photometric consistent video stream. The pixel mean of each video channel is approximately calculated by hardware shifting operation.

[0012] Specifically, the reverse projection recursion includes: When generating a panoramic spatial image of the target using raster scanning, the initial values ​​of the numerator and denominator of the first pixel in the column are pre-calculated at the beginning of each row scan. While sequentially traversing pixels in the current row, the step register is used to increment the step size of the numerator and denominator of the first pixel in the column, and the numerator and denominator of the next pixel are dynamically recursively output. The calculated resampling mapping coordinates are divided into an integer part and a fractional part. The integer part is used to address the off-chip storage medium, and the fractional part is used to participate in the interpolation weight calculation.

[0013] Specifically, the process of generating the target block spatial prediction addressing envelope includes: The target panoramic space is divided into regularly arranged two-dimensional blocks according to a preset fixed size; When processing the current block space, the coordinates of the four corner points of the block space are read, and the projection floating-point coordinates of the four corner points in the source image of the corresponding video channel are mapped using the pre-stored inverse transformation matrix. The maximum and minimum boundaries of the projected floating-point coordinates in the source image are obtained by using a hardware comparator array, and the spatial prediction addressing envelope of the target block is determined.

[0014] Specifically, the continuous long burst read and transfer to the on-chip storage matrix includes: The burst data length is dynamically calculated based on the horizontal width of the target block space prediction address envelope, and the off-chip storage medium is controlled to perform continuous long burst reads with physical address row alignment. The chip is configured with a first storage region and a second storage region to form the on-chip storage matrix; the first storage region receives data from the current block space through the continuous long burst read, and the second storage region synchronously outputs data from the previous block space to perform data reading and interpolation operations in parallel.

[0015] Specifically, the parallel interpolation includes: The on-chip storage matrix is ​​divided into a preset number of storage blocks, and a two-dimensional interleaving addressing mechanism is used to map the low-order bits of the physical addresses of adjacent pixels to different storage blocks; The neighboring pixels required for interpolation are extracted in parallel without conflict within a single master clock cycle using a multiplexer, and the interpolation is completed by combining the fractional part of the resampled mapped coordinates as weighting coefficients, and the geometrically corrected video stream is output.

[0016] Specifically, the process of obtaining the binary seam mask includes: During the forward state transition calculation, two sets of single-row-depth temporary buffers are used to form a row buffer structure, and the cumulative path cost of adjacent rows is read and written alternately, thus limiting the size of the cumulative cost cache to the width of a single row pixel. During the calculation process, a two-bit encoded record is used to generate the precursor direction with the minimum cumulative cost value. A direction pointer matrix is ​​constructed and stored in the on-chip static storage array to complete the compressed storage of the precursor direction pointer. The direction pointer matrix constitutes the precursor direction pointer lookup table. The horizontal coordinate with the lowest cumulative cost is retrieved as the starting point for track finding. The row number is decremented in reverse, and the corresponding preceding direction pointer is obtained by looking up the preceding direction pointer lookup table to generate the binary stitching mask.

[0017] Furthermore, the binary seam mask is calculated from the multi-feature fusion energy cost field of the overlapping region of adjacent video channels. The calculation process of the multi-feature fusion energy cost field specifically includes: The sum of the absolute differences in luminance and chrominance of adjacent video channels within the overlapping area is calculated as the color difference cost. The sum of the absolute values ​​of the horizontal and vertical gradient differences between the two images within the overlapping region is calculated using the edge convolution operator, and this sum is used as the gradient magnitude cost. By comparing the brightness difference between adjacent frames with a programmable motion threshold, a maximum penalty value is applied to moving pixels as the motion detection cost. The color difference cost, gradient magnitude cost, and motion detection cost are proportionally fused using a hardware arithmetic tree to construct the multi-feature fusion energy cost field.

[0018] Specifically, the multi-scale frequency decomposition and adaptive weighted fusion include: The binary stitching mask is fed into a low-pass filter pipeline corresponding to the Gaussian filter in the image space, and the required fusion weight matrix for each scale level is adaptively decomposed and constructed. At each level of detail extraction path, a corresponding depth of synchronization buffer unit is configured to absorb and process the beat difference, so as to achieve zero-deviation timing alignment of each frequency band component in spatial physical coordinates and complete the delay compensation. During the cascaded aggregation reconstruction, the pixel values ​​are expanded to a higher bit width to accommodate intermediate accumulated results without overflowing, and threshold boundary saturation clamping is performed at the output of each level of aggregation reconstruction through pure combinational logic, and finally the panoramic stitched video stream is output.

[0019] Specifically, a video stitching and fusion system based on FPGA hardware acceleration includes: a multi-channel video input interface for accessing multiple heterogeneous video data sources; an off-chip storage medium for storing video frame data; and an FPGA chip electrically connected to the multi-channel video input interface and the off-chip storage medium, wherein the FPGA chip is internally configured with: The frame synchronization unit is used to perform cross-clock domain frame synchronization processing on the multi-channel heterogeneous video data source and output a multi-channel synchronized video stream with pixel-by-pixel beat alignment. A photometric equalization compensation unit is used to perform pixel-level photometric equalization on the aligned multi-channel synchronous video streams, generate a photometric consistent video stream, and write it to an off-chip storage medium. The predictive geometric correction unit is used to perform inverse projection recursion on the photometric consistent video stream, control the off-chip storage medium to perform continuous long burst reads based on the target block spatial prediction addressing envelope, and output the geometrically corrected video stream through parallel interpolation; An adaptive stitching fusion unit is used to obtain a binary stitching mask and perform multi-scale frequency decomposition and adaptive weighted fusion on the geometrically corrected video stream to output a panoramic stitched video stream.

[0020] The beneficial effects of this invention are as follows: (1) This invention adopts a single FPGA full hardware deep pipeline architecture, integrating the complete splicing link of video acquisition synchronization, photometric compensation, geometric transformation, stitching seam search, and multi-scale fusion into a single FPGA. This completely eliminates the dependence of traditional heterogeneous architectures on off-chip general-purpose processors, eliminates the communication bottleneck of repeated transfer of high-definition video data between chips, and reduces end-to-end latency to the millisecond level or even sub-millisecond level, fully meeting the usage requirements of low-latency scenarios such as autonomous driving and high-speed industrial inspection. At the same time, through hardware optimization mechanisms such as predictive burst memory access and row buffer compressed storage, the utilization rate of off-chip memory bandwidth is greatly improved, and the consumption of on-chip logic resources is reduced. The overall system has high integration, low power consumption, and small size, and can be directly deployed in various miniaturized embedded devices.

[0021] (2) This invention adopts a fully dynamic adaptive stitching processing architecture, which does not rely on offline pre-calculated static coordinate lookup tables. It supports online flexible configuration of parameters such as the number of cameras, resolution, and projection curvature. It can adapt to changes in focal length caused by physical vibration and temperature drift, and has strong environmental adaptability. Through the row buffer compression dynamic planning stitching seam search mechanism, it can sense moving targets in the overlapping area in real time and automatically adjust the stitching seam path to avoid image breakage caused by the cutting of moving objects. Combined with the adaptive fusion algorithm of multi-scale frequency decomposition, it takes into account the clarity of high-frequency details and the smoothness of low-frequency colors, with no obvious stitching seams and ghosting artifacts. With the frame-by-frame real-time light intensity equalization compensation, it can eliminate the brightness and color differences of multiple sensors, improve the overall color consistency and visual quality of the panoramic image, and meet the requirements of high-quality applications. Attached Figure Description

[0022] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating a video stitching and fusion method based on FPGA hardware acceleration according to the present invention. Figure 2 This is an architecture diagram of a video stitching and fusion system based on FPGA hardware acceleration according to the present invention; Figure 3 This is a diagram of the global brightness dynamic equalization module architecture of a video stitching and fusion method based on FPGA hardware acceleration according to the present invention. Figure 4 This is a microarchitecture diagram of a dynamic planning module for a video stitching and fusion system based on FPGA hardware acceleration, as described in this invention. Figure 5 This is a schematic diagram illustrating the video splicing and fusion effect of the present invention; Figure 6 This is a rendering of the effect of regular video splicing and merging. Detailed Implementation

[0024] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the specific implementation methods, structures, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0025] Figure 1 The process of the video stitching and fusion method based on FPGA hardware acceleration of the present invention is shown, including the steps of pixel-by-pixel beat alignment, photometric equalization, geometric correction, stitching mask generation and multi-scale fusion. Figure 2 This is an architecture diagram of a video stitching and fusion system based on FPGA hardware acceleration, as described in this invention; Figure 2 As shown, after multiple video streams enter the FPGA chip through the input interface, they are processed sequentially by the frame synchronization unit, the photometric equalization compensation unit, the predictive geometric correction unit, and the adaptive stitching fusion unit. The predictive geometric correction unit interacts with the off-chip storage medium through continuous long burst read and write, and finally outputs the panoramic stitched video through the panoramic stitching video output interface. Figure 5 This is a schematic diagram illustrating the video splicing and fusion effect of the present invention; Figure 6 The image shows the result of a standard video stitching and blending process. Figure 5 and Figure 6 In stark contrast: conventional video splicing ( Figure 6 The bottom of the image shows obvious splicing breaks and misalignments, the track markings and ground edges are discontinuous, and there is a sense of disjointedness near fixed structures such as the right-side pillars; while the video image using the splicing technology of this invention can be clearly seen (e.g. Figure 5 As shown), the overall image is coherent and the transitions are natural. Overlapping areas are seamlessly connected, and the ground lines are smoothly joined without breaks, obvious seams, or ghosting. Furthermore, Figure 5 The overall panoramic view is complete and unified, significantly improving the continuity and visual integrity of the stitched image quality.

[0026] Please see Figures 1-4 A video stitching and fusion method based on FPGA hardware acceleration includes the following steps: Perform pixel-by-pixel beat alignment on the multi-channel synchronous video streams of each video channel; Pixel-level photometric equalization is performed on the aligned multi-channel synchronous video streams to generate a photometric consistent video stream, which is then written to an off-chip storage medium. The photometric consistent video stream is subjected to inverse projection recursion. Based on the target block spatial prediction addressing envelope, the off-chip storage medium is controlled to perform continuous long burst reads. After being transferred to the on-chip storage matrix, the geometrically corrected video stream is obtained through parallel interpolation. Obtain the binary stitching mask, perform multi-scale frequency decomposition and adaptive weighted fusion on the binary stitching mask and the geometrically corrected video stream, and output the panoramic stitched video stream after delay compensation and saturation clamp reconstruction.

[0027] Specifically, the pixel-by-pixel beat alignment employs cross-clock domain frame synchronization processing: The received multi-channel heterogeneous video data sources are written into the cross-clock domain asynchronous ping-pong buffer units corresponding to each video channel under the drive of the corresponding in-channel recovery clock. Under the system's master clock, the frame start characteristic status of each of the cross-clock domain asynchronous ping-pong buffer units is continuously monitored; When the frame start feature is detected to have arrived in the cross-clock domain asynchronous ping-pong buffer unit of all video channels, the read enable signal of all cross-clock domain asynchronous ping-pong buffer units is pulled high synchronously, and the pixel data stream is read synchronously to complete the cross-clock domain alignment.

[0028] Specifically, the pixel-level photometric equalization includes: Global photometric statistical features of multiple synchronous video streams are extracted; photometric gain compensation coefficients are calculated by combining historical frame information; pixel scaling and constant offset operations are performed on the multiple synchronous video streams based on the photometric gain compensation coefficients, and a photometric consistent video stream is output and written to an off-chip storage medium; wherein, the global photometric statistical features include the total brightness value and average absolute deviation value of each video channel, and the historical frame information includes the reference mean and reference deviation of the reference video channel.

[0029] Specifically, the extraction process of the global photometric statistical features includes: The multi-channel synchronous video streams are processed through a hardware sliding window to perform de-mosaic processing, reconstruct red, green, and blue color images, and convert them into a luminance-chrominance separation format through color space transformation; within the current frame, the pixel luminance components of each video channel are dynamically calculated pixel by pixel to obtain the total luminance value of the current frame; the absolute value of the difference between the pixel luminance components and the average value of the latched previous frame is calculated synchronously to generate an average absolute deviation value.

[0030] Specifically, the calculation process of the photometric gain compensation coefficient includes: Using the pre-configured reference video channel in the multi-channel synchronous video stream as the global reference channel, the reference mean and reference deviation of the global reference channel are extracted as the historical frame information; during the frame blanking period, the target parameters of each video channel are subjected to time-axis smoothing iteration through a first-order recursive smoothing filter to calculate the photometric gain compensation coefficient of the corresponding video channel.

[0031] Specifically, the pixel calculation steps for pixel-level luminance equalization include: During the frame blanking period, the constant term offset is calculated using the reference mean and the photometric gain compensation coefficient. During real-time pixel stream transmission, each pixel is scaled based on the photometric gain compensation coefficient by a hardware arithmetic unit, and the constant term offset is incorporated into the scaling result to output the photometric consistent video stream. The pixel mean of each video channel is approximately calculated by hardware shifting operation.

[0032] Specifically, the reverse projection recursion includes: When generating a panoramic spatial image of the target using raster scanning, the initial values ​​of the numerator and denominator of the first pixel in the column are pre-calculated at the beginning of each row scan. While sequentially traversing pixels in the current row, the step register is used to increment the step size of the numerator and denominator of the first pixel in the column, and the numerator and denominator of the next pixel are dynamically recursively output. The calculated resampling mapping coordinates are divided into an integer part and a fractional part. The integer part is used to address the off-chip storage medium, and the fractional part is used to participate in the interpolation weight calculation.

[0033] Specifically, the process of generating the target block spatial prediction addressing envelope includes: The target panoramic space is divided into regularly arranged two-dimensional blocks according to a preset fixed size; When processing the current block space, the coordinates of the four corner points of the block space are read, and the projection floating-point coordinates of the four corner points in the source image of the corresponding video channel are mapped using the pre-stored inverse transformation matrix. The maximum and minimum boundaries of the projected floating-point coordinates in the source image are obtained by using a hardware comparator array, and the spatial prediction addressing envelope of the target block is determined.

[0034] Specifically, the continuous long burst read and transfer to the on-chip storage matrix includes: The burst data length is dynamically calculated based on the horizontal width of the target block space prediction address envelope, and the off-chip storage medium is controlled to perform continuous long burst reads with physical address row alignment. The chip is configured with a first storage region and a second storage region to form the on-chip storage matrix; the first storage region receives data from the current block space through the continuous long burst read, and the second storage region synchronously outputs data from the previous block space to perform data reading and interpolation operations in parallel.

[0035] Specifically, the parallel interpolation includes: The on-chip storage matrix is ​​divided into a preset number of storage blocks, and a two-dimensional interleaving addressing mechanism is used to map the low-order bits of the physical addresses of adjacent pixels to different storage blocks; The neighboring pixels required for interpolation are extracted in parallel without conflict within a single master clock cycle using a multiplexer, and the interpolation is completed by combining the fractional part of the resampled mapped coordinates as weighting coefficients, and the geometrically corrected video stream is output.

[0036] Specifically, the process of obtaining the binary seam mask includes: During the forward state transition calculation, two sets of single-row-depth temporary buffers are used to form a row buffer structure, and the cumulative path cost of adjacent rows is read and written alternately, thus limiting the size of the cumulative cost cache to the width of a single row pixel. During the calculation process, a two-bit encoded record is used to generate the precursor direction with the minimum cumulative cost value. A direction pointer matrix is ​​constructed and stored in the on-chip static storage array to complete the compressed storage of the precursor direction pointer. The direction pointer matrix constitutes the precursor direction pointer lookup table. The horizontal coordinate with the lowest cumulative cost is retrieved as the starting point for track finding. The row number is decremented in reverse, and the corresponding preceding direction pointer is obtained by looking up the preceding direction pointer lookup table to generate the binary stitching mask.

[0037] Furthermore, the binary seam mask is calculated from the multi-feature fusion energy cost field of the overlapping region of adjacent video channels. The calculation process of the multi-feature fusion energy cost field specifically includes: The sum of the absolute differences in luminance and chrominance of adjacent video channels within the overlapping area is calculated as the color difference cost. The sum of the absolute values ​​of the horizontal and vertical gradient differences between the two images within the overlapping region is calculated using the edge convolution operator, and this sum is used as the gradient magnitude cost. By comparing the brightness difference between adjacent frames with a programmable motion threshold, a maximum penalty value is applied to moving pixels as the motion detection cost. The color difference cost, gradient magnitude cost, and motion detection cost are proportionally fused using a hardware arithmetic tree to construct the multi-feature fusion energy cost field.

[0038] Specifically, the multi-scale frequency decomposition and adaptive weighted fusion include: The binary stitching mask is fed into a low-pass filter pipeline corresponding to the Gaussian filter in the image space, and the required fusion weight matrix for each scale level is adaptively decomposed and constructed. At each level of detail extraction path, a corresponding depth of synchronization buffer unit is configured to absorb and process the beat difference, so as to achieve zero-deviation timing alignment of each frequency band component in spatial physical coordinates and complete the delay compensation. During the cascaded aggregation reconstruction, the pixel values ​​are expanded to a higher bit width to accommodate intermediate accumulated results without overflowing, and threshold boundary saturation clamping is performed at the output of each level of aggregation reconstruction through pure combinational logic, and finally the panoramic stitched video stream is output.

[0039] Specifically, a video stitching and fusion system based on FPGA hardware acceleration includes: a multi-channel video input interface for accessing multiple heterogeneous video data sources; an off-chip storage medium for storing video frame data; and an FPGA chip electrically connected to the multi-channel video input interface and the off-chip storage medium, wherein the FPGA chip is internally configured with: The frame synchronization unit is used to perform cross-clock domain frame synchronization processing on the multi-channel heterogeneous video data source and output a multi-channel synchronized video stream with pixel-by-pixel beat alignment. A photometric equalization compensation unit is used to perform pixel-level photometric equalization on the aligned multi-channel synchronous video streams, generate a photometric consistent video stream, and write it to an off-chip storage medium. The predictive geometric correction unit is used to perform inverse projection recursion on the photometric consistent video stream, control the off-chip storage medium to perform continuous long burst reads based on the target block spatial prediction addressing envelope, and output the geometrically corrected video stream through parallel interpolation; An adaptive stitching fusion unit is used to obtain a binary stitching mask and perform multi-scale frequency decomposition and adaptive weighted fusion on the geometrically corrected video stream to output a panoramic stitched video stream.

[0040] Example 1: Single FPGA Binocular Panoramic Video Stitching Application scenario: Binocular vision panoramic stitching scenario, using a single FPGA as the core processing chip, with two heterogeneous video input interfaces and one external storage medium. The resolution of both input video channels is W×H, and the output panoramic stitching video resolution is W. total ×H.

[0041] System Architecture: The FPGA integrates a frame synchronization unit, a photometric equalization compensation unit, a predictive geometric correction unit, and an adaptive stitching fusion unit. External storage media is used for frame-level buffering. Specific Implementation Steps: 1. Cross-clock domain global frame synchronization After the two heterogeneous video data sources are parsed into raw pixel data and line and field synchronization signals by the front-end interface, they are written into the cross-clock domain asynchronous ping-pong buffer unit of the corresponding video channel. The global scheduling controller continuously polls the frame start flag and data depth of all channel buffer units under the system processing master clock. When it is detected that the frame start flags of both buffer units have arrived and both have stored at least one line of valid pixels, the read enable signals of the two buffer units are synchronously pulled up, and two synchronous video streams with strict pixel-by-pixel and line-by-line beat alignment are output.

[0042] 2. Bufferless real-time photometric equalization compensation Two synchronous video streams enter independent preprocessing pipelines, are de-mosaiced and reconstructed into RGB color images through a 3×3 hardware sliding window, and then converted into YUV4:2:2 format with luminance and chrominance separation. A pipelined strategy of "previous frame statistics, subsequent frame compensation" is adopted to accumulate the brightness value of the current frame pixel by pixel to obtain the total brightness value, which is then combined with the latched average value of the previous frame. Calculate the mean absolute deviation instead of the standard deviation to reduce computational complexity: In the formula, Let M be the average absolute deviation of the k-th channel, and M be the total number of pixels in a single frame. Let be the luminance value of the k-th channel pixel (u,v). The mean global brightness of the previous frame for the k-th channel is given; the total number of pixels M is aligned to a power of 2, and the mean is calculated by replacing the divider with a register right shift operation.

[0043] During the frame blanking period, based on pre-configured reference channel statistics. , For reference, a first-order recursive (IIR) smoothing filter is used to perform time-smoothing iterations on the gain coefficients and mean to suppress screen flicker: , , In the formula, N represents the current video frame number, and N-1 represents the previous video frame number; and These represent the luminance gain compensation coefficients of the Nth and N-1th frames of the kth channel after first-order IIR smoothing, respectively. and These represent the average brightness values ​​of the Nth and N-1th frames of the kth channel after first-order IIR smoothing, respectively. The transient target mean is represented by λ, which is a programmable smoothing factor ranging from 0 to 1, typically set to 0. or In hardware, this is manifested as truncating the high-order bits, shifting them, and adding them; g target The transient target gain calculated for the frame blanking period. This mechanism is equivalent to adding a low-pass filter to the time axis, effectively suppressing global image flicker caused by sudden changes in local strong light.

[0044] Offset of pre-calculated constant terms during frame blanking period The real-time pixel stream can complete the compensation in just one cycle of multiplication and addition: In the formula, This is the compensated luminance value. The compensated luminance-consistent video stream is written to off-chip storage medium. , Pre-configured baseline channel statistics.

[0045] like Figure 3 As shown in the global brightness dynamic equalization module architecture diagram, the entire module consists of three parts: a "global statistics engine," a "parameter calculation and smoothing filter," and a "pixel compensation engine." Global Statistics Engine: This module instantiates two completely independent, single-clock-cycle parallel accumulator data paths. When the... Pixel data of a frame As they flow into the assembly line one by one: Accumulator A performs global luminance accumulation calculation for the current frame, performing only unconditional addition logic in each clock cycle. , Accumulator B performs the calculation of the absolute deviation of the current frame, and first extracts the mean of the previous frame stored in the constant register. However, in each clock cycle, subtraction and absolute value calculation logic are performed, followed by accumulation. Finally, the summation of the deviations is completed synchronously during each data stream.

[0046] Parameter Calculation and Smoothing Filter: This module includes a shift register that instantiates a divider and a first-order recursive (IIR) smoothing filter using on-chip DSP resources. Upon receiving the End-of-Frame (EOF) flag, the shift register begins shifting the accumulated data calculated by the global statistics engine, after which the divider is activated for calculation. The calculation results are then fed into a first-order recursive (IIR) smoothing filter for smoothing.

[0047] Pixel Compensation Engine: This module mainly consists of a DSP multiply-accumulate unit, which can receive the YUV image data stream obtained after signal preprocessing and perform pixel-by-pixel brightness value... Calculate using the formula: Finally, the calculated YUV image data stream containing the compensated brightness value is written into an external DDR SDRAM memory for further processing.

[0048] 3. Predictive burst memory access geometric correction Image space transformation is performed using an inverse mapping method. The mapping relationship between the target panoramic coordinates (x, y) and the source image coordinates (u, v) in the homogeneous coordinate system is determined by the 3×3 homography matrix H. , Expanding it, we obtain the perspective division formula for the source image coordinates: , In the formula, h11 For example, h 11 is the element in the first row and first column of the homography matrix, and the same applies to the subsequent ones. Here, w is the scaling factor of the homogeneous coordinates, which has no direct physical meaning and is just an intermediate product of the homogeneous coordinate transformation.

[0049] During raster scanning, an in-row polynomial recursive optimization is employed: at the start of each row scan, the initial values ​​of the numerator and denominator of the first pixel in the column are calculated, and the horizontal step size (u_numerator step size ∆N) is accumulated within the row. ux =h 11 , v molecular step length ∆N vx =h 21 , denominator step size ∆D x =h 31 The numerator and denominator of the next pixel are generated cycle by cycle, and the integer coordinates are separated for addressing and the decimal coordinates are used for interpolation weights, saving DSP resources.

[0050] The target panoramic space is divided into two-dimensional blocks of fixed size, and the source image addressing envelope is calculated using the projected coordinates of the four corner points of each block. min ,u max ]×[v min ,v max [Controls off-chip storage media to perform continuous long burst reads with physical address row alignment, single-row burst length:] In the formula, The up rounding operator, u max and u min These are the maximum and minimum x-coordinates of the mapped address, respectively. Padding is the amount of edge redundant pixels reserved for bilinear interpolation. Pixel_Bytes is the number of bytes per pixel. AXI_Data_Width is the bus width. Max_Burst is the maximum single burst length allowed by the bus protocol. min(·) represents the minimum value function.

[0051] The chip is configured with two sets of storage areas forming a storage matrix, employing a ping-pong switching method to alternately perform data reception and output, achieving full concurrency between storage retrieval and interpolation calculation. Each set of storage areas is divided into four independent storage blocks, using a two-dimensional interleaved addressing mechanism to achieve conflict-free single-cycle reading of 2×2 neighboring pixels. Storage block index: The bilinear interpolation is performed by combining the decimal part of the coordinates, and the geometrically corrected video stream is output.

[0052] 4. Row-buffered compression dynamic programming seam search like Figure 4The microarchitecture diagram of the dynamic programming module shows a forward cost compression and accumulation pipeline, a dynamic projection adaptive go-around optimizer, and a backward backtracking finite state machine. After the image or PIV data stream is input into the dynamic projection adaptive go-around optimizer, the corresponding costs are calculated by the color difference, gradient magnitude, and motion detection cost calculation units in the global statistics engine. These costs are then weighted and fused by the pipeline's adder tree to obtain the total cost. The forward cost compression and accumulation pipeline then calculates the current row's cumulative cost using a three-input pipeline comparator and a saturated adder tree. This cost is compressed into an on-chip full storage matrix, which forms a predecessor direction pointer lookup table. Simultaneously, a ping-pong row cache matrix stores the historical cost of the previous row. Finally, the minimum cost tracker in the backward backtracking finite state machine captures the minimum cost point, retrieves the corresponding predecessor direction pointer from the lookup table, updates the coordinates based on this pointer using a one-dimensional recursive adder, and generates the binary stitching mask by backtracking row by row.

[0053] By extracting the color difference cost, gradient magnitude cost, and motion detection cost corresponding to each pixel (u,v) within the overlapping region of the two video streams, and weighted fusion using a hardware arithmetic tree, a pixel-level energy cost distribution covering the entire overlapping region (i.e., the multi-feature fusion energy cost field) is formed. The overlapping region of the two video streams is extracted, and three branch costs are calculated in parallel: color difference cost E... color (L1 norm calculation of luminance and chromaticity difference), gradient magnitude cost E grad (Sobel operator calculates the difference between horizontal and vertical gradients), motion detection cost E motion (Inter-frame differential detection of moving regions), the total energy cost is obtained through hardware multiply-accumulate tree weighted fusion: In the formula, α, β, and γ are the weight coefficients of the corresponding costs; u and v are the local integer pixel coordinates inside the overlapping area of ​​the two images.

[0054] The cumulative cost is calculated using a two-row ping-pong row buffer architecture, requiring only the previous row of data to calculate the current row, and H×W overlap The global storage requirement is reduced to 2×W overlap State transition formula: In the formula, C(u,v) is the cumulative cost of pixel (u,v).

[0055] During the calculation process, the preceding direction D(u,v) of the optimal path is recorded using 2-bit encoding. This preceding direction is the direction pointer. The direction pointers of all pixels form a direction pointer matrix, which in turn forms a preceding direction pointer lookup table. The entire direction matrix of the overlapping area is stored in the on-chip static storage array, without needing to access the off-chip storage medium.

[0056] After the forward calculation is completed, backtracking is initiated from the minimum cumulative cost point in the bottom row of the overlapping region. The direction pointers in the predecessor direction pointer lookup table are queried row by row upwards, and the seam coordinates are updated accordingly. , In the formula, v: the local row number of the current row of the overlapping region coordinates; v-1: The local line number of the previous line when backtracking upwards; u opt (v): Local column coordinates corresponding to the optimal seam in row v; u opt (v-1): The coordinates of the optimal seam column in the previous row, calculated upwards; D(u opt ,v): Coordinates (u opt The 2-bit preceding direction offset code stored at position v) has a value representing three offsets: left, down, and right. It is used to correct the position of the previous row seam column; finally, a binary seam mask is generated.

[0057] 5. Full-flow multi-scale adaptive fusion Construct a Gaussian pyramid from the binary seam mask to obtain the fusion weights W for each level. l (u,v) enables adaptive fusion with narrow transitions in high frequencies and smooth transitions in low frequencies; Laplacian pyramid decomposition is performed on the geometrically corrected video stream, and the fusion output of the l-th layer uses an algebraic optimization formula, requiring only one DSP multiplier per channel: In the formula, L A,l L B,l These represent the Laplacian high-frequency details of the l-th layer of the left and right images, respectively. u, v: Local integer pixel row and column coordinates within the overlapping region; l: Current decomposition level number of the Gaussian / Laplace pyramid; W l (u,v): The fusion weight coefficient corresponding to the local coordinates (u,v) of the l-th level, which is generated by Gaussian filtering from the binary stitch mask; L A,l : Image channel A at coordinates (u,v), the Laplacian high-frequency detail component of the l-th layer; L B,l : Channel B image at coordinates (u,v), the l-th layer Laplacian high-frequency detail component; F l (u,v): High-frequency detail output values ​​after current level fusion.

[0058] A fixed-length delay-compensated FIFO is used to align the timing of each frequency band, ensuring zero spatial coordinate deviation; during the reconstruction stage, upsampling is performed step by step, and the reconstruction formula is as follows: In the formula, u, v: Local integer pixel row and column coordinates within the overlapping region; l: The current level of the reconstructed pyramid; l+1 is the next level with a higher resolution. R l+1 The next layer (higher level) of reconstructed image data; Upsample(·): Upsampling operation, padding adjacent pixels and lines with zeros, and doubling the image width and height; Blur(·): 5×Gaussian smoothing filter, eliminating block distortion caused by zero padding; Blur(Upsample(R l+1 (u,v): The pixel value at coordinates (u,v) after upsampling and Gaussian smoothing; Coefficient 4: The compensation zero-placing operation causes an overall decrease in image brightness energy; F l (u,v): High-frequency detail components obtained by weighted fusion in this layer; R l (u,v): The reconstructed output pixel value after overlaying the low-frequency upsampled image with details at this level.

[0059] Each reconstruction output undergoes bit-width expansion and saturation clamping to prevent pixel overflow, ultimately outputting a panoramic stitched video stream. Upsampling (also known as interpolation dilation) is implemented in a hardware-friendly manner using zero-value interpolation combined with Gaussian blur: first, zero values ​​are inserted between adjacent pixels and adjacent rows to double the image resolution; then, after smoothing with a 5×5 Gaussian kernel, the image is multiplied by a gain factor of 4 to compensate for the energy attenuation caused by zero insertion. This can be achieved through a fully pipelined row buffer without the need for full-frame buffering, and is used for hierarchical resolution alignment during the Laplacian pyramid decomposition and reconstruction stages.

[0060] Example 2: Four-way fisheye panoramic surround view Application scenario: In-vehicle surround view (AVM) scenario, a single FPGA connects to four ultra-wide-angle fisheye cameras, outputting a top-down panoramic image. The input resolution of each of the four fisheye cameras is W. f ×H f The output panoramic resolution is (W p ×H p ).

[0061] System Architecture: The FPGA integrates four parallel preprocessing and geometry correction links, four parallel stitching fusion engines, and external storage media. Specific Implementation Steps: 1. Four-channel global frame synchronization After being parsed by the front-end interface, the four fisheye video streams are written into four cross-clock domain asynchronous ping-pong buffer units respectively. The global scheduling controller polls the frame start flag and data depth of the four buffers. When all four streams meet the frame start condition and store at least one line of valid pixels, the four read enable is pulled high synchronously, and the four synchronous video streams with strict pixel-by-pixel beat alignment are output, absorbing the phase drift and line offset of the four video streams.

[0062] 2. Four-way parallel photometric equalization and geometric correction The four synchronous video streams are each entered into an independent preprocessing pipeline to complete de-mosaicing, color space conversion and luminance equalization compensation. The luminance equalization is implemented in the same way as the corresponding module in Example 1, resulting in images with consistent brightness and chrominance across the four streams. On-chip configuration of four-channel homography matrix register set H k (k=0,1,2,3) correspond to the top-view inverse projection parameters of the four fisheye cameras respectively; the four inverse projection engines work in parallel, and each one adopts an inline polynomial recursion and bilinear interpolation architecture to complete geometric correction and top-view projection, and output the top-view images after geometric correction of the four channels.

[0063] 3. Multi-region parallel predictive memory access The geometric correction of the four images all adopts a block predictive burst memory access mechanism. Each channel corresponds to an independent PATRM control logic, which initiates burst read requests to the external storage medium in parallel. The on-chip storage matrix is ​​divided into four independent partitions according to the channel. After each channel's data is written to the corresponding partition, interpolation operations are performed in parallel. The four-channel memory access and calculation are completely parallel and conflict-free.

[0064] 4. Parallel seam search in four overlapping areas Four overlapping regions are formed between adjacent channels of the four top-view images. Each overlapping region corresponds to an independent dynamic programming stitching search engine, which performs multi-feature cost calculation and state transition in parallel. The direction pointer matrices for the four overlapping regions are stored in four independent static storage partitions on-chip, each with a storage capacity of W. overlap ×H overlap ×2bit; After the forward computation is completed, the four backtracking state machines work in parallel to synchronously generate four binary stitching masks.

[0065] 5. Multi-regional and multi-scale fusion and panoramic stitching The four stitching seam masks are each fed into an independent Gaussian pyramid decomposition pipeline, which simultaneously generates fusion weights for each level; the four top view images are each fed into the corresponding Laplacian pyramid decomposition pipeline, where weighted fusion is performed in the overlapping area at each level. The fusion implementation method is the same as the corresponding module in Example 1. The four-way fused images are stitched together according to a preset panoramic stitching topology. After edge completion of the stitching boundary pixels, they are processed by uniform saturation clamping to output a complete top-down panoramic image.

[0066] Example 3: Multi-resolution dynamic topology stitching Application scenarios: Dynamic splicing scenarios with variable channel count, resolution, and projection curvature, supporting online parameter configuration without recompiling FPGA logic, and supporting a channel count range of [N]. min N max Resolution supported range [W] min ×H min W max ×H max ].

[0067] System Architecture: FPGA internally integrates a configurable parameter register set and adaptive processing link, supporting interference-free parameter updates during frame blanking. Specific Implementation Steps: 1. Double-buffered configurable register set The chip is equipped with an independent parameter register group, which adopts a double-buffered architecture of working register + shadow register. The working register group is used for pipelined processing of the current frame, and the shadow register group is used to receive new configuration parameters. The register group includes channel enable register, homography matrix register group, block size configuration register, pyramid decomposition level register, motion detection threshold register, and fusion weight configuration register. All registers support online read and write via bus interface.

[0068] 2. Uninterrupted parameter update during frame blanking period After the new configuration parameters are written to the shadow register group, the parameter update logic waits for the frame end flag to be triggered. During the frame blanking period, all parameters in the shadow register group are synchronized to the working register group at once, and a parameter recalculation trigger signal is sent to each processing module. All parameter update operations are completed during the frame blanking period without interfering with the pipeline processing of the current frame, and the output video timing remains continuous and uninterrupted.

[0069] 3. Adaptive parameter reconfiguration for each module Inverse projection module: Based on the updated homography matrix, recalculate the initial values ​​of the column head numerator and denominator of each row and the in-row increment, and update the initial value of the recursion register; Predictive memory access module: Based on the updated block size and homography matrix, recalculate the projection coordinate mapping logic of the four corner points of the block, adjust the parameter range of the burst length calculation, and synchronously adjust the read and write address mapping of the on-chip memory matrix; Dynamic programming module: Adjusts the row buffer depth and the address range of the direction pointer storage based on the updated overlap width; Multi-scale fusion module: Adjusts the line buffer depth and delay compensation FIFO depth according to the updated resolution and pyramid level to ensure timing alignment of each frequency band.

[0070] 4. Dynamic scheduling of channel numbers When the channel number configuration changes, the channel enable register controls the clock gating and data path shutdown of unused channels to reduce power consumption; at the same time, it automatically adjusts the address mapping logic of the stitching topology, increases or decreases the number of overlapping areas and the workload of the stitching engine; the channel number adjustment is also completed within the frame blanking period without the need to reset the system.

[0071] Example 4: High Dynamic Motion Scene Stitching Application scenarios: In splicing scenarios with dense moving targets (such as traffic intersection monitoring, high-speed industrial assembly lines), it enables real-time avoidance of moving targets at the seams to prevent image breakage.

[0072] System Architecture: The FPGA integrates a motion detection pipeline, a high-weight dynamic programming engine, and an adaptive fusion link, supporting updates to the stitching path every frame. Specific Implementation Steps: 1. Parallel motion detection pipeline Each video channel is equipped with an independent motion detection row buffer array to store the luminance data of the corresponding row of the previous frame. When a pixel of the current frame flows in, it is synchronously differentially calculated with the luminance value of the corresponding position in the previous frame, and the absolute value is then compared with the programmable motion threshold T. m Compare and output a single-channel motion marker signal; Within the overlapping area, the motion marker signals of the two channels are ANDed: when both channels detect motion, the motion detection cost at that position is set to the maximum value of 1, otherwise it is set to 0; the motion detection cost, color difference cost, and gradient magnitude cost are calculated in parallel, and the total energy cost is obtained by weighted fusion using a hardware multiply-accumulate tree.

[0073] 2. High-weighted motion cost configuration By configuring registers to increase the weight coefficient γ of motion detection cost, the total energy cost of the moving region is much higher than that of the non-moving region, forcing the optimal path of dynamic programming to automatically bypass the moving target; the cost of the moving region is saturated to ensure that it does not overflow the bit width range of the accumulated cost.

[0074] 3. Low-latency dynamic programming and real-time seam updating The cumulative cost is stored using a two-row ping-pong row buffer architecture, and only two rows of buffers are needed to complete the forward state transition. The state transition logic is the same as the corresponding module in Example 1. The optimal forward direction of each pixel is encoded with 2 bits, and the entire direction matrix of the overlapping area is stored in the on-chip block storage without accessing the off-chip storage. After the forward calculation is completed, backtracking is started from the minimum cumulative cost point of the bottom row of the overlapping area. The dedicated state machine controls the direction pointer to be queried and the stitching coordinates to be updated row by row. The stitching path of the entire overlapping area can be generated within dozens of clock cycles, and a binary stitching mask is output. The stitching update frequency is completely consistent with the input frame rate, and the stitching path can be dynamically adjusted according to the position of the moving target in each frame.

[0075] 4. Motion-adaptive multi-scale fusion The dynamically generated binary seam mask is directly fed into the Gaussian pyramid decomposition pipeline, working in parallel with the image Laplacian decomposition pipeline to automatically generate adaptive fusion weights for each frequency band; a fixed-length delay-compensated FIFO ensures that the seam mask and image data are perfectly aligned in time, avoiding fusion misalignment in moving regions; The fusion and reconstruction logic is the same as the module corresponding to Implementation Example 1, and the panoramic video stream is output after saturation clamping processing.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A video stitching and fusion method based on FPGA hardware acceleration, characterized in that, Includes the following steps: Perform pixel-by-pixel beat alignment on the multi-channel synchronous video streams of each video channel; Pixel-level photometric equalization is performed on the aligned multi-channel synchronous video streams to generate a photometric consistent video stream, which is then written to an off-chip storage medium. The photometric consistent video stream is subjected to inverse projection recursion. Based on the target block spatial prediction addressing envelope, the off-chip storage medium is controlled to perform continuous long burst reads. After being transferred to the on-chip storage matrix, the geometrically corrected video stream is obtained through parallel interpolation. Obtain the binary stitching mask, perform multi-scale frequency decomposition and adaptive weighted fusion on the binary stitching mask and the geometrically corrected video stream, and output the panoramic stitched video stream after delay compensation and saturation clamp reconstruction.

2. The method according to claim 1, characterized in that, The pixel-by-pixel beat alignment employs cross-clock domain frame synchronization processing: The received multi-channel heterogeneous video data sources are written into the cross-clock domain asynchronous ping-pong buffer units corresponding to each video channel under the drive of the corresponding in-channel recovery clock. Under the system's master clock, the frame start characteristic status of each of the cross-clock domain asynchronous ping-pong buffer units is continuously monitored; When the frame start feature is detected to have arrived in the cross-clock domain asynchronous ping-pong buffer unit of all video channels, the read enable signal of all cross-clock domain asynchronous ping-pong buffer units is pulled high synchronously, and the pixel data stream is read synchronously to complete the cross-clock domain alignment.

3. The method according to claim 1, characterized in that, The pixel-level photometric equalization specifically includes: Global photometric statistical features of multiple synchronous video streams are extracted, and photometric gain compensation coefficients are calculated in combination with historical frame information. Based on the photometric gain compensation coefficients, pixel scaling and constant offset operations are performed on the multiple synchronous video streams to output photometric consistent video streams and write them to off-chip storage media. The global photometric statistical features include the total brightness value and average absolute deviation value of each video channel, and the historical frame information includes the reference mean and reference deviation of the reference video channel.

4. The method according to claim 3, characterized in that, The extraction process of the global photometric statistical features specifically includes: The multi-channel synchronous video streams are processed through a hardware sliding window to perform de-mosaic processing, reconstruct red, green, and blue color images, and convert them into a luminance-chrominance separation format through color space transformation; within the current frame, the pixel luminance components of each video channel are dynamically calculated pixel by pixel to obtain the total luminance value of the current frame; the absolute value of the difference between the pixel luminance components and the average value of the latched previous frame is calculated synchronously to generate an average absolute deviation value.

5. The method according to claim 3, characterized in that, The calculation process of the photometric gain compensation coefficient specifically includes: Using the pre-configured reference video channel in the multi-channel synchronous video stream as the global reference channel, the reference mean and reference deviation of the global reference channel are extracted as the historical frame information; during the frame blanking period, the target parameters of each video channel are subjected to time-axis smoothing iteration through a first-order recursive smoothing filter to calculate the photometric gain compensation coefficient of the corresponding video channel.

6. The method according to claim 1, characterized in that, The reverse projection recursion specifically includes: When generating a panoramic spatial image of the target using raster scanning, the initial values ​​of the numerator and denominator of the first pixel in the column are pre-calculated at the beginning of each row scan. While sequentially traversing pixels in the current row, the step register is used to increment the step size of the numerator and denominator of the first pixel in the column, and the numerator and denominator of the next pixel are dynamically recursively output. The calculated resampling mapping coordinates are divided into an integer part and a fractional part. The integer part is used to address the off-chip storage medium, and the fractional part is used to participate in the interpolation weight calculation.

7. The method according to claim 1, characterized in that, The process of generating the target block spatial prediction addressing envelope specifically includes: The target panoramic space is divided into regularly arranged two-dimensional blocks according to a preset fixed size; When processing the current block space, the coordinates of the four corner points of the block space are read, and the projection floating-point coordinates of the four corner points in the source image of the corresponding video channel are mapped using the pre-stored inverse transformation matrix. The maximum and minimum boundaries of the projected floating-point coordinates in the source image are obtained by using a hardware comparator array, and the spatial prediction addressing envelope of the target block is determined.

8. The method according to claim 1, characterized in that, The continuous long burst read and transfer to the on-chip memory matrix specifically includes: The burst data length is dynamically calculated based on the horizontal width of the target block space prediction address envelope, and the off-chip storage medium is controlled to perform continuous long burst reads with physical address row alignment. The chip is configured with a first storage region and a second storage region to form the on-chip storage matrix; the first storage region receives data from the current block space through the continuous long burst read, and the second storage region synchronously outputs data from the previous block space to perform data reading and interpolation operations in parallel.

9. The method according to claim 1, characterized in that, The parallel interpolation specifically includes: The on-chip storage matrix is ​​divided into a preset number of storage blocks, and a two-dimensional interleaving addressing mechanism is used to map the low-order bits of the physical addresses of adjacent pixels to different storage blocks; The neighboring pixels required for interpolation are extracted in parallel without conflict within a single master clock cycle using a multiplexer, and the interpolation is completed by combining the fractional part of the resampled mapped coordinates as weighting coefficients, and the geometrically corrected video stream is output.

10. The method according to claim 1, characterized in that, The process of obtaining the binary seam mask specifically includes: During the forward state transition calculation, two sets of single-row-depth temporary buffers are used to form a row buffer structure, and the cumulative path cost of adjacent rows is read and written alternately, thus limiting the size of the cumulative cost cache to the width of a single row pixel. During the calculation process, a two-bit encoded record is used to generate the precursor direction with the minimum cumulative cost value. A direction pointer matrix is ​​constructed and stored in the on-chip static storage array to complete the compressed storage of the precursor direction pointer. The direction pointer matrix constitutes the precursor direction pointer lookup table. The horizontal coordinate with the lowest cumulative cost is retrieved as the starting point for track finding. The row number is decremented in reverse, and the corresponding preceding direction pointer is obtained by looking up the preceding direction pointer lookup table to generate the binary stitching mask.

11. The method according to claim 1, characterized in that, The multi-scale frequency decomposition and adaptive weighted fusion specifically include: The binary stitching mask is fed into a low-pass filter pipeline corresponding to the Gaussian filter in the image space, and the required fusion weight matrix for each scale level is adaptively decomposed and constructed. At each level of detail extraction path, a corresponding depth of synchronization buffer unit is configured to absorb and process the beat difference, so as to achieve zero-deviation timing alignment of each frequency band component in spatial physical coordinates and complete the delay compensation. During the cascaded aggregation reconstruction, the pixel values ​​are expanded to a higher bit width to accommodate intermediate accumulated results without overflow, and threshold boundary saturation clamping is performed at the output of each aggregation reconstruction stage through pure combinational logic, ultimately outputting the panoramic stitched video stream.

12. A video stitching and fusion system based on FPGA hardware acceleration, characterized in that, include: Multiple video input interfaces are used to connect to multiple heterogeneous video data sources; An external storage medium is used to store video frame data; an FPGA chip is electrically connected to both the multi-channel video input interface and the external storage medium, and the FPGA chip is internally configured with: The frame synchronization unit is used to perform cross-clock domain frame synchronization processing on the multi-channel heterogeneous video data source and output a multi-channel synchronized video stream with pixel-by-pixel beat alignment. A photometric equalization compensation unit is used to perform pixel-level photometric equalization on the aligned multi-channel synchronous video streams, generate a photometric consistent video stream, and write it to an off-chip storage medium. The predictive geometric correction unit is used to perform inverse projection recursion on the photometric consistent video stream, control the off-chip storage medium to perform continuous long burst reads based on the target block spatial prediction addressing envelope, and output the geometrically corrected video stream through parallel interpolation; An adaptive stitching fusion unit is used to obtain a binary stitching mask and perform multi-scale frequency decomposition and adaptive weighted fusion on the geometrically corrected video stream to output a panoramic stitched video stream.

Images