Method for realizing real-time image splicing system based on SIFT (Scale Invariant Feature Transform) algorithm
By improving the SIFT algorithm for feature point extraction, fast matching, and hardware acceleration, the problems of insufficient real-time performance and robustness in image stitching in existing technologies are solved, achieving efficient and real-time image stitching results.
Patent Information
- Application Number
- CN202511599883.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-04
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies struggle to achieve efficient, real-time image stitching in complex scenarios, especially on resource-constrained embedded platforms. Furthermore, the traditional SIFT algorithm has high computational complexity, making it difficult to meet real-time requirements.
By employing an improved SIFT algorithm for feature point extraction, KD-Tree-based fast nearest neighbor matching, perspective transformation hardware IP core, and fade-in/fade-out weighted fusion technology, combined with FPGA hardware acceleration and host computer collaborative processing, efficient image stitching is achieved.
It achieves efficient and real-time image stitching in complex scenarios, generating visually seamless and naturally transitioning stitched images, thus improving the system's computational efficiency and robustness.
Smart Images
Figure CN121504718A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to image stitching in the field of image processing, and more specifically, to the implementation of a real-time image stitching system based on the SIFT algorithm. Background Technology
[0002] In the fields of computer vision and image processing, image stitching technology has always been a hot research topic. Its goal is to synthesize multiple images with overlapping areas into a single, high-resolution image with a wider field of view. Traditional image stitching methods mainly rely on feature point matching and spatial transformation models, such as algorithms based on corner detection or template matching. However, these methods often perform poorly when dealing with large-scale changes in viewpoint, lighting differences, or complex scenes, easily resulting in mismatches and ghosting phenomena. Furthermore, their computational efficiency is low, making it difficult to meet real-time requirements. Early systems often used features such as Harris corners for matching, lacking scale and rotation invariance, which limited their applicability in dynamic environments.
[0003] With the development of feature extraction algorithms, Scale Invariant Feature Transform (SIFT), as a local feature descriptor with scale, rotation, and illumination invariance, has significantly improved the robustness and accuracy of image matching. The SIFT algorithm detects extreme points by constructing a difference-of-Gaussian pyramid and generates high-dimensional feature vectors for similarity calculation, effectively handling image scaling, rotation, and some affine transformations, laying the foundation for high-quality image stitching. However, the classic SIFT algorithm has high computational complexity, and the feature generation and matching processes are time-consuming, making it difficult to directly apply to real-time systems, especially in resource-constrained embedded platforms or high frame rate video stream processing, where its performance bottleneck is particularly prominent.
[0004] In recent years, the demand for real-time image processing has been growing, especially in fields such as drone aerial photography, virtual reality, intelligent monitoring, and medical imaging, which place higher demands on high-speed, high-precision image stitching technology. Although existing research has attempted to optimize the SIFT process through GPU parallel computing, FPGA hardware acceleration, or feature compression algorithms, shortcomings remain in overall system architecture, resource scheduling, and accuracy balance. Existing technologies often struggle to balance speed and accuracy, or suffer from poor stability in complex scenarios, lacking a complete solution that can efficiently integrate feature extraction, rapid matching, and seamless fusion. Therefore, there is an urgent need for a real-time image stitching system based on an improved SIFT algorithm to overcome the technical contradiction between computational efficiency and robustness. Summary of the Invention
[0005] The purpose of this invention is to provide a method for implementing a real-time image stitching system based on the SIFT algorithm, thereby achieving an image stitching system with good stitching effect and high real-time performance.
[0006] The technical solution block diagram proposed in this invention is as follows: Figure 1 As shown, the detailed technical solution is explained below:
[0007] (1) Feature point extraction using SIFT algorithm
[0008] A six-layer Gaussian pyramid was constructed, with each layer of Gaussian image using an independent Gaussian filter kernel of size 21*21. The filter kernel coefficients were calculated offline, and the Gaussian blur coefficients were 1.52, 2^(1 / 3)*1.52, 2^(2 / 3)*1.52, 2^(3 / 3)*1.52, 2^(4 / 3)*1.52, and 2^(5 / 3)*1.52.
[0009] A 20-row, contiguous shift register is used to store 20 rows of pixels, outputting 21 rows of pixels in the same column. The pixel values of each row are buffered through 20 registers, resulting in a 21-row * 21-column filtering window. Then, a three-stage pipelined multiply-accumulator calculates the Gaussian filtering result, such as... Figure 2 As shown.
[0010] To avoid consuming floating-point operation resources, the coefficients actually used are 1024 times the coefficients of the filter kernel, and the result is shifted 10 bits to the right after participating in the calculation.
[0011] The difference between two adjacent Gaussian pyramids is calculated pixel by pixel to obtain a five-layer difference pyramid, and the difference result is a signed number. Extreme points are found in the three adjacent difference pyramids. The relationship between each pixel and its 26 neighboring pixels is determined. If the center pixel and its neighboring pixels satisfy the following relationship, then the point is considered an extreme point.
[0012]
[0013] The thresholding method improves the stability of feature points while consuming fewer resources and avoiding the use of Taylor expansion for image fitting.
[0014] Since a threshold is already applied during extreme point detection, there is no need to spend a lot of DSP resources to calculate weak response points, but edge response points still need to be considered. The second-order Hessian matrix is obtained using the central difference method:
[0015]
[0016] Calculate the trace Tr(H) and determinant Det(H) of the Hessian matrix:
[0017] Tr(H)=D xx +D yy ,
[0018] By discriminatively removing edge response points, key points that satisfy the following conditions are retained (with r set to 10):
[0019] r*Tr(H) 2 <(r+1) 2 *Det(H)
[0020] (2) SIFT algorithm descriptor generation
[0021] The gradient direction and magnitude are calculated using the CORDIC (Coordinate Rotation Digital Computation) method. The gradient direction is quantized into eight intervals (each interval is 45°). The cumulative value of the magnitude of the eight gradient directions within each 4x4 square of a 16x16 region surrounding the current pixel is calculated. Figure 3 As shown, the magnitudes of the 16 squares in eight directions constitute the 128-dimensional descriptor of the center pixel.
[0022] To ensure illumination invariance, the descriptor needs to be normalized. First, the amplitudes in each direction of the descriptor are summed. Conventional methods require dividing the amplitude in each direction by the total amplitude, necessitating the use of 128 dividers. The calculation is shown in the following formula, where N is the original descriptor, M is the normalized descriptor, and the normalization range is 0 to 255.
[0023]
[0024] To reduce the use of dividers and lower resource consumption, the above formula is broken down as follows: only one divider is used to calculate DIV and use it as the coefficient of the magnitude in each direction; the remaining operations are implemented through shifting and multiplication:
[0025]
[0026] (3) Fast nearest neighbor matching
[0027] To address the computational bottleneck of high-dimensional feature descriptor matching, this invention employs a KD-Tree-based approximate nearest neighbor search library (such as FLANN) to accelerate the matching process. Specifically, the feature descriptor set of the reference image is constructed as a KD-Tree index structure. For each feature descriptor in the image to be registered, its two nearest neighbors are quickly found in this tree, and a ratio test is applied to filter reliable preliminary matching pairs by comparing the distance ratio between the nearest and second nearest neighbors. To further eliminate false matches, the system then uses the Random Sample Consensus Algorithm (RANSAC) to robustly estimate the preliminary matching pairs, solve for the optimal homography matrix, and filter out outliers, ultimately obtaining a set of accurate and stable feature point correspondences. This module is mainly implemented using Python or C++ to meet real-time requirements while ensuring accuracy.
[0028] (4) Perspective transformation
[0029] In the perspective transformation stage, the system uses the homography matrix calculated from the matching point pairs to project the image to be registered onto the coordinate system of the reference image. To achieve extremely high computational efficiency, this stage employs High-Level Synthesis (HLS) technology. The system uses the HLS library to transform the core algorithm of perspective transformation into a high-performance hardware IP core. This IP core receives the pixel stream and homography matrix parameters and uses inverse mapping technology to calculate the corresponding coordinates of each pixel in the original image for each pixel in the output image. To address the issue of non-integer coordinates generated by inverse mapping, this module performs bilinear interpolation in parallel to generate a smooth transformed image. This hardware acceleration method significantly improves the speed of geometric correction and provides a crucial guarantee for real-time processing pipelines.
[0030] (5) Image fusion
[0031] Finally, to eliminate brightness differences and visible seams in the overlapping areas, this invention employs a fade-in / fade-out weighted blending method, implemented on an FPGA using the Verilog hardware description language. This method generates a weight value for each pixel in the overlapping area that is proportional to its distance to the boundary of its respective non-overlapping area, such as... Figure 4 As shown, the fusion module, acting as a pipeline unit, synchronously receives pixel streams and corresponding weights from two images. Using built-in multipliers and adders, it completes the weighted calculation of "weight A × pixel A + weight B × pixel B" within one clock cycle. For non-overlapping areas, it directly outputs the original image pixels. This hardware-implemented fusion strategy can generate a visually seamless and naturally transitioning final stitched image in real time, completely eliminating the latency issues caused by software fusion. Attached Figure Description
[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0033] Figure 1 This is a schematic diagram of the real-time image stitching system based on the SIFT algorithm of this invention.
[0034] Figure 2 This is a schematic diagram of the hardware structure of the Gaussian filter core of this invention.
[0035] Figure 3 This is a schematic diagram of the SIFT descriptor histogram statistics of the present invention.
[0036] Figure 4 This is a schematic diagram of image fusion according to the present invention. Detailed Implementation
[0037] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.
[0038] Example 1
[0039] The present invention discloses a real-time image stitching system based on the SIFT algorithm, the specific implementation process of which is as follows: The system uses a Xilinx Kintex-7 series FPGA as the core hardware processing platform, responsible for hardware acceleration of all computationally intensive tasks; while the PC host computer software developed based on Python or C++ is responsible for executing complex algorithms such as feature matching and transformation matrix calculation. The two communicate at high speed via a gigabit Ethernet interface using the UDP protocol.
[0040] In implementation, the compiled hardware logic bitstream file is first configured into the Kintex FPGA, and the image sensor interface and the three-speed Ethernet MAC IP core are initialized. Subsequently, the video stream from the dual image sensors is fed into the SIFT feature extraction pipeline within the FPGA. This pipeline uses a 20-row deep shift register group to construct a 21×21 pixel window and calls Gaussian kernel coefficients pre-stored in Block RAM and processed by a 1024x fixed-point scaling factor. A six-layer Gaussian pyramid is efficiently constructed through a three-stage pipelined multiply-accumulator. Adjacent Gaussian layers are immediately subtracted to generate a five-layer DOG pyramid. The extremum detection module synchronously compares each pixel with its 26 neighbors through a 3×3×3 cubic buffer and, combined with a set contrast threshold, initially filters out stable feature points.
[0041] To eliminate edge responses, subsequent modules use the central difference method to calculate the Hessian matrix of each candidate point and filter them based on the discriminant of its trace and determinant. For the finally confirmed feature points, the descriptor generation module uses the CORDIC algorithm to calculate the gradient magnitude and direction of its neighboring pixels, and divides the 16×16 neighborhood into 4×4 squares. Within each square, the gradient direction is quantized into 8 intervals for histogram statistics, forming a 128-dimensional descriptor. To reduce resource consumption, the normalization process of the descriptor is cleverly decomposed by calculating a common reciprocal coefficient and then multiplying it by the magnitude of each dimension, thereby reducing the required 128 dividers to 1.
[0042] After hardware feature extraction is completed, the FPGA packages the descriptor data and corresponding feature point coordinates of the two images into Ethernet frames via the UDP protocol and sends them to the host computer. Upon receiving the data, the host computer software performs fast nearest neighbor matching: first, it uses the FLANN library to construct a KD-Tree index for the reference image descriptors; then, it finds two nearest neighbors for each descriptor of the image to be registered, applies a ratio test to filter out reliable preliminary matching pairs, and finally uses the RANSAC algorithm to remove false matches and solve for the exact homography matrix between the two images.
[0043] The calculated homography matrix parameters are then sent to the FPGA via another UDP packet. Upon receiving the transformation matrix, the FPGA feeds it into a perspective transformation IP core implemented using Vivado HLS. This IP core employs inverse mapping and bilinear interpolation algorithms to perform real-time geometric correction on the input pixel stream of the image to be registered, projecting it onto the coordinate system of the reference image.
[0044] Finally, the corrected image and the reference image are fed into an image fusion module implemented in Verilog. This module generates a weight map for overlapping regions that is proportional to the distance from the pixel to the non-overlapping boundary. Through pipelined multipliers and adders, it achieves fade-in / fade-out fusion in the form of "weight A × pixel A + weight B × pixel B". For non-overlapping regions, pixels are directly passed through, ultimately generating a seamless stitched result and outputting it to the display device. The entire system achieves end-to-end hardware acceleration and high real-time performance from image input to stitched display through collaborative processing between the FPGA and the host computer.
[0045] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for implementing a real-time image stitching system based on the SIFT algorithm, characterized in that, Includes the following steps: SIFT feature point extraction steps: By constructing a Gaussian pyramid and a difference pyramid, and performing extreme point detection and edge response point removal in the difference pyramid, stable image feature points are extracted. SIFT descriptor generation steps: Calculate the gradient information of pixels in the neighborhood of the feature point to generate a feature descriptor with illumination invariance; Fast nearest neighbor matching steps: On the PC host computer, by constructing a KD-Tree index and applying ratio test and RANSAC algorithm, the feature descriptors of two images are matched quickly and robustly. Perspective transformation steps: Based on the homography matrix obtained by matching, the image is geometrically corrected by inverse mapping and bilinear interpolation using a hardware IP core implemented on the FPGA using HLS. Image fusion steps: On the FPGA, a fade-in / fade-out weighted fusion method is used to perform pixel-level weighted averaging on the overlapping areas to generate a seamless stitched image.
2. The method according to claim 1, characterized in that, The SIFT feature point extraction step specifically includes: constructing a set of six-layer Gaussian pyramids, where each layer of the image is convolved using a 21*21 Gaussian filter kernel, and the filter kernel coefficients are fixed-point processed; subtracting adjacent Gaussian images to obtain a five-layer difference pyramid; in the difference pyramids of adjacent three layers, comparing each pixel with its 26 neighboring pixels, and combining with a preset contrast threshold to detect extreme points; calculating the Hessian matrix of the extreme points, and removing edge response points based on the discriminant formed by its trace and determinant, finally determining stable feature points.
3. The method according to claim 1, characterized in that, The SIFT descriptor generation steps specifically include: calculating the gradient magnitude and direction of pixels in the neighborhood of the feature point using the CORDIC algorithm; quantizing the gradient direction into 8 directional intervals, and calculating the cumulative sum of the gradient magnitudes of the 8 directions in each 4*4 sub-region within a 16*16 pixel area around the feature point to form a 128-dimensional feature descriptor; and normalizing the 128-dimensional feature descriptor by calculating a common reciprocal coefficient and then using multiplication and shift operations to complete the normalization calculation of all dimensions.
4. The method according to claim 1, characterized in that, The fast nearest neighbor matching step is implemented in a PC using Python or C++, and specifically includes: constructing a KD-Tree index structure from the feature descriptor set of the reference image; for each feature descriptor of the image to be registered, searching for its two nearest neighbors in the KD-Tree, and selecting preliminary matching pairs using the ratio test method; applying the RANSAC algorithm to robustly estimate the preliminary matching pairs, solving for the optimal homography matrix, and filtering out mismatched point pairs.
5. The method according to claim 1, characterized in that, The perspective transformation step specifically involves: using the VivadoHLS tool to encapsulate the perspective transformation algorithm into a hardware IP core; the IP core receives homography matrix parameters and image pixel streams sent from the host computer; A reverse mapping technique is used to calculate the corresponding source coordinates in the input image for each pixel coordinate in the output image; bilinear interpolation is then performed on the source coordinates to obtain smooth pixel values, thus completing the geometric correction of the image.
6. The method according to claim 1, characterized in that, The image fusion step is implemented on the FPGA using the Verilog hardware description language. Specifically, it includes: generating a weight value for each pixel in the overlapping region of the images that is proportional to its distance to the boundary of the non-overlapping region; synchronously receiving the pixel streams and corresponding weight values of the two images through a pipelined fusion unit; completing the weighted calculation of "weight A × pixel A + weight B × pixel B" within one clock cycle and outputting the fused pixel value; and directly outputting the original image pixels for pixels in the non-overlapping region.
7. A real-time image stitching system for implementing the method according to any one of claims 1 to 6, characterized in that, include: The FPGA hardware platform is configured to perform SIFT feature point extraction, SIFT descriptor generation, perspective transformation, and image fusion steps. The PC host computer communicates with the FPGA hardware platform via Ethernet UDP protocol. It is configured to receive feature descriptor data from the FPGA, perform a fast nearest neighbor matching step, and send the calculated homography matrix back to the FPGA.