Embedded real-time video stitching method based on dynamic suture line optimization and time domain smoothing
By using an embedded real-time video stitching method with dynamic stitch optimization and temporal smoothing, the problems of limited computing resources, temporal instability, and dynamic ghosting in video stitching technology are solved, achieving higher real-time performance and visual quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2026-01-05
- Publication Date
- 2026-04-21
AI Technical Summary
Existing video stitching technologies face challenges in terms of limited computing resources, temporal instability, and dynamic ghosting, making it difficult to achieve a balance between real-time performance and visual quality.
An embedded real-time video stitching method with dynamic stitch optimization and temporal smoothing is adopted. Through multi-threaded synchronous acquisition, robust estimation, dynamic programming search and multi-band fusion, the computational load and image jitter are reduced, and dynamic ghosting is minimized.
It achieves higher real-time performance, temporal stability, and reduced ghosting in dynamic scenes, with natural seam transitions, thus improving the overall effect of video stitching.
Smart Images
Figure CN121907972A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and image processing technology, and in particular relates to an embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing. Background Technology
[0002] In applications such as panoramic surveillance, real-time video stitching is a core technology. However, existing stitching technologies face challenges during their development. First, computational resources are limited: frame-by-frame feature matching leads to high CPU usage, making it difficult to achieve real-time frame rates. Second, temporal instability exists because the calculated homography matrix fluctuates due to ambient light or camera tremors, causing video flickering and jitter. Finally, due to parallax caused by the physical distance between binocular cameras, noticeable "double reflections" or seams easily appear in overlapping areas, and traditional weighted averaging algorithms cannot simultaneously eliminate ghosting and preserve detail. Therefore, a stitching solution is needed that can simultaneously improve the stability of homography estimation, reduce dynamic ghosting, and ensure embedded real-time performance. Summary of the Invention
[0003] The purpose of this invention is to solve the problems mentioned in the background art and to propose a video stitching scheme that balances computational efficiency and visual quality, so as to reduce feature calculation overhead, improve the stability of homography estimation, and reduce ghosting and seam artifacts in overlapping areas with dynamic targets.
[0004] To achieve the objectives of this invention, a real-time embedded video stitching method based on dynamic stitching optimization and temporal smoothing is disclosed, comprising the following steps:
[0005] Step 1: Perform multi-threaded synchronous acquisition and preprocessing of the left and right video streams;
[0006] Step 2: Extract fast local features on keyframes and match them. Obtain the initial homography matrix through robust estimation. Then, normalize the homography matrix and use exponential moving average for temporal smoothing. Combine this with an asymmetric update strategy to reuse or limit updates when the matching quality is insufficient, thereby suppressing image jitter.
[0007] Step 3: Construct an energy map containing grayscale difference and gradient difference within the overlapping region ROI. Use dynamic programming to search for the optimal vertical stitch line with the minimum cumulative cost from top to bottom, so as to bypass the moving target and reduce dynamic ghosting.
[0008] Step 4: Generate hard and soft masks in the neighborhood of the suture line, which are used for the high-frequency layer and low-frequency layer of multi-band fusion, respectively. In the mixing band, perform high gradient side forced retention at the position where the pixel difference exceeds the threshold. Finally, only construct the Laplacian pyramid near the overlapping area and directly copy the non-overlapping area to reduce the amount of computation.
[0009] Step 5: Output of spliced frames; Output the merged spliced frames to the display module, and repeat steps 1 to 5 for subsequent frames to achieve continuous real-time splicing.
[0010] Furthermore, in step 1, multi-threading technology is used to simultaneously acquire video frames from the left and right USB cameras; the original frames are scaled proportionally according to a preset height and converted into a unified format; the candidate ROI range of the overlapping area is determined according to the camera mounting method, which is used for subsequent suture line search and local fusion.
[0011] Further, in step 2, temporally stable homography matrix estimation is performed. When the keyframe condition is met, local features are extracted from the left and right video frames and described and matched. The local features are described using the SURF algorithm, without calculating the principal rotation direction to reduce computational load. Then, candidate matching point pairs are obtained by K-nearest neighbor matching (KNN) and ratio filtering, and robust estimation is performed using RANSAC to obtain the initial homography matrix. When the number of matching points, reprojection error, or RANSAC confidence does not meet the threshold condition, the homography matrix of the current frame is not updated or is limited to be updated, and the homography matrix of the previous frame is directly reused to suppress image jitter caused by instantaneous abnormal estimation. Finally, an exponential moving average is applied to the normalized homography matrix to obtain a smoothing matrix, and the exponential moving average formula is applied. The coefficients are adjusted to suppress the instantaneous fluctuation of the homography matrix in the video stream.
[0012] Furthermore, the keyframe triggering conditions need to include a set fixed frame and changes in the motion amplitude or number of points within the matching frame exceeding a set threshold; the obtained local features are accelerated SURF features without calculating the main rotation direction, and the matching adopts K-nearest neighbor matching combined with ratio testing to screen out mismatched point pairs; the matching quality evaluation indicators include the number of points within the matching frame, the proportion of points within the matching frame, the reprojection error, or the RANSAC confidence level. When any indicator does not meet the threshold condition, the asymmetric update strategy is to reuse the homography matrix of the previous frame or limit the update increment.
[0013] Furthermore, in step 3, dynamic programming is used to search for the optimal vertical stitching line. One side of the image is transformed to a common coordinate system to obtain an aligned image. An energy map is calculated in the overlapping ROI region. The energy consists of gray-level difference and gradient difference terms. Then, dynamic programming is used to search for a path with the minimum cumulative cost from the upper boundary to the lower boundary in the energy map, which is taken as the optimal vertical stitching line. The path preferably avoids areas with large gradient differences, thus tending to bypass moving objects and obvious parallax edges to reduce dynamic ghosting.
[0014] Furthermore, the cumulative cost of dynamically programming to search for the optimal vertical suture line is satisfied, and the optimal vertical suture line is obtained by backtracking from the bottom.
[0015] Furthermore, the constructed energy map includes grayscale difference terms and gradient difference terms.
[0016] Further, in step 4, local multi-band fusion is completed by first generating dual masks, including hard and soft masks. The hard mask is used for the high-frequency layer of multi-band fusion to prevent semi-transparent ghosting that could cause high-frequency double images. The soft mask is used for the low-frequency layer to achieve a smooth transition between brightness and color. Laplacian pyramids are constructed only in the overlapping ROI or the neighborhood of the seam line, and hierarchical fusion is performed: the high-frequency layer uses hard mask fusion, and the low-frequency layer uses soft mask fusion. Pixel differences are detected within the mixing band. When the difference exceeds a threshold, the pixel with the higher gradient magnitude or its high-frequency component is forcibly selected to improve edge sharpness and suppress ghosting. For non-overlapping areas, memory copy stitching is performed directly without constructing a pyramid, thereby reducing the overall computational load and improving real-time performance.
[0017] Furthermore, the width of the blending band on both sides of the stitching line of the soft mask, and the soft mask weight gradually change in a linear or cosine manner along the lateral direction. When the change exceeds a preset threshold, the pixel or high-frequency component on the side with the higher gradient magnitude is forcibly selected for fusion.
[0018] Compared with existing technologies, the significant advancements of this invention are: 1) Better real-time performance: Employing keyframe-triggered feature calculation, direct memory copying of non-overlapping regions, and pyramid fusion only locally in overlapping ROIs, significantly reducing overall computational load and memory bandwidth consumption; 2) More stable temporal domain: Avoiding abnormal homography updates through matching quality evaluation and asymmetric update strategies, and using exponential moving averages to smooth the homography matrix in the temporal domain, reducing image jitter and flicker; 3) Fewer ghosting in dynamic scenes: Through dynamic programming seam line search based on energy maps, the seam lines tend to avoid moving targets and edges with strong parallax, reducing dynamic ghosting; 4) More natural seam transitions: Using hard and soft dual masks combined with multi-band fusion, balancing clear high-frequency details with smooth transitions in low-frequency brightness and color; combined with a high-gradient side forced retention strategy, ghosting is suppressed in areas of large difference while maintaining sharp edges.
[0019] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description
[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:
[0021] Figure 1 This is a flowchart illustrating a specific method for embedded real-time video stitching based on dynamic stitch optimization and temporal smoothing. Detailed Implementation
[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] An embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing includes the following steps:
[0024] Step 1: Perform multi-threaded synchronous acquisition and preprocessing of the left and right video streams;
[0025] Step 2: Extract fast local features on keyframes and match them. Obtain the initial homography matrix through robust estimation. Then, normalize the homography matrix and use exponential moving average for temporal smoothing. Combine this with an asymmetric update strategy to reuse or limit updates when the matching quality is insufficient, thereby suppressing image jitter.
[0026] Step 3: Construct an energy map containing grayscale difference and gradient difference within the overlapping region ROI. Use dynamic programming to search for the optimal vertical stitch line with the minimum cumulative cost from top to bottom, so as to bypass the moving target and reduce dynamic ghosting.
[0027] Step 4: Generate hard and soft masks in the neighborhood of the suture line, which are used for the high-frequency layer and low-frequency layer of multi-band fusion, respectively. In the mixing band, perform high gradient side forced retention at the position where the pixel difference exceeds the threshold. Finally, only construct the Laplacian pyramid near the overlapping area and directly copy the non-overlapping area to reduce the amount of computation.
[0028] Step 5: Output of spliced frames; Output the merged spliced frames to the display module, and repeat steps 1 to 5 for subsequent frames to achieve continuous real-time splicing.
[0029] Specifically, in step 1, multi-threading technology is used to synchronously acquire video frames from the left and right USB cameras; the original frames are scaled proportionally according to the preset height and converted into a uniform format; the candidate ROI range of the overlapping area is determined according to the camera mounting method, which is used for subsequent suture line search and local fusion.
[0030] Specifically, in step 2, temporally stable homography matrix estimation is performed. When the keyframe condition is met, local features are extracted from the left and right video frames and described and matched. The local features are described using the SURF algorithm, without calculating the principal rotation direction to reduce computational load. Then, candidate matching point pairs are obtained by K-nearest neighbor matching (KNN) and ratio filtering, and robust estimation is performed using RANSAC to obtain the initial homography matrix. When the number of matching points, reprojection error, or RANSAC confidence does not meet the threshold condition, the homography matrix of the current frame is not updated or is limited to be updated, and the homography matrix of the previous frame is directly reused to suppress image jitter caused by instantaneous abnormal estimation. Finally, an exponential moving average is applied to the normalized homography matrix to obtain a smoothing matrix, and the exponential moving average formula is applied. The coefficients are adjusted to suppress the instantaneous fluctuation of the homography matrix in the video stream.
[0031] Specifically, the keyframe triggering conditions need to include a set fixed frame and changes in the motion amplitude or number of points within the matching frame exceeding a set threshold; the obtained local features are accelerated SURF features without calculating the main rotation direction, and the matching uses K-nearest neighbor matching combined with ratio testing to filter out mismatched point pairs; the matching quality evaluation indicators include the number of points within the matching frame, the proportion of points within the matching frame, the reprojection error, or the RANSAC confidence score. When any indicator does not meet the threshold condition, the asymmetric update strategy is to reuse the homography matrix of the previous frame or limit the update increment.
[0032] Specifically, in step 3, dynamic programming is used to search for the optimal vertical stitching line. One side of the image is transformed to a common coordinate system to obtain an aligned image. The energy map is calculated in the overlapping ROI region. The energy consists of gray-level difference and gradient difference. Then, dynamic programming is used to search for a path with the minimum cumulative cost from the upper boundary to the lower boundary in the energy map, which is the optimal vertical stitching line. The path preferably avoids areas with large gradient differences, thus tending to bypass moving objects and obvious parallax edges to reduce dynamic ghosting.
[0033] Specifically, the cumulative cost of dynamically programming to search for the optimal vertical suture line is satisfied, and the optimal vertical suture line is obtained by backtracking from the bottom.
[0034] Specifically, the constructed energy map includes gray-level difference terms and gradient difference terms.
[0035] Specifically, in step 4, local multi-band fusion is completed by first generating a dual mask, including a hard mask and a soft mask. The hard mask is used for the high-frequency layer of multi-band fusion to prevent semi-transparent ghosting that could cause high-frequency double images. The soft mask is used for the low-frequency layer to achieve a smooth transition between brightness and color. Laplacian pyramids are constructed only in the overlapping ROI or the neighborhood of the seam line, and hierarchical fusion is performed: the high-frequency layer uses hard mask fusion, and the low-frequency layer uses soft mask fusion. Pixel differences are detected within the mixing band. When the difference exceeds a threshold, the pixel with the higher gradient magnitude or its high-frequency component is forcibly selected to improve edge sharpness and suppress ghosting. For non-overlapping areas, memory copy stitching is performed directly without constructing a pyramid, thereby reducing the overall computational load and improving real-time performance.
[0036] Specifically, the width of the blending band on both sides of the stitching line of the soft mask, and the soft mask weight gradually change in a linear or cosine manner along the lateral direction. When the change exceeds a preset threshold, the pixel or high-frequency component on the side with the higher gradient magnitude is forcibly selected for fusion.
[0037] Example
[0038] In this embodiment, the hardware implementation includes: two USB cameras, a PC containing a CPU and a GPU processor, and the software processing modules include an acquisition synchronization module, a homography estimation and temporal smoothing module, a dynamic suture line search module, a local multi-band fusion module, and an output module. Figure 1 As shown, the steps for executing the method on the processor are as follows:
[0039] Step 1: Synchronous acquisition and preprocessing of video stream.
[0040] First, separate acquisition threads are created for the left and right cameras. A double-buffered queue is used to write the acquired frames and timestamps into the buffer. Then, the header frames of the two buffers are read. If the reading time difference is very small, the synchronization is considered successful and the frame pair is output. Otherwise, the frame with the earlier timestamp is discarded or the later frame is waited for to ensure the time consistency of the input frame pair. Second, the frames are scaled proportionally according to the preset height and the image format is unified. Finally, based on the camera installation method and calibration results, the candidate ROI regions of the overlapping areas of the left and right fields of view in the common coordinate system are determined for subsequent stitching line search and local fusion.
[0041] Step 2: Calculate the time-domain stable input image homography matrix estimation and smoothing.
[0042] First, feature extraction and matching are triggered according to the set fixed frame interval. Fast local features are extracted from overlapping ROIs on keyframes. The algorithm does not calculate the main rotation direction to reduce overhead. Then, K-nearest neighbor (KNN) matching is used to obtain candidate matching point pairs, and ratio test is used to eliminate false matches. Next, the RANSAC algorithm is used to robustly estimate the candidate matching point pairs to obtain the initial homography matrix. The number of inliers, the proportion of inliers, and the average reprojection error are counted. Based on whether the above three data meet the set threshold conditions, it is determined whether to directly reuse the homography matrix of the previous frame to avoid instantaneous abnormal splicing. Finally, homography matrix normalization and exponential moving average (EMA) are used for temporal smoothing.
[0043] Step 3: Dynamic suture line search.
[0044] First, using the smooth homography matrix obtained in step two, the perspective transformation of the right image is transformed to a common coordinate system to obtain an aligned image. Second, an energy map consisting of gray-level difference and gradient difference terms is constructed within the overlapping ROI. Finally, dynamic programming is used to accumulate costs from top to bottom to obtain a cumulative cost map. The minimum value is taken at the boundary according to the available neighborhood. Finally, the position with the minimum cumulative cost is taken at the bottom and backtracked to obtain the optimal vertical stitch line from top to bottom.
[0045] Step 4: Local multi-band fusion and forced retention of high gradient side.
[0046] First, a blending band is generated on both sides of the optimal seam line. A hard mask is generated outside the blending band, and a soft mask is generated inside the blending band. Then, Laplacian pyramids and Gaussian pyramids of the soft and hard masks are constructed only in the overlapping ROI or the neighborhood of the blending band. High-frequency layers are fused using hard masks to avoid high-frequency semi-transparent ghosting, while low-frequency layers are fused using soft masks to achieve smooth brightness and color transitions. Second, the pixel differences between the two sides are detected within the blending band. When the difference is greater than a preset threshold, the pixel with the higher gradient magnitude or its high-frequency component is selected to participate in the fusion to maintain clear edges and suppress ghosting. Finally, non-overlapping areas are directly copied and stitched in memory without constructing a pyramid to reduce computation and latency.
[0047] Step 5: Outputting the spliced frames.
[0048] The merged spliced frame is output to the display module, and steps one to five are repeated for subsequent frames to achieve continuous real-time splicing.
[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. An embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing, characterized in that, Includes the following steps: Step 1: Perform multi-threaded synchronous acquisition and preprocessing of the left and right video streams; Step 2: Extract fast local features on keyframes and match them. Obtain the initial homography matrix through robust estimation. Then, normalize the homography matrix and use exponential moving average for temporal smoothing. Combine this with an asymmetric update strategy to reuse or limit updates when the matching quality is insufficient, thereby suppressing image jitter. Step 3: Construct an energy map containing grayscale difference and gradient difference within the overlapping region ROI. Use dynamic programming to search for the optimal vertical stitch line with the minimum cumulative cost from top to bottom, so as to bypass the moving target and reduce dynamic ghosting. Step 4: Generate hard and soft masks in the neighborhood of the suture line, which are used for the high-frequency layer and low-frequency layer of multi-band fusion, respectively. In the mixing band, perform high gradient forced retention at the position where the pixel difference exceeds the threshold. Finally, only construct the Laplacian pyramid near the overlapping area and directly copy the non-overlapping area to reduce the amount of computation. Step 5: Output of spliced frames; Output the merged spliced frames to the display module, and repeat steps 1 to 5 for subsequent frames to achieve continuous real-time splicing.
2. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 1, characterized in that, In step 1, multi-threading technology is used to synchronously acquire video frames from the left and right USB cameras; the original frames are scaled proportionally according to the preset height and converted into a uniform format; the candidate ROI range of the overlapping area is determined according to the camera mounting method, which is used for subsequent suture line search and local fusion.
3. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 1, characterized in that, In step 2, temporally stable homography matrix estimation is performed. When the keyframe condition is met, local features are extracted from the left and right video frames and described and matched. The local features are described using the SURF algorithm, without calculating the principal rotation direction to reduce computational load. Then, candidate matching point pairs are obtained by K-nearest neighbor matching (KNN) and ratio filtering, and robust estimation is performed using RANSAC to obtain the initial homography matrix. When the number of matching points, reprojection error, or RANSAC confidence does not meet the threshold condition, the homography matrix of the current frame is not updated or is limited to be updated, and the homography matrix of the previous frame is directly reused to suppress image jitter caused by transient abnormal estimation. Finally, an exponential moving average is applied to the normalized homography matrix to obtain a smoothed matrix, and the exponential moving average formula is applied. The coefficients are adjusted to suppress the transient fluctuations of the homography matrix in the video stream.
4. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 3, characterized in that, The keyframe triggering conditions need to include a set fixed frame and changes in the motion amplitude or number of points within the matching frame exceeding a set threshold. The obtained local features are accelerated SURF features without calculating the main rotation direction. Matching uses K-nearest neighbor matching combined with ratio testing to filter out mismatched point pairs. Matching quality evaluation indicators include the number of points within the matching frame, the proportion of points within the matching frame, reprojection error, or RANSAC confidence. When any indicator does not meet the threshold condition, the asymmetric update strategy is to reuse the homography matrix of the previous frame or limit the update increment.
5. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 1, characterized in that, In step 3, dynamic programming is used to search for the optimal vertical stitching line. One side of the image is transformed to a common coordinate system to obtain an aligned image. The energy map is calculated in the overlapping ROI region. The energy consists of gray-level difference and gradient difference. Then, dynamic programming is used to search for a path with the minimum cumulative cost from the upper boundary to the lower boundary in the energy map, which is the optimal vertical stitching line. The path preferably avoids areas with large gradient differences, thus tending to bypass moving objects and obvious parallax edges to reduce dynamic ghosting.
6. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 5, characterized in that, The cumulative cost of dynamically programming to search for the optimal vertical suture line is satisfied, and the optimal vertical suture line is obtained by backtracking from the bottom.
7. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 5, characterized in that, The constructed energy map includes grayscale difference terms and gradient difference terms.
8. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 1, characterized in that, In step 4, local multi-band fusion is completed. First, dual masks are generated, including hard and soft masks. The hard mask is used for the high-frequency layer of multi-band fusion to prevent semi-transparent ghosting that causes high-frequency double images. The soft mask is used for the low-frequency layer to achieve a smooth transition between brightness and color. Laplacian pyramids are constructed only in the overlapping ROI or the neighborhood of the seam line and hierarchical fusion is performed: the high-frequency layer uses hard mask fusion, and the low-frequency layer uses soft mask fusion. Pixel differences are detected within the mixing band. When the difference exceeds a threshold, the pixel with the higher gradient magnitude or its high-frequency component is forcibly selected to improve edge sharpness and suppress ghosting. For non-overlapping areas, memory copy stitching is performed directly without constructing a pyramid, thereby reducing the overall computational load and improving real-time performance.
9. The embedded real-time video stitching method based on dynamic stitching optimization and temporal smoothing according to claim 8, characterized in that, The width of the blending band on both sides of the stitching line of the soft mask, the soft mask weight gradually changes in a linear or cosine manner along the lateral direction, and when the change exceeds the preset threshold, the pixel or high-frequency component on the side with the higher gradient magnitude is forcibly selected for fusion.