Efficient real-time video stitching method
By using an image stitching network that combines multi-scale feature extraction and feature fusion with homography and TPS transform, the contradiction between accuracy and speed in video stitching is resolved, achieving efficient and real-time video stitching and ensuring the continuity and natural transition of the video.
Patent Information
- Application Number
- CN202411139385.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-20
AI Technical Summary
Existing image stitching methods cannot simultaneously achieve high precision and high speed in video stitching, especially when processing long videos, which can easily lead to problems such as image misalignment, distortion, and video discontinuity.
An image stitching network employing multi-scale feature extraction and feature fusion, combined with homography and TPS transform, achieves accurate image alignment and stitching through a YOLOv8 backbone network and a regression network. A stable reference coordinate system is selected, and the stitching speed is accelerated through local region fusion.
It improves the accuracy and speed of image stitching, ensures the continuity and natural transition of video, avoids problems such as image misalignment and video discontinuity, and meets the needs of real-time video.
Smart Images

Figure CN118660118B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method for splicing images into a video. Background Art
[0002] In the field of low-altitude remote sensing, due to the limitations of drone flight altitude and viewing angle, it is often necessary to stitch multiple images into a complete large picture to achieve an overall observation of the area, and then stitch the images into a video.
[0003] Image alignment, parallax processing, illumination change adaptation, and distortion issues need to be considered in the image stitching process. Existing image stitching methods mainly include feature matching, transformation model estimation, and image fusion. However, in the video stitching process, traditional image stitching methods cannot meet the requirements of high precision and stitching speed. Especially when processing long videos, problems such as image misalignment, distortion, and video incoherence often occur. Summary of the invention
[0004] The present invention proposes an efficient real-time video stitching method, the purpose of which is to solve the problem that accuracy and speed cannot be taken into account in the image stitching process.
[0005] The technical solution of the present invention is as follows:
[0006] An efficient real-time video stitching method, comprising the following steps:
[0007] Step 1: stitch the target image and the reference image at the same time into a complete image frame in real time through an image stitching network;
[0008] Step 2: stitch the complete image frames at different times into a video in chronological order;
[0009] The processing process of the image stitching network in step 1 is: firstly, multi-scale feature extraction is performed on the target image and the reference image respectively to obtain the first scale feature map and the second scale feature map of the target image and the reference image; then, the homography transformation matrix is obtained according to the fused feature map of the first scale feature map of the target image and the first scale feature map of the reference image. , and use the homography transformation matrix The second scale feature map of the target image is homographically transformed to achieve preliminary alignment; then, the control point offset value residual of the homograph transformation is obtained based on the fused feature map of the second scale feature map of the target image after homograph transformation and the second scale feature map of the reference image, and the homograph transformation matrix is converted into The grid control points applied to the TPS transformation obtain the new coordinates of the grid control points of the TPS transformation; then the TPS transformation is solved according to the control point offset value residual and the new coordinates of the grid control points of the TPS transformation, and the TPS transformation is applied to the target image; finally, the target image that has completed the TPS transformation is spliced with the reference image into a complete image frame.
[0010] Further improvement of the efficient real-time video stitching method: the image stitching network includes two weight-shared YOLOv8 backbone networks, and also includes a first regression network and a second regression network.
[0011] A further improvement of the efficient real-time video stitching method is that the processing process of the image stitching network is:
[0012] Step 1-1, the target image and the reference image are respectively input into the corresponding YOLOv8 backbone network to obtain their respective first-scale feature maps and second-scale feature maps;
[0013] Step 1-2: The first scale feature map of the target image and the first scale feature map of the reference image Fusion is performed to obtain fusion features :
[0014] ;
[0015] in, Indicates feature fusion;
[0016] Steps 1-3: Based on fusion features Get the homography transformation matrix :
[0017] ;
[0018] ;
[0019] in, represents the operation of the first regression network, Represents the coordinate offset values of the four corner points of the image, Indicates based on Solving the homography transformation matrix ;
[0020] Step 1-4: Use the homography transformation matrix The second scale feature map of the target image Perform homography transformation to obtain feature map :
[0021] ;
[0022] in, Indicates homography transformation;
[0023] Steps 1-5: Based on the feature map The second size feature map with the reference image Fusion feature map Get the control point offset value residual of the homography transformation :
[0024] ;
[0025] ;
[0026] in, represents the operation of the second regression network;
[0027] Step 1-6: Transform the homography matrix Mesh control points used for TPS transformation Get the new coordinates of the grid control points of the TPS transformation , and then based on the control point offset residual and the new coordinates of the grid control points after TPS transformation Solve the TPS transformation:
[0028] ;
[0029] ;
[0030] in, Represents the grid control points for TPS transformation The parameters are The homography coordinate mapping of Indicates solving TPS transformation;
[0031] Step 1-7: Apply the TPS transformation to the target image, and then splice the target image after the TPS transformation with the reference image into a complete image frame.
[0032] Further improvement of the efficient real-time video stitching method, step 2 specifically includes:
[0033] Step 2-1, selecting a complete image frame from all complete image frames as a reference frame, taking other complete image frames as target frames, and taking the coordinate system of the reference frame as the reference coordinate system for video stitching;
[0034] Step 2-2, calculating the transformation matrix of each target frame to the reference coordinate system, and then each target frame completes the transformation to the reference coordinate system according to the transformation matrix;
[0035] Step 2-3: Obtain the corresponding local area of each target frame according to its transformation matrix, and then fuse the local areas of all target frames with the reference frame in time sequence to obtain the video.
[0036] Further improvement of the efficient real-time video stitching method, the reference coordinate system selection method in step 2-1 is specifically as follows:
[0037] First calculate the distortion values for all complete image frames:
[0038] ;
[0039] in, For the The distortion value of a complete image frame, is the number of complete image frames, For the The image area of a complete image frame, For the The complete image frame is transformed into The image area after the coordinate system of a complete image frame;
[0040] Then The complete image frame corresponding to the minimum value of the distortion values is used as the reference frame, and the coordinate system of the reference frame is used as the reference coordinate system for video stitching.
[0041] Further improvement of the efficient real-time video stitching method, in step 2-2, assuming that the reference frame is Full image frames , for any target frame , first calculate and The intermediate transformation matrices between all adjacent frames are then divided into arrive In the direction of Towards The transformation matrix .
[0042] Further improvement of the efficient real-time video stitching method, in step 2-3, for any target frame ,set up:
[0043] ;
[0044] in, and They are the target frames after transformation to the reference coordinate system. The width and height of ; then:
[0045] , , , , ;
[0046] and The target frame The coordinates of the upper left corner and lower right corner of the rectangular local area that will participate in the fusion.
[0047] Further improvement of the efficient real-time video stitching method: the image stitching and video stitching are completed by a stitching head using unsupervised learning.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] 1. The present invention realizes the fusion of precise local features and abstract global features through multi-scale feature extraction and feature fusion, which can better handle the local transformation and overall transformation trend of pixel details in splicing, improve the accuracy of image splicing, avoid complex feature fusion strategies, reduce memory consumption, and bring about speed improvement.
[0050] 2. The present invention proposes a simple and efficient distortion criterion when selecting a reference coordinate system, which can more stably select the optimal reference system, ensure that each frame of video is spliced under optimal conditions, and improve the stability and continuity of the splicing effect.
[0051] 3. The present invention realizes fast transformation alignment between frames by calculating the intermediate transformation matrix between adjacent frames, which significantly speeds up the splicing processing speed and ensures that the needs of real-time video can be met.
[0052] 4. The present invention reduces time complexity and improves fusion speed by means of local area fusion. Moreover, this method realizes high-quality seamless splicing, ensures the continuity and natural transition of video playback, and avoids problems such as image dislocation, distortion and video discontinuity. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 Schematic diagram of the structure of the image stitching network in the present invention. DETAILED DESCRIPTION
[0054] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:
[0055] An efficient real-time video stitching method, comprising the following steps:
[0056] Step 1: The target image and the reference image at the same moment are stitched into a complete image frame in real time through an image stitching network.
[0057] like Figure 1, the processing process of the image stitching network is: firstly, multi-scale feature extraction is performed on the target image and the reference image respectively to obtain the first scale feature map and the second scale feature map of the target image and the reference image. Then, the homography transformation matrix is obtained based on the fusion feature map of the first scale feature map of the target image and the first scale feature map of the reference image. , and use the homography transformation matrix The second scale feature map of the target image is homographically transformed to achieve preliminary alignment. Then, the control point offset value residual of the homograph transformation is obtained based on the fused feature map of the second scale feature map of the target image after homograph transformation and the second scale feature map of the reference image, and the homograph transformation matrix is converted into The grid control points applied to the TPS transformation obtain the new coordinates of the grid control points of the TPS transformation. Then the TPS transformation is solved based on the control point offset value residual and the new coordinates of the grid control points of the TPS transformation, and the TPS transformation is applied to the target image. Finally, the target image that has completed the TPS transformation is spliced with the reference image into a complete image frame.
[0058] Specifically, the image stitching network includes two weight-sharing YOLOv8 backbone networks, a first regression network and a second regression network. The processing process of the image stitching network is:
[0059] Step 1-1, the target image and the reference image are respectively input into the corresponding YOLOv8 backbone network to obtain their respective first scale feature maps and second scale feature maps. In this embodiment, the size of the first scale feature map is 1 / 16 of the original image size, and the size of the second scale feature map is 1 / 8 of the original image size.
[0060] Step 1-2: The first scale feature map of the target image and the first scale feature map of the reference image Fusion is performed to obtain fusion features :
[0061] ;
[0062] in, Indicates feature fusion.
[0063] Steps 1-3: Based on fusion features Get the homography transformation matrix :
[0064] ;
[0065] ;
[0066] in, represents the operation of the first regression network, Represents the coordinate offset values of the four corner points of the image, Indicates based on Solving the homography transformation matrix .
[0067] Step 1-4: Use the homography transformation matrix The second scale feature map of the target image Perform homography transformation to obtain feature map , to achieve preliminary alignment of the target image and the reference image:
[0068] ;
[0069] in, Indicates homography transformation.
[0070] Homography can describe the projection relationship between two planes and is applicable to the case where the optical center of the camera remains unchanged and the scene being photographed is a plane. This step uses homography to perform preliminary alignment, laying the foundation for subsequent fine-tuning.
[0071] Steps 1-5: Based on the feature map The second size feature map with the reference image Fusion feature map Get the control point offset value residual of the homography transformation :
[0072] ;
[0073] ;
[0074] in, Represents the operation of the second regression network.
[0075] Step 1-6: Transform the homography matrix Mesh control points used for TPS transformation Get the new coordinates of the grid control points of the TPS transformation , and then based on the control point offset residual and the new coordinates of the grid control points after TPS transformation Solve the TPS transformation:
[0076] ;
[0077] ;
[0078] in, Represents the grid control points for TPS transformation The parameters are The homography coordinate mapping of Indicates solving the TPS transformation.
[0079] Step 1-7: Apply the TPS transformation to the target image, and then splice the target image after the TPS transformation with the reference image into a complete image frame.
[0080] This image stitching method can make the control point offset value residual Focus on the difference after homography transformation, and correct the homography transformation on the feature map with larger spatial scale and more positioning details. Therefore, by using the calculated TPS transformation parameters to transform the target image, the shape of the image can be flexibly adjusted to make it more consistent with the reference image in details, ensuring high accuracy and natural transition of the stitching result.
[0081] Step 2: Splice the complete image frames into a video in chronological order.
[0082] Step 2 specifically includes:
[0083] Step 2-1: select a complete image frame from all complete image frames as a reference frame, use other complete image frames as target frames, and use the coordinate system of the reference frame as a reference coordinate system for video stitching.
[0084] The specific selection method is:
[0085] First calculate the distortion values for all complete image frames:
[0086] ;
[0087] in, For the The distortion value of a complete image frame, is the number of complete image frames, For the The image area of a complete image frame, For the The complete image frame is transformed into The image area after the coordinate system of a complete image frame.
[0088] Then The complete image frame corresponding to the minimum value of the distortion values is used as the reference frame, and the coordinate system of the reference frame is used as the reference coordinate system for video stitching.
[0089] Step 2-2, calculate the transformation matrix of each target frame to the reference coordinate system, and then each target frame completes the transformation to the reference coordinate system according to the transformation matrix.
[0090] Specifically, let the reference frame be Full image frames , for any target frame , first calculate and The intermediate transformation matrices between all adjacent frames are then divided into arrive In the direction of Towards The transformation matrix .
[0091] Step 2-3: Obtain the corresponding local area of each target frame according to its transformation matrix, and then fuse the local areas of all target frames with the reference frame in time sequence to obtain the video.
[0092] Specifically, for any target frame ,set up:
[0093] ;
[0094] in, and They are the target frames after transformation to the reference coordinate system. The width and height of ; then:
[0095] , , , , ;
[0096] and The target frame The coordinates of the upper left corner and lower right corner of the rectangular local area that will participate in the fusion.
[0097] The image stitching and video stitching are completed by a stitching head using unsupervised learning. The stitching head learns the geometric relationship between images directly from unlabeled data through unsupervised learning. This method does not need to rely on a large amount of manually labeled data, which fundamentally reduces the difficulty and cost of data acquisition and preparation. Through iterative design, the stitching head can accurately align images to ensure that the stitching results are highly accurate and coherent, thereby significantly improving the stitching quality. The model can automatically adapt to different types and changes of image data during training, has excellent generalization ability, and is suitable for a variety of practical application scenarios. These innovative features make this method perform well and have significant practical value in solving actual video stitching needs.
Claims
1. An efficient real-time video stitching method, comprising the steps of: Step 1: stitch the target image and the reference image at the same time into a complete image frame in real time through an image stitching network; Step 2: stitch the complete image frames at different times into a video in chronological order; Features: The processing process of the image stitching network in step 1 is: firstly, multi-scale feature extraction is performed on the target image and the reference image respectively to obtain the first scale feature map and the second scale feature map of the target image and the reference image; then, the homography transformation matrix is obtained according to the fused feature map of the first scale feature map of the target image and the first scale feature map of the reference image. , and use the homography transformation matrix The second scale feature map of the target image is homographically transformed to achieve preliminary alignment; then, the control point offset value residual of the homograph transformation is obtained based on the fused feature map of the second scale feature map of the target image after homograph transformation and the second scale feature map of the reference image, and the homograph transformation matrix is converted into The grid control points applied to the TPS transformation obtain the new coordinates of the grid control points of the TPS transformation; then the TPS transformation is solved according to the residual of the control point offset value and the new coordinates of the grid control points of the TPS transformation, and then the TPS transformation is applied to the target image; finally, the target image after the TPS transformation is spliced with the reference image into a complete image frame; Step 2 specifically includes: Step 2-1, selecting a complete image frame from all complete image frames as a reference frame, taking other complete image frames as target frames, and taking the coordinate system of the reference frame as the reference coordinate system for video stitching; The specific method for selecting the reference coordinate system in step 2-1 is: First calculate the distortion values for all complete image frames: ; in, For the The distortion value of a complete image frame, is the number of complete image frames, For the The image area of a complete image frame, For the The complete image frame is transformed into The image area after the coordinate system of a complete image frame; Then The complete image frame corresponding to the minimum value among the distortion values is used as the reference frame, and the coordinate system of the reference frame is used as the reference coordinate system for video stitching; Step 2-2, calculating the transformation matrix of each target frame to the reference coordinate system, and then each target frame completes the transformation to the reference coordinate system according to the transformation matrix; In step 2-2, let the reference frame be Full image frames , for any target frame , first calculate and The intermediate transformation matrices between all adjacent frames are then divided into arrive In the direction of Towards The transformation matrix ; Step 2-3, each target frame obtains a corresponding local area according to its transformation matrix, and then the local areas of all target frames are fused with the reference frame in time sequence to obtain a video; In step 2-3, for any target frame ,set up: ; in, and They are the target frames after transformation to the reference coordinate system. The width and height of ; then: , , , , ; and The target frame The coordinates of the upper left corner and lower right corner of the rectangular local area that will participate in the fusion.
2. The efficient real-time video stitching method according to claim 1, characterized in that: The image stitching network includes two weight-shared YOLOv8 backbone networks, and also includes a first regression network and a second regression network.
3. The efficient real-time video stitching method according to claim 2, characterized in that: The processing process of the image stitching network is: Step 1-1, the target image and the reference image are respectively input into the corresponding YOLOv8 backbone network to obtain their respective first-scale feature maps and second-scale feature maps; Step 1-2: The first scale feature map of the target image and the first scale feature map of the reference image Fusion is performed to obtain fusion features : ; in, Indicates feature fusion; Steps 1-3: Based on fusion features Get the homography transformation matrix : ; ; in, represents the operation of the first regression network, Represents the coordinate offset values of the four corner points of the image, According to Solving the homography transformation matrix ; Step 1-4: Use the homography transformation matrix The second scale feature map of the target image Perform homography transformation to obtain feature map : ; in, Indicates homography transformation; Steps 1-5: Based on the feature map The second size feature map with the reference image Fusion feature map Get the control point offset value residual of the homography transformation : ; ; in, represents the operation of the second regression network; Step 1-6: Transform the homography matrix Mesh control points used for TPS transformation Get the new coordinates of the grid control points of the TPS transformation , and then based on the control point offset residual and the new coordinates of the grid control points after TPS transformation Solve the TPS transformation: ; ; in, Represents the grid control points for TPS transformation The parameters are The homography coordinate mapping of Indicates solving TPS transformation; Step 1-7: Apply the TPS transformation to the target image, and then splice the target image after the TPS transformation with the reference image into a complete image frame.
4. The efficient real-time video stitching method according to any one of claims 1 to 3, characterized in that: The image stitching and video stitching are performed by a stitching head using unsupervised learning.
