Video stitching methods, equipment and media for array cameras

Abstract

This application provides a video stitching method, device, and medium for an array camera. The method includes acquiring panoramic video and multiple sub-videos captured by the array camera; performing motion object detection on the panoramic video and multiple sub-videos to determine the optimal moving object; obtaining the position information of each frame in the sub-video within the panoramic video based on the feature information corresponding to the optimal moving object; performing perspective transformation and image cropping on each frame in the sub-video to obtain transformed sub-images; stitching multiple sub-images based on the position information corresponding to each frame to obtain multiple stitched video streams; identifying moving objects in adjacent video streams; determining the positional deviation information of adjacent video streams based on the motion trajectory of the same moving object in the corresponding video stream; eliminating stitching misalignment between adjacent video streams based on the positional deviation information; and completing the stitching of multiple video streams. This method effectively eliminates stitching misalignment and improves stitching accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video stitching method, device and medium for an array camera. Background Technology

[0002] In the field of video stitching using array cameras, existing technologies mainly rely on feature matching algorithms for static images, employing feature detectors such as SIFT and ORB to extract key points and then stitching the images by calculating the homography matrix between adjacent images. However, such methods have significant drawbacks in dynamic scenes:

[0003] 1. Limitations of static feature matching: Traditional algorithms (such as FLANN matching based on SURF features) only utilize static features such as texture and color of a single frame image. When there are color differences or uneven brightness between different cameras, or when there are a lot of repetitive textures in the scene (such as brick walls or lawns), feature mismatch is likely to occur, leading to the accumulation of homography matrix calculation errors and causing splicing misalignment.

[0004] 2. Insufficient adaptability to dynamic scenes: Existing methods are prone to losing static feature points when objects move quickly or are occluded, causing frequent recalculation of the perspective transformation matrix and affecting the stability of real-time stitching.

[0005] 3. Lack of error correction mechanism: Although existing technologies attempt to optimize matching accuracy through feature point weighting, they lack analysis of the continuity of motion trajectory, cannot detect splicing misalignment caused by accumulated errors, and cannot achieve adaptive correction based on motion laws.

[0006] Therefore, there is an urgent need for a video stitching method to solve at least one of the above problems. Summary of the Invention

[0007] This application provides a video stitching method, device, and medium for an array camera, aiming to solve the problems in the prior art where the use of feature detectors such as SIFT and ORB to extract key points of images and to achieve stitching by calculating the homography matrix between adjacent images has limitations in static feature matching, insufficient adaptability to dynamic scenes, and lack of error correction mechanisms.

[0008] In a first aspect, this application provides a video stitching method for an array camera, including:

[0009] A panoramic video and multiple sub-videos captured by an array camera are acquired. Moving object detection is performed on the panoramic video and multiple sub-videos to acquire multiple moving objects and their corresponding feature information, so as to determine the optimal moving object among the multiple moving objects based on the feature information.

[0010] Based on the feature information corresponding to the optimal moving object, obtain the position information of each frame in the sub-video in the panoramic video;

[0011] Perspective transformation and image cropping are performed on each frame of the sub-video to obtain the transformed sub-image. Multiple frames of the sub-image are stitched together according to the position information corresponding to each frame to complete the stitching of the sub-image corresponding to each sub-video and obtain multiple stitched video streams.

[0012] Moving object recognition is performed on adjacent video streams to detect the same moving object in the adjacent video streams. Based on the motion trajectory of the same moving object in the corresponding video stream, the positional deviation information of the adjacent video streams is determined. Based on the positional deviation information, the splicing misalignment of the adjacent video streams is eliminated, and the splicing of multiple video streams is completed.

[0013] In some embodiments, the step of detecting moving objects in the panoramic video and multiple sub-videos to obtain multiple moving objects and their corresponding feature information includes: using a motion region detection algorithm based on optical flow, calculating candidate regions of candidate moving objects in the panoramic video and multiple sub-videos through inter-frame pixel displacement fields; performing color space conversion on the candidate regions to obtain color histogram features and edge gradient features corresponding to the candidate regions; using inter-frame motion vector analysis to obtain candidate regions with continuous motion trajectories as target regions; taking the candidate moving object corresponding to the target region as the moving object, and obtaining the feature information corresponding to the moving object in the target region.

[0014] In some embodiments, determining the optimal moving object among multiple moving objects based on the feature information includes: calculating the visual saliency index of each moving object and the background region, and obtaining a first candidate moving object whose saliency index is higher than a first threshold; calculating the motion trajectory smoothness of each moving object, and obtaining a second candidate moving object whose motion trajectory smoothness is lower than a second threshold; calculating the ratio coefficient of the object pixel area to the image resolution of each moving object, and obtaining a third candidate moving object whose ratio coefficient is within a preset size range; calculating the motion speed of each moving object, and obtaining a fourth candidate moving object whose motion speed is greater than a third threshold and less than a fourth threshold; and determining the optimal moving object based on the first candidate moving object, the second candidate moving object, the third candidate moving object, and the fourth candidate moving object.

[0015] In some embodiments, obtaining the position information of each frame in the sub-video within the panoramic video based on the feature information corresponding to the optimal moving object includes: performing multi-scale feature point detection on the region where the optimal moving object is located using a preset feature detector to obtain the feature information; constructing a Hamming distance-based K-nearest neighbor matching model based on the feature information to filter matching point sets where the distance ratio of matching point pairs is less than a preset ratio; iteratively optimizing the matching point set to calculate the projection error of the homography matrix corresponding to the matching point set; the projection error being less than a preset tolerance; and determining the position information of each frame in the sub-video within the panoramic video based on the projection error.

[0016] In some embodiments, performing perspective transformation and image cropping on each frame of the sub-video to obtain the transformed sub-image includes: performing perspective transformation on the image edges according to the homography matrix corresponding to each frame to generate a perspective-transformed quadrilateral region; the perspective transformation is a bilinear interpolation transformation; calculating the minimum bounding rectangle corresponding to the quadrilateral region, scaling the minimum bounding rectangle according to the target resolution ratio to obtain a mapping box; detecting invalid pixel regions at the edges of the mapping box using an adaptive thresholding method, cropping the image based on the invalid pixel regions at the edges; performing gamma correction on the cropped image to align the image's brightness histogram with the background region corresponding to the panoramic video, thereby obtaining the sub-image.

[0017] In some embodiments, the step of identifying moving objects in adjacent video streams and detecting the same moving objects corresponding to adjacent video streams includes: obtaining scale-invariant feature descriptors of moving objects in adjacent video streams; constructing a feature similarity matrix based on the scale-invariant feature descriptors; calculating a motion direction consistency index corresponding to moving objects within the same time window in adjacent video streams based on a timestamp synchronization mechanism; determining the moving object as the same moving object when both the feature similarity of the feature similarity matrix corresponding to the moving object and the motion direction consistency index exceed a preset threshold; and establishing a joint feature tracking list across video streams for the same moving object.

[0018] In some embodiments, determining the positional deviation information of adjacent video streams based on the motion trajectories of the same moving objects in the corresponding video streams includes: obtaining the motion equations of the corresponding same moving objects based on the motion trajectories; fitting the motion equations of the objects corresponding to adjacent video streams in a time-space coordinate system to obtain the difference in trajectory slope; obtaining the pixel coordinate offsets at the intersection points of the trajectories of adjacent video streams to generate a positional deviation vector field; calculating the local affine transformation parameters of the splicing boundary corresponding to adjacent video streams based on the distribution characteristics of the deviation vector field; and generating the positional deviation information based on the local affine transformation parameters.

[0019] In some embodiments, the method further includes: optimizing the geometric consistency of video stitching by modeling the multi-view geometry of the array camera and analyzing the position and posture of moving objects under different viewpoints.

[0020] Secondly, embodiments of this application provide a computer device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the video stitching method as provided in any embodiment of this application.

[0021] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the video stitching method provided in any embodiment of this application.

[0022] This application provides a video stitching method, apparatus, device, and storage medium. The provided video stitching method primarily addresses the limitations of existing technologies that use feature detectors such as SIFT and ORB to extract key points from images and then stitch them together by calculating the homography matrix between adjacent images. These limitations include static feature matching constraints, insufficient adaptability to dynamic scenes, and a lack of error correction mechanisms. The specific method includes the following steps:

[0023] First, acquire panoramic video and multiple sub-videos captured by the array camera.

[0024] Then, moving object detection is performed on the panoramic video and multiple sub-videos to obtain multiple moving objects and their corresponding feature information. Based on the feature information, the optimal moving object is determined among the multiple moving objects. Based on the feature information corresponding to the optimal moving object, the position information of each frame in the sub-video within the panoramic video is obtained.

[0025] Next, perspective transformation and image cropping are performed on each frame of the sub-video to obtain the transformed sub-image. Based on the positional information corresponding to each frame, multiple sub-image frames are stitched together to complete the stitching of the sub-images corresponding to each sub-video, resulting in multiple stitched video streams.

[0026] Finally, moving object recognition is performed on adjacent video streams to detect the same moving objects in adjacent video streams. Based on the motion trajectories of the same moving objects in the corresponding video streams, the positional deviation information of the adjacent video streams is determined. The positional deviation information is used to eliminate splicing misalignments between adjacent video streams, thus completing the splicing of multiple video streams.

[0027] The provided method has at least the following beneficial effects:

[0028] 1. Improve adaptability to dynamic scenes: By detecting moving objects and determining the optimal moving object, it can better adapt to dynamic scenes and reduce stitching errors caused by dynamic changes in the scene.

[0029] 2. Enhanced error correction mechanism: By detecting the same moving objects in adjacent video streams and determining the positional deviation information based on their motion trajectory, splicing misalignment can be effectively eliminated, and splicing accuracy can be improved.

[0030] 3. Improve stitching efficiency and quality: Through perspective transformation and image cropping, sub-videos can be stitched together more quickly and accurately, generating high-quality panoramic video streams.

[0031] 4. Reduce the limitations of static feature matching: Traditional methods rely on static feature matching, while this method can overcome the limitations of static feature matching and improve the robustness of stitching by using moving object detection and feature information.

[0032] In summary, the video stitching method provided in this application significantly improves the adaptability, accuracy, and efficiency of video stitching by introducing steps such as moving object detection, optimal moving object determination, position information acquisition, image transformation and cropping, and moving object recognition and error correction, and has broad application prospects.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic flowchart illustrating the steps of a video splicing method provided in an embodiment of this application;

[0036] Figure 2 This is a schematic block diagram of the structure of a video splicing device provided in one embodiment of this application;

[0037] Figure 3 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0041] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0042] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0043] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0044] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0045] In the field of video stitching using array cameras, existing technologies mainly rely on feature matching algorithms for static images, employing feature detectors such as SIFT and ORB to extract key points and then stitching the images by calculating the homography matrix between adjacent images. However, such methods have significant drawbacks in dynamic scenes:

[0046] 1. Limitations of static feature matching: Traditional algorithms (such as FLANN matching based on SURF features) only utilize static features such as texture and color of a single frame image. When there are color differences or uneven brightness between different cameras, or when there are a lot of repetitive textures in the scene (such as brick walls or lawns), feature mismatch is likely to occur, leading to the accumulation of homography matrix calculation errors and causing splicing misalignment.

[0047] 2. Insufficient adaptability to dynamic scenes: Existing methods are prone to losing static feature points when objects move quickly or are occluded, causing frequent recalculation of the perspective transformation matrix and affecting the stability of real-time stitching.

[0048] 3. Lack of error correction mechanism: Although existing technologies attempt to optimize matching accuracy through feature point weighting, they lack analysis of the continuity of motion trajectory, cannot detect splicing misalignment caused by accumulated errors, and cannot achieve adaptive correction based on motion laws.

[0049] Therefore, there is an urgent need for a video stitching method to solve at least one of the above problems.

[0050] To solve the above problem, please refer to Figure 1 This application provides a video stitching method. The method runs on a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, laptop, wearable device, or robot, etc.

[0051] like Figure 1 As shown, the provided video stitching method includes steps S101 to S104. Details are as follows:

[0052] Step S101. Acquire panoramic video and multiple sub-videos captured by the array camera, perform moving object detection on the panoramic video and multiple sub-videos, acquire multiple moving objects and corresponding feature information, and determine the optimal moving object among the multiple moving objects based on the feature information.

[0053] Specifically, in step S101, panoramic video and multiple sub-videos captured by the array camera are first acquired. The panoramic video and multiple sub-videos are then processed using a moving object detection algorithm (such as an object detection model based on optical flow or deep learning) to extract multiple moving objects and their corresponding feature information. This feature information includes, but is not limited to, the object's shape, size, color, speed, and direction of motion.

[0054] For example, the YOLO (You Only Look Once) object detection model is used for moving object detection to perform real-time detection of moving objects in video frames. This is achieved by calculating the feature vector (such as color histogram, edge features, etc.) for each moving object and then using the K-means clustering algorithm to perform cluster analysis on all moving objects. The moving object corresponding to the cluster center is selected as the optimal moving object. The selection criteria for the optimal moving object include feature stability, motion trajectory continuity, and its visibility across multiple video frames.

[0055] Motion detection can employ the YOLOv5 model to detect moving objects in video frames in real time. This is achieved by calculating the feature vector of each moving object (e.g., color histogram, edge features), and then using the K-means clustering algorithm to cluster all moving objects. The moving object corresponding to the cluster center is selected as the optimal moving object. The selection criteria for the optimal moving object include feature stability, trajectory continuity, and visibility across multiple video frames. Specifically, feature stability is evaluated by calculating the similarity of the moving object's feature vectors across multiple frames; trajectory continuity is evaluated by calculating the smoothness of the moving object's trajectory across multiple frames; and visibility is evaluated by calculating the frequency of the moving object's appearance across multiple frames. For example: 1. Select moving objects with high saliency in the video stream (e.g., objects with significant differences in color, shape, etc. from the background); 2. Select objects with stable and predictable trajectories, such as objects moving in uniform linear motion; 3. Select objects of moderate size, as objects that are too large or too small may be difficult to track accurately; 4. Select objects with moderate speed, as objects that are too fast or too slow may affect the real-time performance and accuracy of tracking.

[0056] By tracking objects using video streams, more matching points are found in the panoramic image and sub-images over time. This avoids the problem of insufficient accurate matching feature points often found in static images (or feature matching deviations due to too many image repetitions), which leads to errors in perspective transformation matrix calculation and reduces stitching errors. By introducing moving object detection and optimal moving object selection mechanisms, this invention effectively avoids the feature mismatch problem of traditional static feature matching algorithms in dynamic scenes, significantly improving the accuracy and stability of feature matching.

[0057] Step S102. Based on the feature information corresponding to the optimal moving object, obtain the position information of each frame in the sub-video within the panoramic video.

[0058] Specifically, in step S102, the position information of each frame in the sub-video within the panoramic video is obtained based on the feature information corresponding to the optimal moving object. Specifically, the position coordinates of each frame in the panoramic video are determined by calculating the relative position of the optimal moving object in the sub-video frame and the panoramic video frame.

[0059] For example, feature point matching algorithms (such as SIFT or ORB) can be used to match feature points of the optimal moving object in sub-video frames and panoramic video frames. False matches are then eliminated using the RANSAC (Random Sample Consensus) algorithm. The positional offset of the optimal moving object between the two frames is calculated, thereby determining the positional information of the sub-video frame in the panoramic video. Alternatively, the motion speed of the moving object can be detected; if the motion is uniform, the region of the object in the next frame can be predicted, allowing for faster searching of feature points in the next frame and improving matching efficiency.

[0060] The position coordinates of the sub-video frame in the panoramic video are determined by calculating the position offset of the optimal moving object in the sub-video frame and the panoramic video frame. Specifically, the position offset is evaluated by calculating the difference in feature point coordinates of the optimal moving object in the two frames, and the position coordinates are determined by adding the coordinates of the sub-video frame to the position offset.

[0061] By introducing the position information of the optimal moving object, this invention can achieve accurate positioning of sub-video frames in panoramic video, effectively reducing the splicing misalignment problem caused by feature mismatch in traditional methods.

[0062] Step S103. Perform perspective transformation and image cropping on each frame of the sub-video to obtain the transformed sub-image. Then, stitch together multiple frames of the sub-image according to the position information corresponding to each frame to complete the stitching of the sub-image corresponding to each sub-video and obtain multiple stitched video streams.

[0063] Specifically, in step S103, perspective transformation and image cropping are performed on each frame of the sub-video to obtain the transformed sub-image. Based on the position information corresponding to each frame, multiple sub-image frames are stitched together to complete the stitching of the sub-images corresponding to each sub-video, thereby obtaining multiple stitched video streams.

[0064] Perspective transformation is calculated using the homography matrix. Using the known positional information of the sub-video frames within the panoramic video, the perspective transformation matrix from each frame to the panoramic image is calculated. Then, perspective transformation and cropping are performed on each frame to remove invalid regions at the image edges. Finally, multiple sub-frames are stitched together according to their positional information to generate a stitched video stream. Specifically, the homography matrix is ​​evaluated by calculating the correspondence between the feature point coordinates in the sub-video frames and the panoramic video frames, while the perspective transformation matrix is ​​determined by multiplying the coordinates of the sub-video frames by the homography matrix. Perspective transformation and cropping are performed on each frame to remove invalid regions at the image edges. Specifically, image cropping is determined by calculating the boundary of the perspective-transformed image, and invalid regions are evaluated by calculating the difference between the image boundary and the panoramic video boundary.

[0065] For example, obtain the Homography matrix of each frame image; according to the order of the image sequence, perform perspective transformation on the images based on the Homography matrix to obtain the perspective-transformed images; crop the rectangular area of ​​each sub-image by removing the surrounding 5 pixels, and extract the final transformed sub-image image from the perspective-transformed image based on the cropped rectangular area of ​​the sub-image; stitch the sub-image images of each frame according to their position in the panoramic video image.

[0066] By introducing perspective transformation and image cropping mechanisms, this invention can achieve seamless stitching of sub-video frames and panoramic videos, effectively improving the visual effect and stability of the stitched video.

[0067] Step S104. Perform moving object recognition on adjacent video streams, detect the same moving objects corresponding to adjacent video streams, determine the positional deviation information of adjacent video streams based on the motion trajectory of the same moving objects in the corresponding video streams, eliminate the splicing misalignment of adjacent video streams based on the positional deviation information, and complete the splicing of multiple video streams.

[0068] Specifically, in step S104, moving object recognition is performed on adjacent video streams to detect the same moving objects corresponding to adjacent video streams. Based on the motion trajectory of the same moving objects in the corresponding video streams, the positional deviation information of the adjacent video streams is determined. Based on the positional deviation information, the splicing misalignment of the adjacent video streams is eliminated, and the splicing of multiple video streams is completed.

[0069] For example, moving object recognition employs an optical flow-based tracking algorithm to track identical moving objects in adjacent video streams and obtain their trajectories. By calculating the positional deviation of identical moving objects in adjacent video streams, the splicing misalignment is determined, and an interpolation algorithm is used to correct the splicing misalignment, achieving seamless video stream splicing. Specifically, the optical flow method evaluates the motion by calculating the motion vectors of pixels in adjacent video frames, and the motion trajectory is determined by calculating the position coordinates of the moving object across multiple video frames.

[0070] By calculating the positional deviation of identical moving objects in adjacent video streams, the stitching misalignment is determined, and an interpolation algorithm is used to correct the misalignment, achieving seamless stitching of the video streams. Specifically, the positional deviation is evaluated by calculating the difference in positional coordinates of identical moving objects in adjacent video streams, the stitching misalignment is determined by comparing the positional deviation with a preset threshold, and the interpolation algorithm is evaluated by calculating the pixel interpolation corresponding to the stitching misalignment.

[0071] By introducing a moving object recognition and error correction mechanism, this invention can effectively detect and eliminate splicing misalignment caused by accumulated errors, significantly improving the accuracy and stability of video splicing.

[0072] In summary, this invention provides a video stitching method based on moving object detection and trajectory analysis. By introducing mechanisms such as moving object detection, optimal moving object selection, position information acquisition, perspective transformation and image stitching, as well as moving object recognition and error correction, it effectively solves the problems of feature matching error, insufficient stitching stability, and lack of error correction mechanism in dynamic scenes in existing technologies, and significantly improves the accuracy, stability, and real-time performance of video stitching.

[0073] In some embodiments, the step of detecting moving objects in the panoramic video and multiple sub-videos to obtain multiple moving objects and their corresponding feature information includes: using a motion region detection algorithm based on optical flow, calculating candidate regions of candidate moving objects in the panoramic video and multiple sub-videos through inter-frame pixel displacement fields; performing color space conversion on the candidate regions to obtain color histogram features and edge gradient features corresponding to the candidate regions; using inter-frame motion vector analysis to obtain candidate regions with continuous motion trajectories as target regions; taking the candidate moving object corresponding to the target region as the moving object, and obtaining the feature information corresponding to the moving object in the target region.

[0074] Specifically, a motion region detection algorithm based on optical flow is used to process panoramic video and multiple sub-videos. Specifically, candidate regions for moving objects are obtained by calculating the pixel displacement field between adjacent frames. Optical flow effectively detects moving regions in the video by analyzing the motion vectors of pixels between consecutive frames.

[0075] First, the video frames are converted to grayscale. Then, the Lucas-Kanade optical flow method is used to calculate the pixel displacement field between adjacent frames. By setting a displacement threshold, pixel regions with large displacements are selected as candidate motion regions. Color space transformation is performed on the candidate regions to obtain their corresponding color histogram features and edge gradient features. Color space transformation typically involves converting from RGB space to HSV or Lab space for better color information extraction. The pixel values ​​of the candidate regions are converted from RGB space to HSV space, and color histogram features are calculated. Simultaneously, the Sobel operator is used to calculate edge gradient features to enhance the contour information of moving objects. Based on inter-frame motion vector analysis, candidate regions with continuous motion trajectories are selected as target regions. By analyzing the consistency of motion vectors, transient noise regions can be eliminated, retaining target regions with continuous motion trajectories. Motion vector tracking is performed on the candidate regions to calculate their motion trajectories in different frames. By setting a trajectory smoothness threshold, regions with continuous motion trajectories are selected as target regions. Candidate moving objects corresponding to the target regions are identified as moving objects, and their corresponding feature information is extracted from the target regions. These features include color histograms, edge gradients, motion speed, and motion direction. Feature extraction is performed on the target region, calculating the color histogram, edge gradients, motion speed, and motion direction of the moving object, and storing these features as a feature vector of the moving object.

[0076] Through the implementation of the embodiments, the present invention can effectively detect moving objects in videos and extract their corresponding feature information. The motion region detection algorithm based on optical flow can accurately identify moving objects in videos and reduce noise interference. Color space conversion and edge gradient calculation are used to extract the color and contour information of moving objects, enhancing the richness of feature description. Inter-frame motion vector analysis can filter out target regions with continuous motion trajectories, improving the stability and accuracy of moving object detection.

[0077] In some embodiments, determining the optimal moving object among multiple moving objects based on the feature information includes: calculating the visual saliency index of each moving object and the background region, and obtaining a first candidate moving object whose saliency index is higher than a first threshold; calculating the motion trajectory smoothness of each moving object, and obtaining a second candidate moving object whose motion trajectory smoothness is lower than a second threshold; calculating the ratio coefficient of the object pixel area to the image resolution of each moving object, and obtaining a third candidate moving object whose ratio coefficient is within a preset size range; calculating the motion speed of each moving object, and obtaining a fourth candidate moving object whose motion speed is greater than a third threshold and less than a fourth threshold; and determining the optimal moving object based on the first candidate moving object, the second candidate moving object, the third candidate moving object, and the fourth candidate moving object.

[0078] The visual saliency index of each moving object relative to the background region is calculated, and the first candidate moving objects with a saliency index higher than a first threshold are selected. The visual saliency index is evaluated by calculating the contrast and color difference between the moving object and the background region. A visual saliency algorithm based on the Itti-Koch model is used to calculate the saliency map of the moving object and the background region. By setting a saliency threshold, moving objects with a saliency index higher than the first threshold are selected as the first candidate moving objects.

[0079] The smoothness of the motion trajectory of each moving object is calculated, and second candidate moving objects with a smoothness lower than a second threshold are identified. Motion trajectory smoothness is evaluated by calculating the curvature of the moving object's trajectory across multiple frames of video. Curve fitting is performed on the moving object's trajectory, and its curvature is calculated. By setting a smoothness threshold, moving objects with a smoothness lower than the second threshold are selected as second candidate moving objects.

[0080] Calculate the ratio of the pixel area of ​​each moving object to the image resolution, and obtain the third candidate moving objects whose ratio falls within a preset size range. The ratio is evaluated by calculating the ratio of the pixel area of ​​the moving object to the image resolution. Calculate the pixel area of ​​the moving object and compare it with the image resolution. By setting a range for the ratio, moving objects whose ratio falls within the preset size range are selected as the third candidate moving objects.

[0081] The motion velocity of each moving object is calculated, and fourth candidate moving objects with motion velocities greater than a third threshold and less than a fourth threshold are selected. Motion velocity is evaluated by calculating the ratio of the displacement of a moving object to the time interval across multiple frames of video. The velocity of the moving object's trajectory is calculated, and by setting a velocity threshold range, moving objects with motion velocities greater than the third threshold and less than the fourth threshold are selected as fourth candidate moving objects.

[0082] The optimal moving object is determined based on the first, second, third, and fourth candidate moving objects. By comprehensively evaluating the feature information of each candidate moving object, the optimal moving object is selected for subsequent processing. A weighted score is applied to the feature information of each candidate moving object, and the moving object with the highest score is selected as the optimal moving object.

[0083] Through the implementation of the embodiments, this invention can accurately determine the optimal moving object from multiple moving objects. By calculating the visual saliency index, moving objects with high contrast to the background area are selected, improving the accuracy of feature matching. By calculating the smoothness of the motion trajectory, moving objects with smooth trajectories are selected, improving the stability of moving object detection. By calculating the ratio of object pixel area to image resolution, moving objects of reasonable size are selected, avoiding interference from excessively large or small moving objects. By calculating motion speed, moving objects with suitable speeds are selected, improving the real-time performance and accuracy of moving object detection.

[0084] In some embodiments, obtaining the position information of each frame in the sub-video within the panoramic video based on the feature information corresponding to the optimal moving object includes: performing multi-scale feature point detection on the region where the optimal moving object is located using a preset feature detector to obtain the feature information; constructing a Hamming distance-based K-nearest neighbor matching model based on the feature information to filter matching point sets where the distance ratio of matching point pairs is less than a preset ratio; iteratively optimizing the matching point set to calculate the projection error of the homography matrix corresponding to the matching point set; the projection error being less than a preset tolerance; and determining the position information of each frame in the sub-video within the panoramic video based on the projection error.

[0085] Multi-scale feature point detection is performed on the region containing the optimal moving object using a pre-defined feature detector (such as SIFT, SURF, or ORB) to obtain feature information. Multi-scale feature point detection can adapt to images of different resolutions and scales, improving the robustness of feature points. The SIFT feature detector is used to perform multi-scale Gaussian pyramid decomposition on the region containing the optimal moving object, extracting key points and their corresponding feature descriptors. By setting scale space parameters, stable feature points can be detected at different resolutions.

[0086] A K-nearest neighbor (KNN) matching model based on Hamming distance is constructed based on feature information to filter matching point sets where the distance ratio between matching point pairs is less than a preset ratio. Hamming distance can quickly calculate the similarity between binary feature descriptors, improving matching efficiency. The FLANN (Fast Approximate Nearest Neighbor) matching algorithm is used to perform KNN matching on feature points in sub-video frames and panoramic video frames. By setting a distance ratio threshold (e.g., 0.7), matching point sets where the distance ratio between matching point pairs is less than the preset ratio are filtered out.

[0087] The matching point set is iteratively optimized, and the projection error of the homography matrix corresponding to the matching point set is calculated. Mismatched points are eliminated using the RANSAC (Random Sample Consistency) algorithm, thus optimizing the calculation accuracy of the homography matrix. The RANSAC algorithm is used to iteratively optimize the matching point set, and the projection error of the homography matrix is ​​calculated. By setting a projection error tolerance (e.g., 2 pixels), the accuracy of the homography matrix is ​​ensured to meet the requirements.

[0088] The position information of each frame in the sub-video within the panoramic video is determined based on the projection error. The coordinates of the sub-video frames are mapped to the panoramic video using a homography matrix to obtain their position information. Based on the optimized homography matrix, the coordinates of the sub-video frames are mapped to the panoramic video, and their position coordinates within the panoramic video are calculated. The accuracy of the position information is verified using projection error analysis to ensure stitching precision.

[0089] Through the implementation of the embodiments, this invention can accurately obtain the position information of sub-video frames in panoramic video. It can adapt to images of different resolutions and scales, improving the robustness and detection accuracy of feature points. The Hamming distance-based K-nearest neighbor matching model can quickly filter out high-quality matching point pairs, improving matching efficiency. The RANSAC algorithm is used to optimize the calculation of the homography matrix, ensuring that the projection error is within a preset tolerance, thus improving the accuracy of position information.

[0090] In some embodiments, performing perspective transformation and image cropping on each frame of the sub-video to obtain the transformed sub-image includes: performing perspective transformation on the image edges according to the homography matrix corresponding to each frame to generate a perspective-transformed quadrilateral region; the perspective transformation is a bilinear interpolation transformation; calculating the minimum bounding rectangle corresponding to the quadrilateral region, scaling the minimum bounding rectangle according to the target resolution ratio to obtain a mapping box; detecting invalid pixel regions at the edges of the mapping box using an adaptive thresholding method, cropping the image based on the invalid pixel regions at the edges; performing gamma correction on the cropped image to align the image's brightness histogram with the background region corresponding to the panoramic video, thereby obtaining the sub-image.

[0091] Perspective transformation is performed on the image edges based on the homography matrix corresponding to each frame, generating quadrilateral regions after perspective transformation. Bilinear interpolation is used for the perspective transformation to ensure the smoothness and continuity of the image transformation. Based on the homography matrix, perspective transformation is performed on the four corner points of the sub-video frame to generate quadrilateral regions. A bilinear interpolation algorithm is then used to interpolate the pixels within the quadrilateral regions to ensure the smoothness of the transformed image.

[0092] Calculate the minimum bounding rectangle corresponding to the quadrilateral region. Scale the minimum bounding rectangle according to the target resolution ratio to obtain the bounding box. The minimum bounding rectangle effectively encompasses the quadrilateral region, avoiding the loss of image information. The minimum bounding rectangle of the quadrilateral region is calculated and scaled according to the target resolution ratio (e.g., the resolution of the panoramic video) to generate the bounding box. This scaling operation ensures that the sub-image has the same resolution as the panoramic video.

[0093] An adaptive thresholding method is used to detect invalid pixel regions at the edges of the bounding box, and the image is then cropped based on these invalid pixel regions. The adaptive thresholding method dynamically adjusts the threshold based on local image features, improving the detection accuracy of invalid pixel regions. The adaptive thresholding method is used to detect the edge regions of the bounding box and identify invalid pixel regions. Based on the location of these invalid pixel regions, the image is cropped to remove invalid pixels at the edges, ensuring the validity of the sub-image.

[0094] Gamma correction is applied to the cropped image to align its brightness histogram with the corresponding background area of ​​the panoramic video, resulting in a sub-image. Gamma correction adjusts the brightness and contrast of the image to ensure visual consistency between the sub-image and the panoramic video. The brightness histogram of the cropped image is calculated and compared with the background area of ​​the panoramic video to determine the gamma correction parameters. The brightness and contrast of the image are then adjusted using gamma correction to align it with the background area of ​​the panoramic video, generating the final sub-image.

[0095] Through the implementation of the embodiments, the present invention can generate high-quality sub-images, ensuring seamless stitching with panoramic videos. Specific beneficial effects include: employing bilinear interpolation to ensure the smoothness and continuity of image transformation, avoiding image distortion; detecting invalid pixel regions using an adaptive thresholding method to accurately crop image edges, ensuring the validity of the sub-images; and adjusting the brightness and contrast of the images through gamma correction to align them with the background area of ​​the panoramic video, improving the visual effect of the stitched video.

[0096] In some embodiments, the step of identifying moving objects in adjacent video streams and detecting the same moving objects corresponding to adjacent video streams includes: obtaining scale-invariant feature descriptors of moving objects in adjacent video streams; constructing a feature similarity matrix based on the scale-invariant feature descriptors; calculating a motion direction consistency index corresponding to moving objects within the same time window in adjacent video streams based on a timestamp synchronization mechanism; determining the moving object as the same moving object when both the feature similarity of the feature similarity matrix corresponding to the moving object and the motion direction consistency index exceed a preset threshold; and establishing a joint feature tracking list across video streams for the same moving object.

[0097] The motion equations of corresponding moving objects are obtained based on their motion trajectories. These equations describe the motion patterns by fitting the motion trajectory of the moving object in the time-space coordinate system. Curve fitting is performed on the motion trajectories of identical moving objects in adjacent video streams to obtain their motion equations. The least squares method is used to fit the motion trajectories, ensuring the accuracy of the motion equations.

[0098] The motion equations of objects corresponding to adjacent video streams are fitted in a time-space coordinate system to obtain the difference in trajectory slope. The difference in trajectory slope is evaluated by calculating the difference in slope of the motion equations of moving objects in adjacent video streams. The slopes of the motion equations of objects in adjacent video streams are calculated to obtain their trajectory slope differences. By calculating the slope difference, the difference in the motion trajectory of moving objects in adjacent video streams is evaluated.

[0099] The pixel coordinate offsets at the intersection points of the trajectories of adjacent video streams are obtained to generate a position deviation vector field. The pixel coordinate offsets are evaluated by calculating the coordinate differences of moving objects in adjacent video streams at their trajectory intersection points. The intersection points of the motion equations of moving objects in adjacent video streams are calculated to obtain the pixel coordinate offsets at their trajectory intersection points. The position deviation vector field is then generated by calculating the coordinate differences.

[0100] Based on the distribution characteristics of the deviation vector field, the local affine transformation parameters of the stitching boundary corresponding to adjacent video streams are calculated. These local affine transformation parameters describe the positional deviation between adjacent video streams by fitting the distribution characteristics of the deviation vector field. The distribution characteristics of the deviation vector field are analyzed, and the local affine transformation parameters are fitted using the least squares method. The positional deviation between adjacent video streams is described by calculating the affine transformation matrix.

[0101] Positional deviation information is generated based on local affine transform parameters. This information describes the positional deviation between adjacent video streams and is used for subsequent splicing misalignment correction. The positional deviation information is then used to adjust the splicing boundaries and eliminate splicing misalignment.

[0102] Through the implementation of the embodiments, this invention can accurately generate positional deviation information between adjacent video streams for splicing misalignment correction. By fitting the motion equation of an object, the motion law of the moving object is accurately described, improving the accuracy of trajectory analysis. By calculating the difference in trajectory slope and pixel coordinate offset, a positional deviation vector field is generated to accurately assess the positional deviation between adjacent video streams. Positional deviation information is generated through local affine transformation parameters to adjust the splicing boundary, eliminate splicing misalignment, and improve the accuracy and stability of video splicing.

[0103] In step S102, this application further obtains scale-invariant feature descriptors (such as SIFT or SURF descriptors) of moving objects in adjacent video streams. Scale-invariant feature descriptors can adapt to different scales and rotations, improving the robustness of feature matching. The SIFT algorithm is used to detect feature points of moving objects in adjacent video streams and extract their corresponding scale-invariant feature descriptors. By setting feature point detection parameters, stable feature descriptors can be extracted under different scales and rotations.

[0104] A feature similarity matrix is ​​constructed based on scale-invariant feature descriptors. This matrix evaluates the matching degree of moving objects by calculating the similarity between feature descriptors of moving objects in adjacent video streams. Euclidean distance is used to calculate the similarity between feature descriptors of moving objects in adjacent video streams to construct the feature similarity matrix. By setting a similarity threshold, feature descriptor pairs with high similarity are selected.

[0105] Based on a timestamp synchronization mechanism, a motion direction consistency index is calculated for moving objects within the same time window in adjacent video streams. The motion direction consistency index is evaluated by calculating the difference in motion direction between moving objects in adjacent video streams. Timestamps of moving objects in adjacent video streams are synchronized, and their motion directions within the same time window are calculated. The motion direction consistency index is then evaluated by calculating the angular difference between the motion directions.

[0106] When both the feature similarity and motion direction consistency indices of the feature similarity matrix corresponding to a moving object exceed a preset threshold, the moving objects are identified as the same moving object. By comprehensively evaluating feature similarity and motion direction consistency, the accuracy of identifying the same moving object is improved. Feature similarity thresholds and motion direction consistency thresholds are set, and moving objects whose feature similarity and motion direction consistency both exceed the thresholds are selected as the same moving object.

[0107] For identical moving objects, a joint feature tracking list is established across video streams. This list records the feature information and trajectory of the same moving object in adjacent video streams, facilitating subsequent error correction and stitching optimization. The feature information and trajectory of the same moving object in adjacent video streams are recorded to establish the joint feature tracking list. Through this list structure, feature tracking and trajectory analysis across video streams are achieved.

[0108] Through the implementation of the embodiments, this invention can accurately identify the same moving objects in adjacent video streams and establish a joint feature tracking list. Scale-invariant feature descriptors are used to adapt to different scales and rotations, improving the robustness of feature matching. A timestamp synchronization mechanism and a motion direction consistency index are used to accurately evaluate the matching degree of moving objects, improving the accuracy of identifying the same moving objects. The establishment of a joint feature tracking list enables feature tracking and trajectory analysis across video streams, providing data support for subsequent error correction and stitching optimization.

[0109] In some embodiments, determining the positional deviation information of adjacent video streams based on the motion trajectories of the same moving objects in the corresponding video streams includes: obtaining the motion equations of the corresponding same moving objects based on the motion trajectories; fitting the motion equations of the objects corresponding to adjacent video streams in a time-space coordinate system to obtain the difference in trajectory slope; obtaining the pixel coordinate offsets at the intersection points of the trajectories of adjacent video streams to generate a positional deviation vector field; calculating the local affine transformation parameters of the splicing boundary corresponding to adjacent video streams based on the distribution characteristics of the deviation vector field; and generating the positional deviation information based on the local affine transformation parameters.

[0110] Simultaneously, this application can use a feature detector to perform feature detection on the panoramic image and detail image respectively to calculate feature points. The k-nearest neighbor matching (knnmatch) algorithm is used to perform feature matching on the feature points of the panoramic image and detail image to obtain the feature points after feature matching. The feature points after feature matching are filtered to ensure that the distance between feature points is within a certain proportion to obtain the best matching term. After obtaining the best matching term for the panoramic image and detail image, the homography matrix is ​​calculated using a robust method based on RANSAC to obtain the transformation matrix T. Based on the transformation matrix T, perspective transformation is performed on four points of the detail image to obtain four points of the detail image on the panoramic image. The bounding rectangle of these four points is calculated to obtain the mapping box of the detail image on the panoramic image.

[0111] The existing method for calculating magnification scale involves an m x n array camera. The bounding matrix is ​​calculated based on the region of each detail image in the panoramic image (green mapping box), resulting in the boundRect (red box) of all detail images in the panoramic image. This region is then cropped from the panoramic image and magnified using the formula: (3840*m + 200, 2160*n + 200). The +200 value can be adjusted based on the actual size of the black borders after the detail images are transformed, reserving space for subsequent black border cropping to ensure the final stitched image has a resolution of (3840*m, 2160*n). After magnifying the panoramic image, feature detection and matching are performed again with each detail image to calculate the corresponding transformation matrix. The resulting transformation matrix represents the magnified image, and the transformed detail images also represent the magnified resolution.

[0112] The scaling method used in this application involves an m x n array camera. Based on the region (green mapping box) of each detail image on the panoramic image, a scaling factor is calculated. The calculation method is as follows: find the minimum width and height within all region boxes (ensuring the minimum can be scaled to the specified resolution (3840, 2160), larger ones can be cropped). The scaling factor is obtained by calculating the scale ratio between the minimum width and height and (target width and height + edge increment). The edge increment is used for subsequent black border cropping after transformation, ensuring that each detail image reaches the target resolution (3840, 2160) after cropping. The calculation formula is: Sfactor = (min(rect.witdh) / (Wtarget + △W), min(rect.height) / (Htarget + △H)). After obtaining the scaling factor, the scaling factor parameter in the transformation matrix is ​​replaced with the scaling factor. The detail image is then transformed using the transformation matrix to obtain the magnified transformation result.

[0113] Compared with existing technologies, this application avoids the need to zoom in on panoramic images and perform feature detection, matching, and transformation matrix calculations again. This is particularly important at high resolutions where feature detection, matching, and transformation matrix calculations are extremely time-consuming. This significantly reduces the computation time required for the transformation matrix.

[0114] The specific technical features of feature transformation matching are as follows: In terms of hardware structure, it's essential to ensure that all detail images are on the panoramic image, and that there is a certain overlap between each detail image. This allows the sub-image to calculate its correspondence with the panoramic image and between the various detail images, thus enabling the stitching of the detail images. The main process of the stitching calculation involves processing image feature points, calculating the homography matrix, and matching bounding boxes to prepare for image stitching. This process includes multiple steps such as feature point extraction, matching, homography calculation, bounding box processing, and image scaling. Finally, we generate adjusted stitching parameters and save them to a file.

[0115] 1. Feature Point Extraction and Matching: For the baseline image and each frame of detail image, feature points and descriptors are extracted and then matched. Successfully matched feature points are used to calculate the homography matrix.

[0116] 2. Calculate the homography matrix: Use feature matching to calculate the homography matrix between the reference image and the detail image.

[0117] 3. Boundary processing: Calculate matching bounding boxes for each image. Calculate the enclosing rectangle of all bounding boxes as the overall layout for stitching the images.

[0118] 4. Image Scaling: Images are adjusted by calculating a scaling factor to fit the stitched panoramic image. The scaling factor is calculated based on the minimum size of the matching rectangle.

[0119] 5. Display Rectangle: Calculate the final display position based on the scaled rectangle and calculate a suitable display area for the stitched image.

[0120] 6. Save stitching parameters: The final generated stitching parameters (including homography matrix, rectangle, scaling factor, etc.) will be saved in the configuration file for use in subsequent image stitching.

[0121] The main calculation process and details are as follows:

[0122] void calculate_stitching_parameters(const cv::Mat&base_img);

[0123] / / 1. Extract feature points and descriptors from the baseline image;

[0124] kps_descs_t base_img_kps_descs = detect_and_compute_features(base_img);

[0125] / / 2. Process each detailed image;

[0126] for (auto const&video_name : video_stream.get_video_list_map()) {if(video_name.first == "base") continue; / / Skip the specified frame;

[0127] / / Read the detail image and extract feature points and descriptors;

[0128] cv::Mat img = cv::imread(video_name.second);

[0129] kps_descs_t img_kps_descs = detect_and_compute_features(img);

[0130] / / 3. Calculate the homography matrix and the matching rectangle;

[0131] homography_result_t homography_result = calculate_homography(base_img, img, base_img_kps_descs, img_kps_descs);

[0132] / / 4. Obtain the homography matrix and rectangle;

[0133] cv::Mat homograph = homography_result.first;

[0134] cv::Rect rect = homography_result.second;

[0135] / / 5. Save the homography matrix and rectangle (may be stored or further processed in the actual code);

[0136] save_homography_and_rect(homograph, rect);

[0137] };

[0138] / / 6. Calculate the bounding rectangle of the matching rectangles for all detail images

[0139] std::vector <cv::rect>rects = get_all_rects_from_previous_calculations();

[0140] cv::Rect bound_rect = calculate_bounding_rect(rects);

[0141] / / 7. Calculate the scaling factor;

[0142] std::vector <int>edge_inc = get_edge_increment(); / / Get the edge increment;

[0143] float scale_factor = calculate_scale_factor(rects, edge_inc);

[0144] / / 8. Calculate the scaled homography matrix and the bounding box;

[0145] for (auto const&frame_name : video_stream.get_video_list_map()) {cv::Mat homograph = load_homograph(frame_name);

[0146] cv::Rect rect = load_rect(frame_name);

[0147] / / Calculate the scaled matrix and rectangle;

[0148] homography_scale_result_t scale_result = calculate_homography_scale(homograph, rect, scale_factor);

[0149] / / Save the scaled matrix and rectangle;

[0150] save_scaled_homograph_and_rect(scale_result);

[0151] };

[0152] / / 9. Calculate the display rectangle;

[0153] std::vector <cv::rect>display_rects = calculate_display_rects();

[0154] cv::Rect display_bound_rect = calculate_bounding_rect(display_rects);

[0155] / / 10. Save the final parameters (such as displaying the rectangle);

[0156] save_final_parameters(display_bound_rect);

[0157] };

[0158] float calculate_scale_factor(const std::vector <cv::rect>&rects, conststd::vector <int>&edge_inc) ;

[0159] Calculate scaling factor

[0160] return calculate_scale(rects, edge_inc);

[0161] };

[0162] cv::Rect calculate_bounding_rect(const std::vector <cv::rect>&rects);

[0163] / / Calculate the enclosing rectangle of all rectangles

[0164] return get_bound_rect(rects);

[0165] };

[0166] homography_scale_result_t calculate_homography_scale(const cv::Mat&homograph, const cv::Rect&rect, float scale_factor)

[0167] / / Adjust the homography matrix and rectangle according to the scaling factor

[0168] return apply_scale_to_homography_and_rect(homograph, rect, scale_factor);

[0169] };

[0170] std::vector <cv::rect>calculate_display_rects()

[0171] / / Calculate the rectangle used for the final display

[0172] return get_display_rects();

[0173] }

[0174] In some embodiments, a deep learning model based on YOLOv8 or Mask R-CNN is used to detect and segment moving objects in panoramic videos and sub-videos. Fine-tuning is performed using a pre-trained model and a custom dataset (such as a moving object dataset for dynamic scenes) to improve the model's detection accuracy in specific scenarios. The model is pre-trained using the COCO dataset and then transferred to a dynamic scene dataset for transfer learning to optimize model parameters. Moving object features extracted by the deep learning model (such as bounding boxes, masks, and class probabilities) are fused with traditional features (such as SIFT and ORB feature points) to generate multimodal feature descriptors. Feature fusion enhances the feature representation capability of moving objects. Dimensionality reduction is applied to the features extracted by the deep learning model, and a weighted fusion is performed with traditional features to generate multimodal feature descriptors. An adaptive feature matching model based on the multimodal feature descriptors is constructed, dynamically adjusting the matching strategy according to the scene. For example, deep learning features are prioritized in dynamic scenes, while traditional features are prioritized in static scenes. A dynamic weight allocation mechanism is employed to dynamically adjust the weights of multimodal features according to the scene, optimizing feature matching accuracy.

[0175] Deep learning models can accurately detect and segment moving objects, adapting to complex and dynamic scenes. Feature fusion enhances the feature representation of moving objects, improving the robustness of feature matching. Matching strategies are dynamically adjusted based on the scene to optimize feature matching accuracy and efficiency.

[0176] In some embodiments, spatiotemporal consistency constraint modeling is performed on the stitching boundary of adjacent video streams. By analyzing the motion trajectory and feature consistency of moving objects at the stitching boundary, the smoothness and continuity of the stitching boundary are optimized. A spatiotemporal consistency constraint algorithm is used to smooth the trajectory of moving objects at the stitching boundary, ensuring the spatiotemporal consistency of the stitching boundary. Based on the spatiotemporal consistency constraint model, the stitching boundary is optimized to eliminate stitching misalignment and visual artifacts. For example, image inpainting algorithms are used to fill and repair invalid pixel areas at the stitching boundary. Deep learning-based image inpainting algorithms (such as DeepFill) are used to fill and repair invalid pixel areas at the stitching boundary, optimizing the visual effect of the stitching boundary. During real-time video stitching, the stitching boundary optimization strategy is dynamically adjusted, and the stitching boundary is optimized in real time according to scene changes. For example, the weight of the spatiotemporal consistency constraint is increased in dynamic scenes and decreased in static scenes. A dynamic weight allocation mechanism is adopted to adjust the weight of the spatiotemporal consistency constraint in real time according to scene changes, optimizing the real-time performance and stability of the stitching boundary. By modeling with spatiotemporal consistency constraints, the smoothness and continuity of the stitching boundary are optimized, eliminating stitching misalignment and visual artifacts. The stitching boundary optimization strategy is dynamically adjusted to adapt to scene changes, improving the stability and visual effect of real-time video stitching.

[0177] In some embodiments, multi-view geometry of the array camera is modeled, and the geometric consistency of video stitching is optimized by analyzing the position and pose of moving objects under different viewpoints. For example, a multi-view geometric constraint algorithm is used to correct the position and pose of moving objects. This ensures the geometric consistency of video stitching. Based on the multi-view geometric model, the geometric consistency of video stitching is optimized to improve stitching accuracy. For example, a multi-view geometric correction algorithm is used to correct geometric distortion in the stitched video. During real-time video stitching, the multi-view geometric optimization strategy is dynamically adjusted to optimize stitching accuracy in real time according to scene changes. For example, the weight of multi-view geometric constraints is increased in dynamic scenes and decreased in static scenes. A dynamic weight allocation mechanism is used to adjust the weight of multi-view geometric constraints in real time according to scene changes, optimizing the real-time performance and stability of stitching accuracy.

[0178] By using multi-view geometric modeling, the geometric consistency of video stitching is optimized, improving stitching accuracy. Dynamic adjustments to the multi-view geometric optimization strategy adapt to scene changes, enhancing the stability and visual quality of real-time video stitching.

[0179] In some embodiments, a deep learning model based on U-Net or GAN is employed to detect and correct splicing errors. Fine-tuning is performed using a pre-trained model and a custom dataset (such as a splicing error dataset) to improve the model's correction accuracy in specific scenarios. The model is pre-trained using the splicing error dataset and then transferred to a dynamic scene dataset for transfer learning to optimize model parameters. Splicing errors are detected in the spliced ​​video, and splicing misalignments and visual artifacts are identified using a deep learning model. For example, a U-Net model is used to segment the spliced ​​video and identify splicing error regions. Based on the splicing error regions detected by the deep learning model, the spliced ​​video is corrected. For example, a GAN model is used to repair and correct the splicing error regions. Finally, a GAN model is used to repair and correct the splicing error regions, optimizing the visual effect of the spliced ​​video.

[0180] Deep learning models can accurately detect splicing errors and adapt to complex and dynamic scenes. By using deep learning models to repair and correct splicing errors, the visual effect of spliced ​​videos can be optimized.

[0181] In some embodiments, to address the inter-frame position fluctuation problem caused by using only single-frame feature matching in existing embodiments, a proposed method is to utilize temporal Transformer encoding to correlate features between sub-video frames, combined with temporal consistency constraints of the homography matrix, to optimize the position estimation of sub-video frames in panoramic video. This scheme can leverage the continuity of video to improve the stability and accuracy of position estimation, and is particularly suitable for scenes with fast-moving objects or complex backgrounds.

[0182] Feature point sequence extraction: For the sub-video frame sequence where the optimal moving object is located (such as frames t-2, t-1, t, t+1, t+2), the SIFT feature detector is used to extract the feature points of each frame (such as 100 feature points), resulting in a feature point sequence (each feature point contains coordinates and descriptors).

[0183] The temporal Transformer encoder inputs the feature point sequence into the temporal Transformer encoder (each feature point is a token), captures the correlation between feature points between frames through a self-attention layer (such as the similarity between the feature points of frame t and the feature points of frames t-1 and t+1), and outputs temporally enhanced feature point descriptors (which incorporate inter-frame context information).

[0184] The SuperGlue feature matching model is used to match the feature points of the temporally enhanced sub-video frames with the feature points of the corresponding frames of the panoramic video to obtain the initial matching point set; the RANSAC algorithm is used to calculate the initial homography matrix (H_init).

[0185] Temporal consistency constraint optimization introduces a homography matrix temporal consistency constraint: the homography matrix change between adjacent frames should not be too large (e.g., the Frobenius norm difference between H_t and H_{t-1} ≤ ε, where ε is a preset threshold, such as 0.1). If the threshold is exceeded, then: using the inter-frame association information of the Transformer encoder, inconsistent matching points in the initial matching point set are deleted (e.g., points where the feature point of frame t matches the feature point of frame t-1 but the position changes too much); the homography matrix (H_opt) is recalculated until the temporal consistency constraint is satisfied.

[0186] The location information is determined by calculating the coordinates of each pixel in the sub-video frame in the panoramic video based on the optimized homography matrix (H_opt). For example, the pixel coordinates (x,y) of the sub-video frame are mapped to the coordinates (X,Y) of the panoramic video through H_opt, thus obtaining the location information of the sub-video frame in the panorama.

[0187] In some embodiments, to address the issues of image blurring after perspective transformation, inconsistency with the panoramic background, and truncation of moving objects due to cropping relying on invalid edge pixels in existing embodiments, a proposed solution is to use a conditional GAN ​​to generate sub-images consistent with the panoramic background (the condition being panoramic background features), and to automatically determine the cropping region (based on the position of the moving object and background complexity) through a self-supervised cropping network. This approach improves the quality of the sub-images, ensures the integrity of moving objects, and reduces background redundancy.

[0188] The initial perspective transformation is performed on the sub-video frames using bilinear interpolation based on the homography matrix (H_opt) obtained in S102 to obtain the initial transformed image.

[0189] Panoramic background feature extraction uses the VGG16 network to extract background features (such as background color histogram and texture features) of the corresponding region in the panoramic video, which are then used as the conditional input of the conditional GAN.

[0190] Conditional GAN ​​consists of a generator (G) and a discriminator (D): Generator input: initial transformed image + panoramic background features; Generator output: optimized transformed image (reduced blur, adjusted colors to match the panoramic background); Discriminator input: optimized transformed image + panoramic background features, output the probability of "real" (panoramic image) or "generated" (sub-image); Loss function: adversarial loss (GAN loss) + content loss (L1 loss between optimized image and initial transformed image) + background consistency loss (cosine similarity loss between optimized image and panoramic background features).

[0191] The self-supervised cropping network design constructs a self-supervised cropping network (such as the U-Net structure). The inputs are: the optimized transformed image; the bounding box of the optimal moving object (from S101, such as (x1,y1,x2,y2)); and the outputs are the coordinates of the cropping region (such as the top left corner (x_min,y_min) and bottom right corner (x_max,y_max) of the rectangle).

[0192] Self-supervised loss functions: Moving object integrity loss: The cropped region must contain the bounding box of the optimal moving object (e.g., (x_min ≤ x1) ∧ (y_min ≤ y1) ∧ (x_max ≥ x2) ∧ (y_max ≥ y2), using 0-1 loss); Background redundancy loss: The smaller the proportion of background pixels outside the cropped region, the smaller the loss (e.g., divide the number of pixels outside the cropped region by the total number of pixels in the image, and take the reciprocal as the loss);

[0193] Image resolution loss is achieved by ensuring that the resolution of the cropped image meets the target requirements (e.g., 1920×1080, using L2 loss).

[0194] The final sub-image generation uses the cropping region output by the cropping network to crop the optimized transformed image, resulting in the final sub-image (containing the complete moving object, with little background redundancy, and consistent with the panoramic background).

[0195] In some embodiments, to address the low efficiency and poor consistency issues caused by the separation of "identical moving object recognition" and "positional deviation estimation" in existing embodiments, a proposed approach is to associate moving object features from adjacent video streams using a cross-video stream Transformer, combined with multi-task learning (simultaneously performing identical object recognition and deviation estimation), thereby improving the accuracy and real-time performance of adjacent video stream stitching. This scheme can utilize contextual information across video streams to reduce misjudgments in identical object recognition, while making deviation estimation more consistent with the actual changes in motion trajectories.

[0196] Moving object feature extraction involves using a ResNet50 network to extract the appearance features (e.g., 2048-dimensional vectors) of each moving object from two adjacent video streams (e.g., video stream A and video stream B), and using optical flow to extract motion features (e.g., average optical flow vectors). These are then fused to obtain a comprehensive moving object feature (e.g., appearance features + motion features, totaling 2048 + 2 = 2050 dimensions).

[0197] Cross-video stream Transformer encoding inputs the moving object feature sequence (such as features of frames t-1, t, and t+1) of video stream A and the moving object feature sequence of video stream B into the cross-video stream Transformer. By capturing the correlation of moving objects in the two video streams (such as the similarity between object features in video stream A and object features in video stream B) through a cross-attention layer, it outputs an enhanced feature representation across the video stream.

[0198] The output of the cross-video stream Transformer is divided into two branches: classification branch (identical object recognition): the fully connected layer outputs probability values ​​(indicating whether the moving objects in the two video streams are the same object, such as if the probability is ≥0.8, then they are judged to be the same); regression branch (positional deviation estimation): the fully connected layer outputs a deviation vector (such as (Δx, Δy), which represents the positional offset of video stream B relative to video stream A).

[0199] Classification loss includes cross-entropy loss (L_cls = -[y log§ + (1-y) log(1-p)], where y is the true label and p is the classification probability); regression loss: L2 loss (L_reg = ||Δx_pred - Δx_gt|| 2 + ||Δy_pred -Δy_gt||², where Δx_gt and Δy_gt are the true deviations, calculated from the intersection of the motion trajectories; if the classification branch determines that the objects are the same, then the deviation vector of the regression branch should be consistent with the difference in the motion trajectory (e.g., L_cons = ||Δx_pred - (x_B - x_A)|| 2 + ||Δy_pred - (y_B - y_A)|| 2 x_A and y_A are the coordinates of an object in video stream A, and x_B and y_B are the coordinates of an object in video stream B.

[0200] Deviation correction and stitching use a trained multi-task model to process moving objects in adjacent video streams: if the classification branch determines that they are the same object, the deviation vector (Δx, Δy) of the regression branch is used to perform a local affine transformation on the stitching boundary of video stream B (adjusting the position of video stream B so that the trajectories of the same objects are aligned); the above process is repeated until the stitching misalignment of all adjacent video streams is eliminated, and the stitching of multiple video streams is completed.

[0201] In some embodiments, to address the image blurring and semantic distortion (such as object shape distortion) caused by perspective transformation in existing embodiments, a proposed approach is to utilize multi-view depth information from an array camera to guide the interpolation process (depth-aware interpolation, with higher weighting for near-field pixels) to reduce blurring. Simultaneously, a self-supervised semantic alignment model is used to ensure that the semantic content of the sub-image (such as object category and background structure) is consistent with the corresponding region in the panoramic video. This scheme improves the image clarity and semantic consistency after perspective transformation and is suitable for complex scenes (such as indoor multi-object scenes and outdoor distant views).

[0202] Multi-view depth maps are obtained from multi-view images (such as images from the left, right, and front views) captured by an array camera. The depth map (depth value of each pixel in meters) of the sub-video frame is calculated using the multi-view stereo vision algorithm (COLMAP).

[0203] The depth-aware perspective transformation performs perspective transformation on its sub-video frames based on the homography matrix (H_opt) obtained in S102. The interpolation process uses depth-aware interpolation: for the target pixel (X,Y), the corresponding source pixel (x,y) is found through the inverse matrix of H_opt; the depth value d of the source pixel (x,y) is obtained from the depth map (the smaller d is, the closer the object is); the weight w = 1 / d of the source pixel is calculated (the smaller the depth, the larger the weight); the weight w is used to perform a weighted average of the four neighboring pixels around the source pixel (bilinear interpolation) to obtain the grayscale value (or RGB value) of the target pixel.

[0204] Semantic feature extraction uses the Mask R-CNN semantic segmentation model to extract the semantic masks of sub-images (such as the category label of each pixel, "person", "vehicle", "wall" etc.) and the semantic masks of corresponding regions in the panoramic video.

[0205] Construct a self-supervised semantic alignment model (such as the U-Net structure). The input is the semantic mask of the corresponding region of the panoramic video through the depth-sensing interpolated sub-image; the output is the semantically aligned sub-image (adjusting the shape of the object so that the semantic mask is consistent with the panoramic view).

[0206] The semantic consistency loss is calculated using Dice loss to determine the difference between the subgraph semantic mask and the panoramic semantic mask (L_dice = 1 - 2×|A∩B| / (|A|+|B|), where A is the subgraph semantic mask and B is the panoramic semantic mask).

[0207] Image quality loss is calculated using SSIM loss to determine the sharpness difference between the semantically aligned sub-image and the depth-aware interpolated sub-image (L_ssim = 1 - SSIM(I_align, I_depth)).

[0208] Sub-image generation involves cropping the semantically aligned sub-images by using an adaptive thresholding method to detect invalid pixels at the edges, resulting in a final sub-image (clear, semantically consistent, and free of invalid pixels).

[0209] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a video stitching device 200 provided in an embodiment of this application. The video stitching device 200 is used to perform the steps of the video stitching method shown in any embodiment of this application. The video stitching device 200 can be a single server or a server cluster, or it can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.

[0210] like Figure 2 As shown, the video splicing device 200 includes:

[0211] The video acquisition unit 201 is used to acquire panoramic video and multiple sub-videos captured by the array camera, perform moving object detection on the panoramic video and multiple sub-videos, acquire multiple moving objects and corresponding feature information, and determine the optimal moving object among the multiple moving objects based on the feature information.

[0212] The position acquisition unit 202 is used to acquire the position information of each frame of the sub-video in the panoramic video based on the feature information corresponding to the optimal moving object;

[0213] The cropping and transformation unit 203 is used to perform perspective transformation and image cropping on each frame of the sub-video to obtain the transformed sub-image, and to stitch multiple frames of the sub-image according to the position information corresponding to each frame to complete the stitching of the sub-image corresponding to each sub-video and obtain multiple stitched video streams.

[0214] The stitching completion unit 204 is used to identify moving objects in adjacent video streams, detect the same moving objects in adjacent video streams, determine the positional deviation information of adjacent video streams based on the motion trajectory of the same moving objects in the corresponding video streams, eliminate the stitching misalignment of adjacent video streams based on the positional deviation information, and complete the stitching of multiple video streams.

[0215] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the video splicing device 200 and its modules described above can be found by referring to... Figure 1 The corresponding processes in the video stitching method embodiments described in the corresponding examples will not be repeated here.

[0216] Figure 1 The corresponding video stitching method can be implemented as a computer program, which can perform tasks such as... Figure 2 It runs on the device shown.

[0217] Please see Figure 3 , Figure 3 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.

[0218] The storage medium can store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform... Figure 1 Any corresponding video splicing method.

[0219] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0220] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any video stitching method.

[0221] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0222] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0223] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0224] A panoramic video and multiple sub-videos captured by an array camera are acquired. Moving object detection is performed on the panoramic video and multiple sub-videos to acquire multiple moving objects and their corresponding feature information, so as to determine the optimal moving object among the multiple moving objects based on the feature information.

[0225] Based on the feature information corresponding to the optimal moving object, obtain the position information of each frame in the sub-video in the panoramic video;

[0226] Perspective transformation and image cropping are performed on each frame of the sub-video to obtain the transformed sub-image. Multiple frames of the sub-image are stitched together according to the position information corresponding to each frame to complete the stitching of the sub-image corresponding to each sub-video and obtain multiple stitched video streams.

[0227] Moving object recognition is performed on adjacent video streams to detect the same moving object in the adjacent video streams. Based on the motion trajectory of the same moving object in the corresponding video stream, the positional deviation information of the adjacent video streams is determined. Based on the positional deviation information, the splicing misalignment of the adjacent video streams is eliminated, and the splicing of multiple video streams is completed.

[0228] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform as described in this application. Figure 1 The steps of the video stitching method provided in any corresponding embodiment.

[0229] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0230] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / cv::rect> < / cv::rect> < / int> < / cv::rect> < / cv::rect> < / int> < / cv::rect>

Claims

1. A video stitching method for an array camera, characterized in that, include: A panoramic video and multiple sub-videos captured by an array camera are acquired. Moving object detection is performed on the panoramic video and multiple sub-videos to acquire multiple moving objects and their corresponding feature information, so as to determine the optimal moving object among the multiple moving objects based on the feature information. Based on the feature information corresponding to the optimal moving object, obtain the position information of each frame in the sub-video in the panoramic video; The process involves performing perspective transformation and image cropping on each frame of the sub-video to obtain transformed sub-images. This includes: performing perspective transformation on the image edges based on the homography matrix corresponding to each frame to generate a perspective-transformed quadrilateral region; the perspective transformation is a bilinear interpolation transformation; calculating the minimum bounding rectangle corresponding to the quadrilateral region, scaling the minimum bounding rectangle according to the target resolution ratio to obtain a bounding box; detecting invalid pixel regions at the edges of the bounding box using an adaptive thresholding method, and cropping the image based on these invalid pixel regions; performing gamma correction on the cropped image to align the image's brightness histogram with the background region corresponding to the panoramic video, thus obtaining the sub-image; and stitching multiple frames of the sub-images together based on the position information corresponding to each frame to complete the stitching of the sub-images corresponding to each sub-video, thereby obtaining multiple stitched video streams. Moving object recognition is performed on adjacent video streams to detect identical moving objects in adjacent video streams. Based on the motion trajectories of the identical moving objects in their respective video streams, positional deviation information for adjacent video streams is determined. This includes: obtaining the motion equations of the corresponding identical moving objects based on their motion trajectories; fitting the motion equations of the objects in adjacent video streams to obtain the difference in trajectory slope in a time-space coordinate system; obtaining the pixel coordinate offsets at the intersection points of the trajectories of adjacent video streams to generate a positional deviation vector field; calculating the local affine transformation parameters of the splicing boundary for adjacent video streams based on the distribution characteristics of the deviation vector field; generating the positional deviation information based on the local affine transformation parameters; and eliminating splicing misalignments for adjacent video streams based on the positional deviation information to complete the splicing of multiple video streams.

2. The method according to claim 1, characterized in that, The step of detecting moving objects in the panoramic video and multiple sub-videos to obtain multiple moving objects and their corresponding feature information includes: Based on the motion region detection algorithm based on optical flow, candidate regions of candidate moving objects are obtained by calculating the inter-frame pixel displacement field of the panoramic video and multiple sub-videos. The candidate regions are subjected to color space conversion to obtain the color histogram features and edge gradient features corresponding to the candidate regions; Based on inter-frame motion vector analysis, candidate regions with continuous motion trajectories are obtained from the candidate regions as target regions; The candidate moving object corresponding to the target region is selected as the moving object, and the feature information corresponding to the moving object is obtained in the target region.

3. The method according to claim 1, characterized in that, The step of determining the optimal moving object among the multiple moving objects based on the feature information includes: Calculate the visual saliency index of each moving object and the background region, and obtain the first candidate moving object whose saliency index is higher than the first threshold. Calculate the smoothness of the motion trajectory of each moving object, and obtain a second candidate moving object whose motion trajectory smoothness is lower than a second threshold. Calculate the ratio of the object pixel area to the image resolution for each moving object, and obtain the third candidate moving object whose ratio is within a preset size range; Calculate the motion speed of each of the moving objects, and obtain the fourth candidate moving object whose motion speed is greater than a third threshold and less than a fourth threshold; The optimal moving object is determined based on the first candidate moving object, the second candidate moving object, the third candidate moving object, and the fourth candidate moving object.

4. The method according to claim 1, characterized in that, The step of obtaining the position information of each frame in the sub-video within the panoramic video based on the feature information corresponding to the optimal moving object includes: The feature information is obtained by performing multi-scale feature point detection on the region where the optimal moving object is located according to a preset feature detector. Based on the aforementioned feature information, a K-nearest neighbor matching model based on Hamming distance is constructed to filter the set of matching points whose distance ratio is less than a preset ratio. The matching point set is iteratively optimized, and the projection error of the homography matrix corresponding to the matching point set is calculated; the projection error is less than a preset tolerance. The position information of each frame in the sub-video within the panoramic video is determined based on the projection error.

5. The method according to claim 1, characterized in that, The step of identifying moving objects in adjacent video streams and detecting the same moving objects in adjacent video streams includes: Obtain scale-invariant feature descriptors of moving objects in adjacent video streams, and construct a feature similarity matrix based on the scale-invariant feature descriptors; Based on the timestamp synchronization mechanism, the consistency index of motion direction of moving objects within the same time window is calculated in adjacent video streams; When the feature similarity of the feature similarity matrix corresponding to the moving object and the consistency index of the motion direction both exceed a preset threshold, the moving object is determined to be the same moving object; for the same moving object, a joint feature tracking list across video streams is established.

6. The method according to claim 1, characterized in that, The method further includes: By modeling the multi-view geometry of the array camera and analyzing the position and posture of moving objects from different perspectives, the geometric consistency of video stitching is optimized.

7. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 1 to 6.

Images