Video splicing and fusion method, system and terminal

Abstract

The present application relates to a video splicing and fusion method, system and terminal, and belongs to the technical field of video processing. The video splicing and fusion method includes: synchronizing the timestamps of the videos received from the master device and the slave device; obtaining the overlapping area of ​​the videos from the master device and the slave device, and marking the overlapping area of ​​the master device as segment A, and marking the overlapping area of ​​the slave device as segment B; extracting the feature vector of the key frame A1 of segment A and the feature vector of the key frame B1 of segment B; calculating the key frame similarity based on the feature vector of key frame A1 and the feature vector of key frame B1; obtaining the time difference and location similarity between key frame A1 and key frame B1; calculating the comprehensive similarity based on the key frame similarity, time difference and location similarity; judging whether the comprehensive similarity is greater than the similarity threshold; if so, splicing and fusing segment B with the previous frame of segment A. The present application improves the situation where obvious splicing marks appear after video splicing and fusion.

Description

Technical Field

[0001] The present application relates to the technical field of video processing, and in particular to a video splicing and fusion method, system, and terminal. Background Art

[0002] In today's video applications, such as security surveillance, live events, and virtual reality, multiple devices are often used simultaneously to capture more comprehensive and richer video information. These devices can be divided into master and slave devices, each capturing video from different perspectives. However, effectively stitching and fusing the videos captured by these different devices to form a complete, coherent, and high-quality video image is a challenging task. This requires not only solving the problem of temporal and spatial synchronization of the videos, but also ensuring a smooth transition at the stitching point to avoid noticeable stitching artifacts.

[0003] Currently, some existing technologies consider the time synchronization of videos when performing video splicing and fusion, and use certain methods to calibrate the video timestamps of different devices. Some technologies also attempt to identify overlapping areas of videos and stitch them together. In terms of determining the matching degree of video splicing, some technologies extract feature information of the videos and calculate similarity. However, most existing technologies only focus on information in a single dimension, such as focusing only on the feature similarity of videos, or only considering time synchronization factors, and lack comprehensive consideration of multiple important factors. Due to the lack of comprehensive consideration of multiple dimensions, various problems are prone to occur when performing video splicing and fusion.

[0004] For example, relying solely on feature similarity for stitching can lead to incorrect stitching due to similar features between videos shot at different times and locations, resulting in confusion in the temporal and spatial logic of the stitched video. Alternatively, if only considering time synchronization while ignoring feature matching and location information, the resulting stitched video may experience unnatural transitions and noticeable stitching artifacts, severely impacting the quality and viewing experience of the stitched video. Summary of the Invention

[0005] In order to improve the situation where obvious splicing traces appear after video splicing and fusion, the present application provides a video splicing and fusion method, system and terminal.

[0006] In the first aspect, the present application provides a video splicing and fusion method, which adopts the following technical solutions:

[0007] A video splicing and fusion method, comprising:

[0008] Synchronize the timestamps of the videos received from the master and slave devices;

[0009] Obtaining the overlapping area of ​​the video of the master device and the video of the slave device, and marking the overlapping area of ​​the master device as segment A, and marking the overlapping area of ​​the slave device as segment B;

[0010] Extracting a feature vector of the key frame A1 of the A segment and a feature vector of the key frame B1 of the B segment;

[0011] Calculate the key frame similarity based on the feature vector of key frame A1 and the feature vector of key frame B1;

[0012] Get the time difference and location similarity between key frame A1 and key frame B1;

[0013] Calculating a comprehensive similarity based on the key frame similarity, the time difference and the location similarity;

[0014] Determining whether the comprehensive similarity is greater than a similarity threshold;

[0015] If yes, then the B segment is spliced ​​and fused with the previous frame of the A segment;

[0016] If not, a transition segment is generated, and the transition segment is spliced ​​and fused with the A segment and the B segment.

[0017] By employing the above technical solution, the timestamps of the videos on the master and slave devices are synchronized, ensuring temporal consistency between the two videos. This avoids issues such as jumpy frames or temporal logic confusion in the spliced ​​video due to time differences, resulting in a smooth and continuous spliced ​​video, improving the accuracy and logical consistency of the video content. By extracting the feature vectors of key frames and calculating similarity, and combining time differences and location similarity to calculate a comprehensive similarity, the degree of match between the overlapping areas of the master and slave videos can be accurately determined. This multi-dimensional matching approach significantly improves the accuracy of the splicing process, resulting in a smooth transition at the spliced ​​area and minimizing splicing artifacts. By extracting the feature vectors of key frames for similarity calculation, rather than processing all frames, the computational effort and processing time are significantly reduced. This significantly improves the efficiency of splicing and fusion when processing large amounts of video data, reduces the demand for computing resources, and makes the video splicing and fusion process more efficient and rapid. Setting a similarity threshold and performing a judgment enables a quick decision on whether to proceed with the splicing operation. This simple and effective decision-making mechanism avoids complex calculations and judgments, further improving the efficiency of video stitching and making the entire stitching and fusion process more automated and intelligent. Furthermore, this technical solution not only considers the feature similarity of key frames, but also combines time difference and location similarity for comprehensive judgment. This multi-dimensional approach makes video stitching and fusion more flexible, adapting to the needs of video stitching in different scenarios and improving the situation where obvious stitching artifacts appear after video stitching and fusion.

[0018] Optionally, the step of synchronizing timestamps of the videos received from the master device and the slave device specifically includes:

[0019] Control the master device's clock source to periodically broadcast the time reference to the slave device, and the slave device's clock performs time calibration based on the deviation compensation formula;

[0020] A frame with a timestamp is generated every 15 frames from the received videos from the master device and the slave device, and marked as a time frame to achieve synchronization of the timestamps.

[0021] By implementing the above technical solution, the master device's clock source periodically broadcasts a time reference to the slave device. The slave device's clock is then aligned using a deviation compensation formula. This effectively eliminates time deviations between the master and slave devices. Due to hardware differences, operating environment, and other factors, the clocks of different devices may have slight errors. These errors can be amplified during the video splicing process, resulting in issues such as image jumps and time logic confusion in the spliced ​​video. By broadcasting the time reference from the master device and performing deviation compensation on the slave devices, the master and slave devices achieve high time consistency, ensuring temporal continuity and accuracy in the spliced ​​video. A time-stamped frame is generated every 15 frames from the received videos from the master and slave devices and marked as a time frame to achieve time stamp synchronization. This method generates time frames at fixed frame intervals, ensuring overall video time synchronization while avoiding the additional computational burden of frequent time frame generation. This regularized time frame generation method creates an orderly temporal correspondence between the videos of the master and slave devices, further enhancing the accuracy of time synchronization.

[0022] Optionally, after the comprehensive similarity is greater than the similarity threshold and before the overlapping areas are stitched together, the steps include:

[0023] Get the current coordinates of the selected matching point and the transformed coordinates of the selected matching point after being transformed by the homography matrix H;

[0024] Calculating a stitching reprojection error based on the current coordinates and the transformed coordinates;

[0025] Determining whether the stitching reprojection error is greater than an error threshold;

[0026] If so, optimize the homography matrix H.

[0027] By employing the above technical solution, the current coordinates of the selected matching points and their transformed coordinates after transformation using the homography matrix H are obtained, and the stitching reprojection error is calculated. This allows the precise measurement of the matching degree of the overlapping area between the master and slave devices before stitching. The reprojection error reflects the deviation between the transformed matching points and their actual positions. By calculating and evaluating this error, the stitching accuracy can be intuitively understood. When the stitching reprojection error is determined to be greater than the error threshold, the homography matrix H is optimized, which is a key step in improving stitching accuracy. The homography matrix H describes the projective transformation relationship between the two image planes, and its accuracy directly affects the stitching effect. Assessing the stitching reprojection error and performing matrix optimization before stitching enables the system to better adapt to various complex shooting scenarios. Different shooting environments may cause the characteristics of the matching points to change, thus affecting the accuracy of the homography matrix H.

[0028] Optionally, the steps after obtaining the overlapping area of ​​the videos of the master device and the slave device include:

[0029] Taking the A segment as the target splicing area, determining whether the B segment is jittering;

[0030] If yes, then obtaining pixel motion vectors of two adjacent frames in the B segment;

[0031] Determine a homography matrix H according to the pixel motion vector;

[0032] Eliminate jitter based on the inverse homography matrix.

[0033] By employing this technical solution, segment A, marked by the overlap area on the master device, is used as the target stitching area. The slave device determines whether segment B is experiencing jitter. This approach allows for the timely detection of unstable factors that could affect the stitching effect. When jitter is detected in segment B, the pixel motion vectors of two adjacent frames within segment B are obtained, and the homography matrix H is determined based on these pixel motion vectors. Pixel motion vectors reflect the displacement of pixels between adjacent frames. Analysis of these vectors accurately captures the characteristics and patterns of image jitter. The homography matrix H describes the projective transformation relationship between the two frames. Using pixel motion vectors to determine the homography matrix H accurately quantifies the degree and direction of image jitter. Eliminating jitter using the inverse homography matrix is ​​a highly effective method for image stabilization. The inverse homography matrix, the inverse of the determined homography matrix H, transforms the jittered image inversely, thereby offsetting the jitter effect. By applying the inverse homography matrix, the jittery B segment can be corrected, ensuring a smooth transition between it and the target stitching area, the A segment, when stitched together, avoiding unnatural artifacts caused by jitter. Jitter can cause the position and angle of the image to shift. If left unaddressed, this can lead to inaccurate alignment of the overlapping areas during stitching, resulting in stitching errors. By detecting and eliminating jitter in the B segment, pixels in the overlapping area can be accurately aligned during stitching, reducing stitching errors.

[0034] Optionally, the step before eliminating jitter according to the inverse homography matrix includes:

[0035] Calculating an optical flow error term according to the pixel motion vector;

[0036] According to the rotation matrix of the previous frame, the rotation matrix of the current frame is obtained;

[0037] Calculating a rotation smoothing term according to the rotation matrix of the previous frame and the rotation matrix of the current frame;

[0038] Get the translation smoothing items of the current frame and the previous frame;

[0039] Calculating an overall optimization objective according to the optical flow error term, the rotation smoothing term, and the translation smoothing term;

[0040] Determining whether the total optimization target is less than an optimization threshold;

[0041] If so, the jitter is eliminated according to the inverse homography matrix.

[0042] By employing the above technical solution, the optical flow error term is calculated based on pixel motion vectors, quantifying the pixel motion inconsistencies between video frames due to jitter. The optical flow error term reflects the deviation between the actual and ideal pixel motion between adjacent frames. By calculating this error term, the jitter location and extent can be precisely located. Separately calculating the rotational and translational smoothing terms facilitates further analysis of jitter from different dimensions. The rotational smoothing term, calculated using the rotation matrices of the previous and current frames, measures the degree of rotational jitter. The translational smoothing term, obtained from the current and previous frames, assesses the jitter in the translational direction. An overall optimization objective is calculated based on the optical flow error term, the rotational smoothing term, and the translational smoothing term, integrating multiple factors influencing jitter for evaluation. This multi-factor approach avoids the limitations of single-factor evaluation and more accurately reflects the overall jitter condition of the video. By calculating the overall optimization objective, it is possible to determine whether the jitter in the current video has reached a level requiring treatment, providing a sound basis for determining whether to perform subsequent jitter reduction operations. Determining whether the total optimization target is less than the optimization threshold sets a clear standard for whether to use the inverse homography matrix to eliminate jitter. Only when the total optimization target is less than the optimization threshold is the jitter elimination operation performed, which can avoid unnecessary calculations and processing and improve system efficiency. At the same time, it also ensures that the operation is only performed when it is really necessary to eliminate jitter, ensuring the effectiveness and pertinence of jitter elimination. Through the above series of steps, jitter is accurately analyzed and effectively judged, and the jitter problem of the video on the slave device can be properly addressed before splicing. The stable video image provides a good foundation for the subsequent splicing and fusion with the video on the master device, making the image transition at the splicing point more natural and reducing the splicing marks and image discontinuity caused by jitter.

[0043] Optionally, the video splicing and fusion method further includes:

[0044] Calculate the RGB mean of the A segment and the B segment;

[0045] Based on the color statistics of the overlapping area, a linear transformation matrix T is established;

[0046] According to the original pixel value of the B segment, the linear transformation matrix T, the RGB mean value of the A segment, and the RGB mean value of the B segment, the B segment is mapped to the color space of the A segment to obtain a corrected pixel value.

[0047] By employing this technical solution, during the video stitching process, due to differences in hardware parameters (such as sensor sensitivity and lens filters) and shooting environment (such as lighting conditions and angle) between the master and slave cameras, significant color differences between Clips A and B may occur. By calculating the RGB mean of Clips A and B, the color characteristics of the two clips can be quantified. A linear transformation matrix T is established based on the color statistics of the overlapping regions, providing accurate conversion rules for color mapping. Based on this information, Clip B is mapped to the color space of Clip A, effectively eliminating the color differences between the two clips and achieving more consistent color harmony in the stitched video.

[0048] Optionally, the video splicing and fusion method further includes:

[0049] According to the target brightness value and the pre-built optimization objective function, an optimized brightness value is obtained;

[0050] Calculating the error between the optimized brightness value and the target brightness value;

[0051] Determining whether the error value is less than an error threshold;

[0052] If not, the parameters of the optimization objective function are optimized.

[0053] By employing the above technical solution, an optimized brightness value is obtained based on the target brightness value and a pre-established optimization objective function. This process can adapt to the varying brightness conditions of the videos on the master and slave devices. Due to differences in shooting environments, hardware parameters, and other factors, video brightness often varies between devices, resulting in abrupt brightness differences during stitching and fusion. By calculating the optimized brightness value and then comparing it with the target brightness value, the system continuously adjusts the video brightness to approximate the target brightness, thereby ensuring brightness consistency in the stitched video. The system then determines whether the error value is less than an error threshold. If not, the parameters of the optimization objective function are optimized. This mechanism enables the video stitching and fusion system to adaptively adjust. When the brightness error is large, the system automatically adjusts the parameters of the optimization objective function to better adapt to varying video brightness conditions. As video brightness characteristics change due to factors such as ambient lighting or device aging, the system can continuously optimize the brightness adjustment to ensure high quality in the stitched video. Through brightness optimization and adaptive adjustment, this video stitching and fusion method is applicable to a variety of different application scenarios.

[0054] In the second aspect, this application provides a video splicing and fusion system, which adopts the following technical solutions:

[0055] A video splicing and fusion system, comprising:

[0056] A time synchronization module is used to synchronize the timestamps of the videos received from the master device and the slave device;

[0057] an overlapping region determining module, configured to obtain an overlapping region of the videos of the master device and the slave device, and mark the overlapping region of the master device as segment A, and mark the overlapping region of the slave device as segment B;

[0058] A feature vector extraction module, configured to extract a feature vector of the key frame A1 of the segment A and a feature vector of the key frame B1 of the segment B;

[0059] a calculation processing module, configured to calculate key frame similarity based on a feature vector of key frame A1 and a feature vector of key frame B1; the calculation processing module is further configured to obtain a time difference and a location similarity between key frame A1 and key frame B1, and calculate a comprehensive similarity based on the key frame similarity, the time difference, and the location similarity;

[0060] A judgment module, configured to judge whether the comprehensive similarity is greater than a similarity threshold;

[0061] The splicing and fusing module is configured to splice and fuse the B segment with the previous frame of the A segment when the judging module judges that the response is yes.

[0062] In a third aspect, the present application provides a terminal that adopts the following technical solution:

[0063] A terminal, comprising:

[0064] A memory storing a video splicing and fusion program;

[0065] The processor is used to execute the program stored in the memory to implement the steps of the above-mentioned video splicing and fusion method.

[0066] In summary, this application has at least the following beneficial effects:

[0067] Synchronizing the timestamps of the videos on the master and slave devices ensures temporal consistency between the two videos, preventing issues such as jumpy frames or temporal logic confusion in the spliced ​​video due to time differences. This ensures a continuous and smooth spliced ​​video, improving the accuracy and logical consistency of the video content. By extracting keyframe feature vectors and calculating similarity, and combining time difference and location similarity to calculate a comprehensive similarity, the matching degree of the overlapping areas of the master and slave videos can be accurately determined. This multi-dimensional matching approach significantly improves the accuracy of the splicing process, resulting in a smooth transition at the spliced ​​area and minimizing splicing artifacts. Setting a similarity threshold and performing a judgment allows for a quick decision on whether to proceed with the splicing operation, avoiding complex calculations and judgments, further improving the efficiency of video splicing and making the entire splicing process more automated and intelligent. Furthermore, this technical solution not only considers keyframe feature similarity but also combines time difference and location similarity for a comprehensive judgment. This multi-dimensional approach makes video splicing and fusion more flexible, adapting to the needs of different video splicing scenarios and reducing the occurrence of noticeable splicing artifacts after splicing and fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a first flow chart of an embodiment of the method of the present application;

[0069] Figure 2 This is a second flow chart of the method embodiment of the present application;

[0070] Figure 3 This is a third flow chart of the method embodiment of the present application;

[0071] Figure 4 This is a fourth flow chart of the method embodiment of the present application;

[0072] Figure 5 This is a fifth flow chart of the method embodiment of the present application;

[0073] Figure 6 This is the sixth flow chart of the method embodiment of the present application. DETAILED DESCRIPTION

[0074] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the appended drawings of the embodiments of the present invention. Figure 1 -Attached Figure 6 The technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0075] The first embodiment of the present application discloses a video splicing and fusion method. Figure 1 As an implementation of the video splicing and fusion method, the video splicing and fusion method may include S110-S190:

[0076] S110, synchronizing timestamps of videos received from the master device and the slave device;

[0077] S120, obtaining the overlapping area of ​​the videos on the master device and the slave device, marking the overlapping area on the master device as segment A, and marking the overlapping area on the slave device as segment B;

[0078] S130, extracting the feature vector of the key frame A1 of the segment A and the feature vector of the key frame B1 of the segment B;

[0079] S140, calculating key frame similarity based on the feature vector of key frame A1 and the feature vector of key frame B1;

[0080] S150, obtaining the time difference and location similarity between key frame A1 and key frame B1;

[0081] S160, calculating comprehensive similarity based on key frame similarity, time difference, and location similarity;

[0082] S170, determining whether the comprehensive similarity is greater than a similarity threshold;

[0083] S180, if yes, then splice and merge segment B with the previous frame of segment A;

[0084] S190: If not, generate a transition segment, and splice and fuse the transition segment with the A segment and the B segment.

[0085] Regarding S110, the step of synchronizing the timestamps of the videos received from the master device and the slave device specifically includes:

[0086] Control the master device's clock source to periodically broadcast the time reference to the slave device, and the slave device's clock performs time calibration based on the deviation compensation formula;

[0087] A frame with a timestamp is generated every 15 frames from the videos received from the master device and the slave device, and marked as a time frame to achieve synchronization of the timestamps.

[0088] Specifically, the transmission delay from the master clock of the master device to each slave device is measured; then the master clock source can broadcast the time reference through the PTPv2 protocol, and the slave device clock determines the adjustment deviation according to the deviation compensation formula, and then corrects the time according to the adjusted deviation. The deviation compensation formula is , average delay = ; is the timestamp of the master clock, is the timestamp of the slave clock, The delay from the master device to the slave device, is the delay from the device to the master device. , The timestamp is the corrected timestamp from the slave device. The master device and the slave device in this application can be understood as the master camera and the slave camera.

[0089] One implementation method for S120-S150 is:

[0090] Common image processing libraries (such as OpenCV) can be used to read corresponding frames from the master and slave video frames. SIFT feature points and descriptors are then extracted from each of the master and slave video frames. Descriptors represent the local feature information of the feature points. Feature matching is then performed using FLANN (Fast Nearest Neighbor Search), which efficiently finds matching feature point pairs in the master and slave video frames. Based on the matched feature point pairs, the RANSAC (Random Sample Consensus) algorithm is used to calculate the homography matrix H. The homography matrix H describes the projective transformation between two planes and can be used to determine the overlapping region between the video frames. The calculated overlapping region in the master video is labeled segment A, and the overlapping region in the slave video is labeled segment B.

[0091] The preprocessed key frame is input into the pre-trained ResNet50 model to extract the feature vector before its fully connected layer. The feature vector can represent the main feature information of the key frame.

[0092] Keyframe similarity , is the feature vector of key frame A1, is the feature vector of key frame B1.

[0093] One implementation method for S160-S180 is:

[0094] Calculate the time difference between segment A and segment B using video timestamps ,like , is the maximum acceptable time difference, then ;like ,but .

[0095] Using the location information (such as longitude and latitude) carried in the video, calculate the location distance d corresponding to segment A and segment B. If , is the maximum acceptable distance between locations, then ;like ,but Comprehensive similarity .

[0096] When the comprehensive similarity is greater than the similarity threshold, in order to avoid time misalignment of the overlapping area, an image processing library (such as OpenCV) can be used to read the previous frame of segment A from the video on the main device, which is recorded as , then directly replace the A fragment with the B fragment, with Perform splicing and fusion.

[0097] One implementation of generating a transition segment and splicing and fusing the transition segment with segment A and segment B may be:

[0098] The feature vectors of segments A and B are fed into the GAN generator to generate video frames with a transition effect. The SIFT (Scale-Invariant Feature Transform) algorithm is then used to extract feature points and descriptors from the transition segment's starting frame and the frame preceding segment A. FLANN (Fast Nearest Neighbor Search) is then used for feature matching to find matching feature point pairs. Based on these matching feature point pairs, the RANSAC (Random Sample Consensus) algorithm is used to calculate a homography matrix to geometrically align it with segment A. The transformed transition segment's starting frame is then fused with the frame preceding segment A using a multi-band fusion algorithm. This algorithm decomposes the image into sub-images of different frequency bands, fuses them separately, and then reconstructs the image, effectively reducing splicing artifacts. The ORB (Oriented Fast and Rotated Brief) algorithm is then used to extract feature points and descriptors from the transition segment's ending frame and segment B. A brute-force matcher is then used to perform feature matching to find matching feature point pairs. According to the matched feature point pairs, the transformation matrix is ​​calculated using the affine transformation algorithm, and the end frame of the transition segment is affine transformed to align it geometrically with the B segment. The fade-in and fade-out fusion method is used to perform gradient processing at the junction of the end frame of the transition segment and the B segment to make the transition more natural.

[0099] Reference Figure 2 After the comprehensive similarity is greater than the similarity threshold and before the overlapping regions are stitched together, the steps include S210-S240:

[0100] S210, obtaining the current coordinates of the selected matching point and the transformed coordinates of the selected matching point after being transformed by the homography matrix H;

[0101] S220, calculating a stitching reprojection error based on the current coordinates and the transformed coordinates;

[0102] S230, determining whether the stitching reprojection error is greater than an error threshold;

[0103] S240: If yes, optimize the homography matrix H.

[0104] Specifically, a pair or a certain number (e.g., 10-20) of matching points can be randomly selected from the matching point pairs obtained through the feature matching described above as selected matching points. This random selection can be achieved using a random number generator. For each selected matching point, its coordinates in the video frame on the master device are recorded, and then the homography matrix H is used to transform the current coordinates of the selected matching point.

[0105] Taking a pair of matching points as an example, the stitching reprojection error , is the transformed coordinate of the matching point after transformation, is the current coordinate. If multiple pairs of matching points are selected, the actual stitching reprojection error is the sum of squares or mean square error of the stitching reprojection errors of the multiple pairs of matching points. When the stitching reprojection error is greater than the error threshold, the homography matrix H is optimized.

[0106] Reference Figure 3 The steps after obtaining the overlapping area of ​​the master device and the slave device include S310-S340:

[0107] S310, using segment A as the target splicing area and determining whether segment B is jittering;

[0108] S320, if yes, obtain pixel motion vectors of two adjacent frames in the B segment;

[0109] S330, determining a homography matrix H according to the pixel motion vector;

[0110] S340: Eliminate jitter according to the inverse homography matrix.

[0111] Specifically, the Shi-Tomasi corner detection algorithm is used to detect feature points in the first frame of segment B. Shi-Tomasi corner detection is a commonly used feature point detection method that can detect corner points with obvious features in an image. The Lucas-Kanade optical flow algorithm is used to track the positions of these feature points in adjacent frames. The Lucas-Kanade optical flow algorithm is based on the assumption that the image grayscale remains unchanged and calculates the optical flow by calculating the displacement of feature points in adjacent frames. The average displacement of feature points between adjacent frames is calculated. If the average displacement is greater than a preset threshold (for example, set to 5 pixels based on experience), segment B is considered to have jittered.

[0112] The Farneback optical flow algorithm is used to calculate the optical flow between two adjacent frames in segment B. This algorithm is a global optical flow calculation method based on polynomial expansion and produces a relatively dense optical flow field. The motion vector of each pixel is extracted from the optical flow field. The motion vector contains information about the horizontal and vertical displacement of the pixel. A sufficient number of matching points (generally at least four pairs) are selected from the pixel motion vectors of the two adjacent frames. The matching points are selected based on the stability and uniformity of the feature points. The RANSAC (Random Sample Consensus) algorithm is then used to calculate the homography matrix H using the selected matching points. The RANSAC algorithm effectively eliminates incorrect matching points, improving the accuracy of the homography matrix H calculation. By calculating the inverse matrix of the homography matrix H, the jittered frames are transformed inversely to eliminate the jitter.

[0113] Reference Figure 4 , according to the inverse homography matrix, the steps before eliminating the jitter include S410-S470:

[0114] S410, calculating an optical flow error term based on the pixel motion vector;

[0115] S420, obtaining a rotation matrix of the current frame according to the rotation matrix of the previous frame;

[0116] S430, calculating a rotation smoothing term based on the rotation matrix of the previous frame and the rotation matrix of the current frame;

[0117] S440, obtaining translation smoothing items of the current frame and the previous frame;

[0118] S450, calculating the overall optimization target based on the optical flow error term, the rotation smoothing term, and the translation smoothing term;

[0119] S460, determining whether the total optimization target is less than the optimization threshold;

[0120] S470: If yes, eliminate the jitter according to the inverse homography matrix.

[0121] Specifically, the optical flow error term is , V is the pixel motion vector of two adjacent frames, The pixel motion vector of two adjacent frames when there is ideal no motion.

[0122] Estimate the rotation matrix of the previous and next frames and , feature point matching and posture estimation algorithms such as RANSAC can be used. The rotation smoothness term is used to constrain the rotation changes between adjacent frames to avoid severe rotation jitter. The smoothness of the rotation is measured by calculating the difference between the rotation matrices of two adjacent frames; the difference between the rotation matrices can be expressed by the Frobenius norm of the matrix. The rotation smoothness term is .

[0123] Convert the rotation matrix into a rotation vector, and then calculate the angle between the rotation vectors of two adjacent frames to measure the smoothness of the rotation. Suppose the rotation vector of the previous frame is , the rotation vector of the next frame is , rotational smoothing term .

[0124] Overall optimization goal = ,If the total optimization target is less than the optimization threshold, a reverse transformation is performed to eliminate the jitter.

[0125] Reference Figure 5 As another implementation of the video splicing and fusion method, the video splicing and fusion method may include S510-S530:

[0126] S510, calculating the RGB mean of segment A and segment B;

[0127] S520, establishing a linear transformation matrix T based on the color statistics of the overlapping area;

[0128] S530 , mapping segment B to the color space of segment A according to the original pixel values ​​of segment B, the linear transformation matrix T, the RGB mean values ​​of segment A and segment B, to obtain corrected pixel values.

[0129] Specifically, for segments A and B, three variables Rsum, Gsum, and Bsum are initialized to accumulate the pixel values ​​for each channel. A variable ‌Pixel Count‌ is also initialized to record the total number of pixels. A double loop is used to iterate over each pixel in segments A and B. For each pixel, the R, G, and B channel values ​​are obtained and added to the corresponding Rsum, Gsum, and Bsum. The accumulated Rsum, Gsum, and Bsum values ​​are divided by the total number of pixels, ‌Pixel Count, to obtain the RGB mean values ​​for segments A and B. Within the overlapping region, the color distribution of segments A and B is calculated separately. The color space can be divided into several intervals, and the number of pixels within each interval is counted. For each color channel (R, G, B), the corresponding relationship between the color distributions of segments A and B within the overlapping region is determined. For example, a least squares method can be used to fit a straight line that describes the linear relationship between the two color distributions. The linear transformation matrix T is constructed based on the slope and intercept of the fitted line. The linear transformation matrix T is a 3×3 diagonal matrix, with the diagonal elements corresponding to the transformation coefficients for the R, G, and B channels. A double loop is used to iterate over each pixel in the B segment. For each pixel's R, G, and B channel value, the RGB mean of the B segment is subtracted, then multiplied by the corresponding coefficients in the linear transformation matrix T. Finally, the RGB mean of the A segment is added to obtain the corrected pixel value. The corrected pixel value is clamped to the range [0, 255] to prevent out-of-range pixel values. If the pixel value is less than 0, it is set to 0; if the pixel value is greater than 255, it is set to 255.

[0130] Reference Figure 6 As another implementation of the video splicing and fusion method, the video splicing and fusion method may further include S610-S630:

[0131] S610, obtaining an optimized brightness value according to the target brightness value and a pre-built optimization objective function;

[0132] S620, calculating the error between the optimized brightness value and the target brightness value;

[0133] S630, determining whether the error value is less than an error threshold;

[0134] S640: If not, optimize the parameters of the objective function.

[0135] Specifically, the optimization objective function is ; Assuming the target brightness value is [100,105,110], then if , the optimization objective function is expanded to

[0136] , then Taking the derivative and setting it to zero, we get , , When the error between the optimized brightness value and the target brightness value is greater than or equal to the error threshold, the optimization can be performed. .

[0137] The implementation principle of this embodiment is:

[0138] The videos received from the master and slave devices are timestamped, and then the overlapping area of ​​the videos from the master and slave devices is obtained, and the overlapping area of ​​the master device is marked as segment A, and the overlapping area of ​​the slave device is marked as segment B; segment A is used as the target splicing area to determine whether segment B is jittering. If so, the pixel motion vectors of the two adjacent frames in segment B are obtained, and the homography matrix H is determined based on the pixel motion vectors; the optical flow error term is calculated based on the pixel motion vector, and the rotation matrix of the current frame is obtained based on the rotation matrix of the previous frame. The rotation smoothing term is calculated based on the rotation matrix of the previous frame and the rotation matrix of the current frame; the translation smoothing term of the current frame and the previous frame is obtained, and the total optimization target is calculated based on the optical flow error term, the rotation smoothing term and the translation smoothing term, and it is determined whether the total optimization target is less than the optimization threshold. If so, the jitter is eliminated based on the inverse homography matrix; then the image is extracted. The feature vector of key frame A1 of fragment A and the feature vector of key frame B1 of fragment B are obtained, and then the key frame similarity is calculated based on the feature vector of key frame A1 and the feature vector of key frame B1; the time difference and location similarity between key frame A1 and key frame B1 are obtained, and the comprehensive similarity is calculated based on the key frame similarity, time difference and location similarity, and it is judged whether the comprehensive similarity is greater than the similarity threshold. If so, the current coordinates of the selected matching point and the transformed coordinates of the selected matching point after transformation by the homography matrix H are obtained, and the splicing reprojection error is calculated based on the current coordinates and the transformed coordinates, and it is judged whether the splicing reprojection error is greater than the error threshold. If so, the homography matrix H is optimized until the splicing reprojection error is not greater than the error threshold, and then the B fragment is directly spliced ​​and fused with the previous frame of the A fragment; if not, a transition fragment is generated, and the transition fragment is spliced ​​and fused with the A fragment and the B fragment.

[0139] Based on the above method embodiments, the second embodiment of the present application discloses a video splicing and fusion system. The video splicing and fusion system of the embodiment of the present application can implement any of the above video splicing and fusion methods, and the specific working process of each module in the video splicing and fusion system can refer to the corresponding process in the above method embodiments.

[0140] For ease of understanding, an example is given below: A video splicing and fusion system includes:

[0141] A time synchronization module is used to synchronize the timestamps of the videos received from the master device and the slave device;

[0142] An overlapping region determination module is used to obtain the overlapping region of the videos on the master device side and the slave device side, and mark the overlapping region on the master device side as segment A, and mark the overlapping region on the slave device side as segment B;

[0143] A feature vector extraction module, used to extract the feature vector of the key frame A1 of the segment A and the feature vector of the key frame B1 of the segment B;

[0144] A calculation processing module is used to calculate the key frame similarity based on the feature vector of key frame A1 and the feature vector of key frame B1; the calculation processing module is also used to obtain the time difference and location similarity between key frame A1 and key frame B1, and calculate the comprehensive similarity based on the key frame similarity, time difference and location similarity;

[0145] A judgment module, used to judge whether the comprehensive similarity is greater than a similarity threshold;

[0146] The splicing and fusion module is used to splice and fuse the B segment with the previous frame of the A segment when the judgment module determines that the response is yes.

[0147] The third embodiment of the present application provides a terminal. As an implementation of the terminal, the terminal may include: a memory and a processor; wherein,

[0148] The memory is used to store the video splicing and fusion program;

[0149] The processor is used to execute the program stored in the memory to implement the steps of the above-mentioned video splicing and fusion method.

[0150] The memory may be communicatively connected to the processor via a communication bus, and the communication bus may be an address bus, a data bus, a control bus, or the like.

[0151] In addition, the memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0152] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0153] The above are all preferred embodiments of the present application and are not intended to limit the scope of protection of the present application. Unless otherwise specified, any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features. In other words, unless otherwise specified, each feature is merely an example of a series of equivalent or similar features.

Claims

1. A video splicing and fusion method, characterized in that: include: Synchronize the timestamps of the videos received from the master and slave devices; Obtaining an overlapping area of ​​the videos on the master device and the slave device, and marking the overlapping area on the master device as segment A, and marking the overlapping area on the slave device as segment B, wherein the segment B contains multiple consecutive frames and can perform inter-frame operations, including determining whether the segment B is jittery, and if so, obtaining pixel motion vectors of two adjacent frames in the segment B; Extracting a feature vector of the key frame A1 of the A segment and a feature vector of the key frame B1 of the B segment; Calculate the key frame similarity based on the feature vector of key frame A1 and the feature vector of key frame B1; Get the time difference and location similarity between key frame A1 and key frame B1; Calculating a comprehensive similarity based on the key frame similarity, the time difference and the location similarity; Determining whether the comprehensive similarity is greater than a similarity threshold; If yes, then the B segment is spliced ​​and fused with the previous frame of the A segment; If not, a transition segment is generated, and the transition segment is spliced ​​and fused with the A segment and the B segment.

2. A video splicing and fusion method according to claim 1, characterized in that: The step of synchronizing the timestamps of the videos received from the master device and the slave device specifically includes: Control the master device clock source to periodically broadcast the time reference to the slave device, and the slave device clock performs time calibration according to the deviation compensation formula; A frame with a timestamp is generated every 15 frames from the videos received from the master device and the slave device, and marked as a time frame to achieve synchronization of the timestamps.

3. The video splicing and fusion method according to claim 2, characterized in that: After the comprehensive similarity is greater than the similarity threshold and before the overlapping areas are spliced, the steps include: Get the current coordinates of the selected matching point and the transformed coordinates of the selected matching point after being transformed by the homography matrix H; Calculating a stitching reprojection error based on the current coordinates and the transformed coordinates; Determining whether the stitching reprojection error is greater than an error threshold; If so, optimize the homography matrix H.

4. The video splicing and fusion method according to claim 2, characterized in that: The steps after obtaining the overlapping area of ​​the master device end video and the slave device end video include: Taking the A segment as the target splicing area, determining whether the B segment is jittering; If yes, then obtaining pixel motion vectors of two adjacent frames in the B segment; Determine a homography matrix H according to the pixel motion vector; Eliminate jitter based on the inverse homography matrix.

5. A video splicing and fusion method according to claim 4, characterized in that: The steps before eliminating jitter according to the inverse homography matrix include: Calculating an optical flow error term according to the pixel motion vector; According to the rotation matrix of the previous frame, the rotation matrix of the current frame is obtained; Calculating a rotation smoothing term according to the rotation matrix of the previous frame and the rotation matrix of the current frame; Get the translation smoothing items of the current frame and the previous frame; Calculating an overall optimization objective according to the optical flow error term, the rotation smoothing term, and the translation smoothing term; Determining whether the total optimization target is less than an optimization threshold; If so, the jitter is eliminated according to the inverse homography matrix.

6. The video splicing and fusion method according to claim 1, characterized in that: The video splicing and fusion method further includes: Calculate the RGB mean of the A segment and the B segment; Based on the color statistics of the overlapping area, a linear transformation matrix T is established; According to the original pixel value of the B segment, the linear transformation matrix T, the RGB mean value of the A segment, and the RGB mean value of the B segment, the B segment is mapped to the color space of the A segment to obtain a corrected pixel value.

7. The video splicing and fusion method according to claim 1, characterized in that: The video splicing and fusion method further includes: According to the target brightness value and the pre-built optimization objective function, an optimized brightness value is obtained; Calculating the error between the optimized brightness value and the target brightness value; Determining whether the error value is less than an error threshold; If not, the parameters of the optimization objective function are optimized.

8. A video splicing and fusion system, characterized in that: The video splicing and fusion method according to any one of claims 1 to 7 is executed, wherein the video splicing and fusion system comprises: A time synchronization module is used to synchronize the timestamps of the videos received from the master device and the slave device; an overlapping region determination module, configured to obtain an overlapping region of the videos on the master device and the slave device, and mark the overlapping region on the master device as segment A, and mark the overlapping region on the slave device as segment B, wherein the segment B contains multiple consecutive frames and is capable of performing inter-frame operations, including determining whether jitter occurs in the segment B and, if so, obtaining pixel motion vectors of two adjacent frames within the segment B; A feature vector extraction module, configured to extract a feature vector of the key frame A1 of the segment A and a feature vector of the key frame B1 of the segment B; a calculation processing module, configured to calculate key frame similarity based on a feature vector of key frame A1 and a feature vector of key frame B1; the calculation processing module is further configured to obtain a time difference and a location similarity between key frame A1 and key frame B1, and calculate a comprehensive similarity based on the key frame similarity, the time difference, and the location similarity; A judgment module, configured to judge whether the comprehensive similarity is greater than a similarity threshold; The splicing and fusion module is used to splice and fuse the B segment with the previous frame of the A segment when the judgment module judges to be yes; and is used to generate a transition segment and splice and fuse the transition segment with the A segment and the B segment when the judgment module judges to be no.

9. A terminal, characterized in that: include: A memory storing a video splicing and fusion program; A processor is used to execute the program stored in the memory to implement the steps of the video splicing and fusion method according to any one of claims 1 to 7.

Images