A method and device for splicing multiple videos, an electronic device, and a storage medium

Abstract

Embodiments of the application disclose a multi-path video splicing method and device, electronic equipment and a storage medium. The method comprises the following steps: firstly, at least one to-be-processed video is parsed to obtain a video frame set; secondly, in the video frame set, M key frame image pairs are determined, each key frame image pair comprising one reference image and one to-be-matched image; thirdly, feature points in the M key frame image pairs are extracted to obtain a feature point set and a descriptor set comprising descriptors corresponding to the feature points in the feature point set; fourthly, a stitching line matched with part or all of the key frame image pairs is generated according to the feature point set and the descriptor set; fifthly, a wide field video of the corresponding key frame image pair is determined by applying each stitching line; and finally, the wide field videos are spliced based on the time stamps of the wide field videos. The embodiments of the application can realize real-time splicing of multi-path videos while ensuring the splicing quality and visual effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for splicing multiple video streams. Background Technology

[0002] With the development of digital twin technology, video stitching technology, as one of the core technologies for realizing panoramic monitoring and virtual scene construction, is widely used in smart cities, industrial manufacturing, security monitoring, and other fields. Digital twin technology maps data from the physical world to the virtual world, enabling real-time monitoring and prediction of physical systems. Video stitching technology, as a key component in this process, registers, fuses, and stitches video images captured by multiple cameras to generate panoramic videos with a wide field of view, providing more complete and accurate visual information for digital twin systems.

[0003] However, in practical applications, due to the limited field of view of a single shooting device, multiple shooting devices need to be deployed to work together to achieve panoramic monitoring of large scenes. Currently, there are three main ways to acquire panoramic video: first, by adding a rotating pan-tilt unit to the shooting device to obtain multi-angle information, but this method cannot guarantee temporal and spatial consistency; second, by deploying multiple shooting devices to simultaneously capture videos from different angles, but simple image stacking appears cluttered and lacks spatial continuity; and third, by using a panoramic camera to directly capture 360-degree images, but these devices typically use fisheye lenses, which suffer from severe distortion and are costly.

[0004] Therefore, there is an urgent need for an efficient video splicing technology solution that can achieve real-time splicing processing of multiple video streams while ensuring splicing quality and visual effects. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for splicing multiple video streams, which enables real-time splicing processing of multiple video streams while ensuring splicing quality and visual effects.

[0006] In a first aspect, one embodiment of this application provides a method for stitching together multiple video streams, including: At least one video to be processed is parsed to obtain a set of video frames; In the video frame set, M keyframe image pairs are determined; each keyframe image pair includes a reference image and a matching image, the timestamp of each reference image is earlier than the timestamp of the corresponding matching image; the time difference between the timestamp of each reference image and the timestamp of the corresponding matching image is less than a set time threshold; the distance between the source shooting device of each reference image and the source shooting device of the corresponding matching image is less than a set distance threshold. Feature points are extracted from M keyframe image pairs to obtain a set of feature points and a set of descriptors consisting of descriptors that correspond one-to-one with the feature points in the set. Based on the feature point set and descriptor subset, generate a stitching line matching some or all keyframe image pairs; Apply each stitch to determine the corresponding keyframe image pair in the wide field of view video; Based on the timestamps of each wide field of view video, the wide field of view videos are stitched together.

[0007] Secondly, one embodiment of this application provides a multi-channel video splicing device, comprising: The processing unit is used to: parse at least one video to be processed to obtain a set of video frames; The processing unit is further configured to: determine M keyframe image pairs in the video frame set; wherein each keyframe image pair includes a reference image and a matching image, the timestamp of each reference image is earlier than the timestamp of the corresponding matching image; the time difference between the timestamp of each reference image and the timestamp of the corresponding matching image is less than a set time threshold; and the distance between the source shooting device of each reference image and the source shooting device of the corresponding matching image is less than a set distance threshold. The processing unit is also used to: extract feature points from M keyframe image pairs to obtain a set of feature points and a set of descriptors consisting of descriptors that correspond one-to-one with the feature points in the set of feature points; The processing unit is also used to: generate a stitching line matching some or all keyframe image pairs based on the feature point set and the descriptor set; The processing unit is also used to: apply each stitch to determine the wide field-of-view video of the corresponding keyframe image pair; The stitching unit is used to stitch together the various wide-field-of-view videos based on their timestamps.

[0008] Thirdly, one embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0009] Fourthly, one embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of any of the above methods.

[0010] Fifthly, one embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0011] In this embodiment, at least one video to be processed is first parsed to obtain a set of video frames. Within this set, M keyframe image pairs are determined. This allows processing only a portion of the video frames, reducing computational complexity and improving processing efficiency. Next, feature points are extracted from the M keyframe image pairs, resulting in a feature point set and a descriptor set consisting of descriptors corresponding to each feature point in the feature point set. This ensures registration accuracy while adapting to real-time processing requirements. Based on the feature point set and descriptor set, a stitching line is generated to match some or all of the keyframe image pairs. Then, each stitching line is applied to determine the wide-field-of-view video of the corresponding keyframe image pair, which solves the stitching artifact problem caused by moving objects in dynamic scenes and improves fusion quality. Furthermore, the timestamps of each wide-field-of-view video are used. Finally, the wide-field-of-view videos are stitched together to achieve high-quality real-time panoramic integration of multiple video streams, meeting the real-time requirements of practical applications. Therefore, this embodiment can achieve real-time stitching processing of multiple video streams while ensuring stitching quality and visual effects. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a method for splicing multiple video streams according to an embodiment of this application; Figure 2 A schematic diagram of the structure of a multi-channel video splicing device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0015] Definitions: (1) Stitching line: refers to the boundary line used to connect images from adjacent shooting devices during the multi-channel video stitching process. Its function is to ensure the continuity and visual consistency of the stitched image. Its position is located in the overlapping area of ​​the images, determining which pixels come from which shooting device.

[0016] The number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinction only and has no limiting meaning.

[0017] Against the backdrop of the booming development of digital twin technology, video stitching technology, as one of the core technologies for realizing panoramic monitoring and virtual scene construction, is being widely applied in various fields such as smart cities, industrial manufacturing, and security monitoring. Digital twin technology maps data from the physical world to the virtual world, enabling real-time monitoring and prediction of physical systems. Video stitching technology, as a key component in this process, registers, fuses, and stitches video images captured by multiple cameras to generate panoramic videos with a wide field of view, providing more complete and accurate visual information for digital twin systems.

[0018] However, in practical applications, due to the limited field of view of a single camera, achieving panoramic monitoring of large scenes requires the deployment of multiple cameras working in tandem. Traditional multi-view monitoring methods suffer from problems such as fragmented images, spatial discontinuity, and a lack of overall coherence, failing to meet the demands of modern monitoring systems for panoramic, real-time, and high-quality video. Furthermore, due to differences in position and angle, as well as lens distortion, the video images captured by different cameras are prone to inaccurate registration, obvious stitching seams, ghosting, and other phenomena during stitching. This is especially challenging in dynamic scenes, where handling moving targets in overlapping video areas presents a significant challenge.

[0019] In existing technologies, there are three main ways to acquire panoramic video: First, by adding a rotating gimbal device to the shooting equipment to obtain multi-angle information, but this method cannot guarantee the consistency of time and space; second, by deploying multiple shooting devices to shoot videos from different angles at the same time, but the simple stacking of images appears cluttered and lacks spatial continuity; and third, by using a panoramic camera to directly capture 360-degree images, but these devices usually use fisheye lenses, which have serious distortion and are expensive.

[0020] Therefore, there is an urgent need for an efficient video stitching method that can achieve real-time stitching of multiple video streams while ensuring stitching quality and visual effects. This method needs to address issues such as long image registration time, low feature matching rate, weak image texture features, obvious stitching seams, and ghosting. By optimizing feature point extraction and matching algorithms, improving image fusion strategies, and optimizing the search for the optimal stitching line, this method achieves high-quality, low-latency, and real-time panoramic video stitching effects. This will provide better panoramic visual data support for digital twin systems and promote technological development and application innovation in related fields such as intelligent surveillance and virtual reality.

[0021] Based on the aforementioned deficiencies in the existing technology, this application aims to address the following core technical issues: 1) Achieve efficient intelligent keyframe extraction from videos. To address this problem, we propose an intelligent keyframe extraction method that combines deep learning object detection and segmentation. The YOLOv5s-seg model is used to achieve high-quality keyframe selection, reduce redundant computation, and ensure that key information is not lost.

[0022] 2) Improve the accuracy and efficiency of image registration. To address this problem, an improved SURF feature extraction algorithm is proposed. By introducing HOG features, region limitation, dimensionality reduction, gridding strategy, and accelerated computation optimization, the algorithm significantly improves computational efficiency while ensuring registration accuracy.

[0023] 3) Achieve high-quality image fusion in dynamic scenes. To address this issue, a video fusion algorithm based on suture line updates is proposed. Through multimodal motion detection, intelligent suture line dynamic planning, and adaptive image fusion processing, the suture line is dynamically adjusted to bypass moving objects, avoiding artifacts and image discontinuities.

[0024] 4) Achieve real-time panoramic integration of multiple video streams. To address this issue, an efficient real-time stitching system is constructed. Through time synchronization and cache management, time-aligned panoramic stitching processing, and real-time panoramic stream encapsulation and output, high-quality real-time panoramic integration of multiple video streams is achieved.

[0025] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0026] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0027] The technical solutions provided in the embodiments of this application are described below.

[0028] refer to Figure 1 This application provides a method for stitching together multiple video streams, including the following steps: S101: Parse at least one video to be processed to obtain a set of video frames.

[0029] S102: In the set of video frames, identify M keyframe image pairs.

[0030] S103: Extract feature points from M keyframe image pairs to obtain a set of feature points and a set of descriptors consisting of descriptors that correspond one-to-one with the feature points in the set.

[0031] S104: Generate stitching lines for matching some or all keyframe image pairs based on the feature point set and descriptor set.

[0032] S105: Apply each stitch to determine the wide field of view video of the corresponding keyframe image pair.

[0033] S106: Based on the timestamps of each wide field of view video, stitch together the various wide field of view videos.

[0034] In this embodiment, at least one video to be processed is first parsed to obtain a set of video frames. Within this set, M keyframe image pairs are determined. This allows processing only a portion of the video frames, reducing computational complexity and improving processing efficiency. Next, feature points are extracted from the M keyframe image pairs, resulting in a feature point set and a descriptor set consisting of descriptors corresponding to each feature point in the feature point set. This ensures registration accuracy while adapting to real-time processing requirements. Based on the feature point set and descriptor set, a stitching line is generated to match some or all of the keyframe image pairs. Then, each stitching line is applied to determine the wide-field-of-view video of the corresponding keyframe image pair, which solves the stitching artifact problem caused by moving objects in dynamic scenes and improves fusion quality. Furthermore, the timestamps of each wide-field-of-view video are used. Finally, the wide-field-of-view videos are stitched together to achieve high-quality real-time panoramic integration of multiple video streams, meeting the real-time requirements of practical applications. Therefore, this embodiment can achieve real-time stitching processing of multiple video streams while ensuring stitching quality and visual effects.

[0035] Regarding S101, the videos to be processed can be in formats such as MP4 or AVI. For example, each video to be processed is a single video file, and at least one video to be processed can be three video files (e.g., represented as camera1.mp4, camera2.mp4, and camera3.mp4). Using OpenCV video decoding tools, the three video files are read and decoded sequentially. Each video file is then decomposed into a continuous sequence of frames arranged in chronological order, and each frame is stored in grayscale format (frame_sequence_1, frame_sequence_2, frame_sequence_3), thus generating a video frame sequence (which can be called a video frame set) that can be used for subsequent processing, providing basic data support for subsequent keyframe extraction. In a specific example, each video file yields 1000 video frames (or 1000 frames of images), so the three video files will yield 3000 video frames (or 3000 frames of images). This is just an example and does not constitute a specific limitation.

[0036] The video frame set can then be filtered to initially extract the keyframe set. For example, this filtering process can be implemented as follows: Image preprocessing techniques are used to scale and normalize each frame of each video stream, generating normalized images in a uniform format to ensure they meet the input requirements of the YOLOv5s-seg model, resulting in preprocessed images (e.g., approximately 3000 preprocessed frames). These preprocessed images are then input into the YOLOv5s-seg model. In the model, a backbone network (CSPDarknet) is used to extract multi-scale feature maps of the image. For each frame, multiple feature maps of different scales (such as 16×16, 32×32, 64×64, etc.) are generated, resulting in a multi-scale feature map (more than 3,000 feature maps, for example, 10,000 feature maps). Then, a neck network (PANet) is used to perform feature fusion on the multi-scale feature maps to obtain a fused feature map (for example, 8,000 feature maps). Then, a detection head is used to perform bounding box and class prediction on the fused feature map to generate bounding boxes and class labels for potential targets in each frame, thus obtaining the detection results. At the same time, a segmentation head is used to perform pixel-level segmentation mask generation on the fused feature map to generate a pixel-level segmentation mask for each potential target. Based on the target detection and segmentation results, the importance score of each frame is calculated. Finally, based on the importance score and temporal distribution, a non-redundant selection strategy is used to extract keyframes from the frame importance score, resulting in a set of keyframes containing important visual information (for example, extracting 2,700 video frames as keyframes from 3,000 video frames).

[0037] In this embodiment, the YOLOv5s-seg deep learning model is used for object detection and segmentation, achieving end-to-end feature learning. Based on a multi-dimensional importance evaluation mechanism considering factors such as the number of targets, category richness, and degree of motion variation, a non-redundant selection strategy ensures the representativeness and diversity of the keyframe set. Combined with spatiotemporal matching analysis, keyframe image pairs are automatically constructed, providing optimized input for subsequent registration. Compared to traditional keyframe extraction methods based on inter-frame difference or histogram changes, this approach can more accurately identify frames with important visual information, significantly reduce the amount of data required for subsequent processing, lower the computational complexity of video processing, improve processing efficiency, and simultaneously ensure the integrity of key information.

[0038] In S102, M key frame image pairs are determined from the set of key frames selected from the set of video frames.

[0039] Each keyframe image pair includes a reference image and a matching image. The timestamp of each reference image is earlier than the timestamp of the corresponding matching image. The time difference between the timestamp of each reference image and the timestamp of the corresponding matching image is less than a set time threshold. The distance between the source camera of each reference image and the source camera of the corresponding matching image is less than a set distance threshold.

[0040] In a specific example, the reference image, which serves as the standard image for registration, is typically an image with an earlier timestamp or a fixed position in adjacent shooting devices; the image to be matched, which needs to be aligned with the reference image, is typically an image with a relatively later timestamp or an adjacent position in adjacent shooting devices.

[0041] Optionally, this process can be achieved through steps A1-A4: A1: Sort the video frames in the video frame set according to time and calculate the similarity between adjacent video frames.

[0042] In practice, if the video frame set is filtered, this video frame set refers to the keyframe set, such as the keyframe set of 2700 images mentioned in the above embodiment. Using time-series sorting technology, the keyframe set is sorted chronologically to establish the correct time-series relationship, resulting in a time-sorted keyframe set.

[0043] Then, the similarity between adjacent video frames is calculated using the feature vector distance metric.

[0044] A2: Based on the obtained similarity, each video frame in the video frame set is filtered to obtain the first video frame set.

[0045] Optionally, based on the calculated similarity, similar redundant frames are identified and removed to obtain a deduplicated and optimized set of keyframes, such as 2400 frames.

[0046] A3: Perform time uniformity adjustment on each video frame in the first video frame set to obtain the second video frame set.

[0047] Specifically, a temporal uniformity distribution technique can be used to adjust the temporal distribution of keyframes in the first video frame set. This prevents keyframes from being overly concentrated in a particular time period, ensuring that representative frames are obtained throughout the entire video frame sequence comprised of the first video frame set. This results in a temporally optimized keyframe set, referred to as the second video frame set. In this second video frame set, the difference in the number of video frames within adjacent time periods is less than a set threshold, thus achieving a uniform distribution of keyframes.

[0048] A4: Based on the timestamps of each video frame in the second video frame set and the location of the source shooting device, determine M keyframe image pairs.

[0049] For example, by using multi-channel video spatiotemporal matching analysis technology, spatiotemporal matching analysis is performed on each video frame in the second video frame set. Keyframes from different shooting devices with similar timestamps are automatically paired according to the spatial adjacency of the shooting devices to construct keyframe image pairs for image registration. The reference image and the image to be matched in each pair are determined to obtain a set of keyframe image pairs (for example, represented by M keyframe image pairs, where M can be 1000, that is, there are 1000 pairs of reference images and their paired images to be matched), providing input data for subsequent image registration processing.

[0050] Regarding S103, after obtaining M keyframe image pairs, feature points are extracted from the M keyframe image pairs to obtain a set of feature points and a set of descriptors consisting of descriptors that correspond one-to-one with the feature points in the feature point set.

[0051] Optionally, this process can be achieved through steps B1-B4: B1: Fuse the directional gradient histogram information into the M keyframe image pairs to obtain the processed M keyframe image pairs.

[0052] Optionally, the Histogram of Oriented Gradients (HOG) feature fusion technique is used to construct the Hessian matrix of the M keyframe image pairs (reference image and image to be matched). In the Hessian matrix construction step, HOG feature information is fused to enhance the feature point description capability, resulting in HOG-SURF fused feature objects, which are the processed M keyframe image pairs.

[0053] B2: Determine the extraction region, extract P feature points from the processed M keyframe image pairs within the extraction region, and calculate P descriptors corresponding to the P feature points.

[0054] For example, the feature point detection region limitation technique is used to perform region constraint operation on the processed M keyframe image pairs, and the feature point extraction range is constrained to the image overlap area (that is, the extraction area, which is usually the 30%-50% overlap area of ​​the image). For the left image, feature points are extracted only in the right region, and for the right image, feature points are extracted only in the left region, resulting in a set of P feature points (for example, 10,000 feature points).

[0055] After obtaining P feature points, the dimension of the P feature points within the region is reduced by using dimensionality reduction technology. Circular neighborhoods are used instead of traditional square neighborhoods to calculate the descriptor, reducing the dimension of the feature descriptor from the traditional 64 dimensions to 20 dimensions, resulting in P dimensionality-reduced feature descriptors.

[0056] B3: Perform gridding uniform processing on P feature points to obtain Q feature points.

[0057] Optionally, a gridded distribution constraint technique is used to perform a gridded distribution operation on the Q descriptors after dimensionality reduction. This divides the detection region into regular grids and controls the density of feature points within each grid to achieve a uniform distribution of feature points, resulting in gridded feature points (e.g., filtering 10,000 feature points and leaving 9,000). Subsequently, an accelerated computation optimization technique is used to perform accelerated computation on the gridded feature points. Integral image and box filter techniques are used to replace traditional Gaussian filtering calculations, enabling fast approximate calculations in the Hessian matrix calculation and feature point response value evaluation processes, resulting in accelerated feature points (e.g., 9,000 optimized feature points). In this way, the Q feature points are located within the region formed by the defined grid.

[0058] B4: Recalculate the Q new descriptors corresponding to the Q feature points.

[0059] For example, the principal direction calculation technique is used to calculate the principal direction of the accelerated feature points (Q feature points). The gradient direction distribution in the neighborhood of each feature point is statistically analyzed to determine the principal direction of each feature point, generating rotation-invariant feature descriptors (9000 new descriptors). This results in an improved SURF feature point set and its corresponding 20-dimensional descriptors, providing high-quality feature representations for subsequent feature matching. Thus, the Q feature points constitute the feature point set, and the Q new descriptors constitute the descriptor set.

[0060] In S104, stitching lines are generated for matching some or all keyframe image pairs based on the feature point set and descriptor set.

[0061] Optionally, the process of generating partial or complete keyframe image pairs to match the seam lines can be achieved through steps C1-C5: C1: Based on the descriptive subset, the feature points in the feature point set are paired to obtain a set of matching point pairs.

[0062] In practical applications, the process of determining the set of matching point pairs can be achieved through steps C1-1 to C1-4: C1-1: Based on Q feature points and Q descriptors, determine the first set of matching feature points for each reference image feature point, and the second set of matching feature points for each feature point to be matched.

[0063] Specifically, the Fast Library for Approximate Nearest Neighbors (FLANN) algorithm is used to perform a fast nearest neighbor search on the SURF feature points of the reference image and the image to be matched. Euclidean distance is used as a similarity metric to evaluate the similarity between feature points, and the fast nearest neighbor search results are obtained. That is, the first set of matching feature points for each feature point of the reference image and the second set of matching feature points for each feature point to be matched are determined.

[0064] In this case, the similarity between each feature point in the first feature point set and the feature points in the reference image is less than a set similarity threshold, and the similarity between each feature point in the second feature point set and the feature points in the image to be matched is less than a set similarity threshold.

[0065] In a specific example, the correspondence between image frames and feature points is known. That is, for any feature point in a reference image, its nearest neighbors, for example, 100 feature points belonging to the image to be matched, are found as the nearest neighbor search results, i.e., the first set of feature points.

[0066] C1-2: In the first set of feature points, determine the target feature points of the target image to be matched for feature point matching of the reference image.

[0067] This process is called the forward matching process. The forward matching technique is used to perform forward matching on the fast nearest neighbor search results. The feature points of the reference image are used as query points. The nearest and second nearest neighbors of each query point are searched in the feature point set (first feature point set) of the image to be matched. The ratio of these two distances is calculated and a threshold is set to filter out high-quality potential matching points (target feature points of the target image to be matched), and the forward matching results are obtained.

[0068] C1-3: In the second set of feature points, determine the target feature points of the target reference image to match the feature points of the image to be matched.

[0069] This process is called the reverse matching process. The reverse matching technique is used to perform a reverse matching operation on the forward matching result, and the matching direction is exchanged. The feature points of the image to be matched are used as query points. The nearest neighbor and the second nearest neighbor points are also searched in the feature point set (second feature point set) of the reference image. The same distance ratio threshold strategy is used to filter potential matching points (target feature points of the target reference image) to obtain the reverse matching result object.

[0070] C1-4: Based on the target feature points matched in each reference image and the target feature points matched in each image to be matched, determine the set of matching point pairs.

[0071] Specifically, the bidirectional matching verification technique performs a consistency check on the reverse matching results, verifying both the forward and reverse matching results. Only those point pairs that successfully match in both directions are retained, resulting in a set of matched point pairs. This bidirectional verification mechanism significantly reduces the number of false matches and improves the reliability of the matching results. In a specific example, the set of matched point pairs may include 500 successfully matched feature points.

[0072] In practical applications, the above set of matching point pairs can be a coarse matching result, and it can be filtered and optimized to obtain a precise set of matching point pairs. Optionally, the filtering and optimization process is as follows: The grid-based motion statistics (GMS) algorithm is used to remove mismatches from the coarse matching point pair set (e.g., 500 feature points). The grid-based statistical analysis method is used to identify and remove unreliable matching points, resulting in a mismatch removal result (e.g., 400 feature points). Next, the mismatch point removal results are divided into a grid using a grid partitioning technique. The image region to be processed is divided into a regular grid structure, typically using a 40×40 grid partitioning scheme. The distribution of matching points within each grid cell is statistically analyzed in detail to establish a gridded statistical model of matching point distribution, resulting in a gridded statistical result (e.g., 300 pairs of feature points). Then, the reliability of the gridded statistical result is evaluated using motion consistency constraint evaluation technology. Based on the principle of motion consistency constraint, the motion vector of each matching point pair between adjacent frames is calculated, the number of matching points with the same motion pattern is counted, and isolated matching points with inconsistent motion patterns are removed, resulting in a purified set of accurate matching point pairs (e.g., 200 pairs of feature points). Finally, the purified set of accurate matching point pairs is optimized using the least squares method to estimate parameters. The homography matrix is ​​calculated by optimizing the parameter estimation process using the least squares method, resulting in a set of accurate matching point pairs (e.g., 100 pairs of feature points).

[0073] In this embodiment, the SURF algorithm is improved by introducing HOG features to enhance descriptive capabilities and improve the rotation and scale invariance of feature points. A strategy for limiting the feature point detection region restricts the detection range to overlapping areas, reducing unnecessary computation. Circular neighborhood dimensionality reduction reduces the descriptor from 64 dimensions to 20 dimensions, improving computational efficiency. A gridded distribution constraint is introduced to achieve uniform distribution of feature points, avoiding excessive local concentration. A combined strategy of bidirectional matching FLANN and GMS fine matching improves matching accuracy and robustness, significantly increasing computational efficiency while maintaining registration accuracy, thus meeting real-time processing requirements.

[0074] C2: Based on the pre-defined correspondence between feature points and keyframe images, determine the reference keyframe image pairs corresponding to the matching point pair set.

[0075] Taking the optimization of the coarse matching result as an example, the set of matching point pairs refers to the set of precise matching point pairs in the above embodiment. Since it is known which keyframe image the feature points come from, that is, the object relationship between the feature points and the keyframe images is preset, the reference keyframe image pairs (e.g., 100 reference keyframe image pairs) corresponding to the set of precise matching point pairs can be determined based on this preset correspondence.

[0076] C3: For each reference keyframe image pair, determine the first information of the moving objects included in the reference keyframe image pair by calculating the historical Gaussian distribution characteristics of each pixel, determine the second information of the moving objects included in the reference keyframe image pair by calculating the motion vector of each pixel, and determine the third information by classifying the moving objects included in the reference keyframe image pair.

[0077] Optionally, the first information represents the identifier of the moving object, and the calculation process of the first information is as follows: Background modeling is performed on keyframe image pairs (e.g., 100 keyframe pairs or 100 keyframe image pairs) after image registration by a background elimination algorithm based on Gaussian mixture model. The foreground and background regions are distinguished by statistically analyzing the historical Gaussian distribution features of each pixel. Moving objects in the static background are extracted, and preliminary foreground detection results are generated. The resulting Gaussian mixture background modeling result is called the first information.

[0078] For example, the second information represents the motion of the moving object, and the calculation process of the second information is as follows: Motion estimation is performed on keyframe image pairs (e.g., 100 keyframe pairs or 100 keyframe image pairs) after image registration using optical flow. Motion vectors of pixels between adjacent frames are calculated to detect sub-pixel level motion information, and the optical flow field motion detection result is obtained. This detection result is called the second information.

[0079] For example, the third information represents the motion of different categories of moving objects, and the calculation process of the third information is as follows: A deep learning object detection model is used to perform semantic recognition on keyframe image pairs (e.g., 100 keyframe pairs) after image registration. By classifying and locating moving objects in the images, the detection accuracy of moving objects in complex scenes is improved, generating semantic-level moving object detection results. This semantic-level detection result is called the third information, which represents higher-level motion information. For example, it can not only identify the existence of objects, but also the category of objects.

[0080] C4: Fuse the first, second, and third information to obtain a binarized foreground marker image and a reliability assessment map.

[0081] Optionally, the results of the three detection methods can be integrated through a weighted fusion strategy to generate a high-precision binary foreground marker image (e.g., 100 binary foreground marker images), in which the area with a pixel value of 1 represents the detected moving object and the area with a pixel value of 0 represents the background.

[0082] Reliability assessment techniques are used to perform reliability mapping on high-precision binarized foreground marker images. Based on the high-precision binarized foreground marker images generated during the fusion process, a reliability assessment map of the overlapping area is established. Each pixel is assigned a confidence value to evaluate the confidence level of each region in the fusion process, resulting in a reliability assessment map (e.g., 100 reliability assessment maps), which provides a decision-making basis for subsequent suture planning.

[0083] C5: Apply binarized foreground marker images and reliability assessment maps to determine the suture line for matching reference keyframe images.

[0084] Optionally, the process of determining the seam line matching the reference keyframe image can be implemented through steps C5-1 to C5-3: C5-1: Perform a query operation on the binarized foreground marker image and the reliability assessment map to determine whether the stitching line used in the previous frame reference keyframe image pair intersects with the foreground region of the current frame. If so, predict the motion trend of the object based on the motion information and velocity vector of the moving object in the previous frame reference keyframe image pair, determine the avoidance range based on the motion trend, and generate kinematic constraints based on the avoidance range.

[0085] Optionally, a query operation is performed on the high-precision binarized foreground marker image and the reliability assessment map using motion detection analysis technology. This checks whether the optimal stitching line used in the previous frame reference keyframe image intersects with the foreground region of the current frame. The necessity of updating the stitching line is determined by combining the credibility information in the reliability assessment map. When a moving object is detected crossing the existing stitching line or the region credibility changes significantly, it is determined that the stitching line needs to be replanned.

[0086] Then, the motion trend prediction operation is performed on the above-mentioned suture line update logic through the motion trajectory prediction mechanism. The motion information and velocity vector of the moving object in the historical frame are analyzed to predict the future motion trend of the object. Based on the prediction results, the range of the area that the suture line needs to avoid is determined, and kinematic constraints are provided for the subsequent suture line replanning.

[0087] In the above embodiments, historical frames refer to all frames processed before the current frame. Their function is to provide historical trajectory information of moving objects for predicting future motion trends. Data sources include extracted keyframes and planned stitching lines. The stitching line of the previous frame refers to the optimal stitching line determined during the processing of the previous frame image. Its function is to serve as a reference benchmark for planning the stitching line in the current frame. The initial value setting principle for the stitching line of the first frame is that the system will use a default strategy (such as the middle position, fixed ratio, etc.) to determine the initial stitching line. Subsequently, as subsequent frames are processed, the stitching line will be dynamically adjusted according to the actual scene.

[0088] C5-2: Generating loss functions based on kinematic constraints.

[0089] Optionally, a comprehensive loss function construction technique is used to integrate kinematic constraints, constructing a comprehensive loss function containing multiple constraints for suture optimization evaluation. This loss function integrates spatial coherence, saliency, moving object penalty, and visual attention constraint. The moving object penalty applies a high-cost penalty to the path traversing the motion region using a high-precision binarized foreground marker image. The saliency term uses image gradient information to prevent the suture from passing through complex texture regions. The spatial coherence term ensures the smoothness of the path and avoids drastic jumps. The visual attention constraint term optimizes path selection based on human visual characteristics. Finally, a mathematical model for comprehensively evaluating suture quality is obtained, resulting in the comprehensive loss function, which can also be called the loss function.

[0090] C5-3: The suture line obtained when the loss value of the loss function is minimized is the matching suture line.

[0091] The optimal path search operation is performed on the comprehensive loss function using dynamic programming algorithm. The optimal stitching line is searched in the overlapping area of ​​the image. The comprehensive loss function constructed above is used as the path evaluation standard. The cumulative cost of each candidate path under the mathematical model is calculated. The path solution that minimizes the loss function is solved by iterative optimization method. In this way, the best stitching line that satisfies all constraints and maintains the smooth and natural characteristics of the path is found, and the final optimized stitching line is generated.

[0092] Regarding S105, each stitch is used to determine the wide field of view video of the corresponding keyframe image pair.

[0093] Optionally, this process can be achieved through steps D1-D3: D1: Based on the object motion information provided by the stitching path, and the texture features of each region in each pair of keyframe images, determine the fusion algorithm to be used for each region.

[0094] Optionally, an adaptive fusion strategy selection technique is used to perform region analysis on the optimized seam line and the keyframe image pair after image registration. Based on the moving object position information provided by the optimized seam line and the image texture features in the keyframe image pair, the texture characteristics and motion state of each region of the image are analyzed, and the most suitable fusion algorithm is dynamically selected. For static high-texture regions, the optimal seam line algorithm is used to maintain detail clarity. For dynamic regions, a fade-in / fade-out weighted average algorithm is used to avoid motion blur. For low-texture regions, a multi-band fusion algorithm is used to enhance the smooth transition effect, thus obtaining an adaptive fusion strategy.

[0095] D2: Apply a fusion algorithm to calculate the offset of each keyframe image pair.

[0096] Optionally, a unified computational framework for adaptive fusion strategies is constructed using the MVC image fusion algorithm based on convolutional pyramids. Small-sized convolutional kernels are used to approximate the convolutional effect of large-sized kernels to reduce computational complexity while maintaining good fusion quality. The MVC algorithm uses the mean coordinate interpolation method to calculate the offset distribution within image blocks, providing efficient numerical computation support for different fusion strategies and resulting in a general offset calculation framework.

[0097] Then, boundary condition constraint definition techniques are used to set boundary conditions for the general offset calculation framework. Based on the different fusion strategies selected in the first step, corresponding boundary condition constraints are defined on both sides of the optimized seam line. The MVC calculation framework constructed in the second step is used for specific numerical calculations. The seam line boundary offset is set to the difference between corresponding pixels in the two keyframe images to ensure a smooth transition. The image's own boundary offset is set to 0 to maintain the original image features. Based on these boundary conditions, the mean coordinate interpolation method in the MVC algorithm is used to calculate the offset distribution within the entire image block, generating a precise offset film, also known as the offset. In other words, the offset of each keyframe image is determined.

[0098] D3: Process keyframe image pairs based on offsets to obtain the corresponding wide field-of-view video frames.

[0099] Optionally, the offset film can be seamlessly fused using MVC boundary interpolation fusion technology. The calculated offset film is superimposed onto the original images of the keyframe image pair for fusion processing. This MVC-based boundary interpolation method achieves seamless fusion, generating high-quality video frames with a wide field of view and no obvious seams, resulting in seamlessly fused wide-field-of-view video frames. This provides preliminary fusion results for subsequent quality assessment. In a specific example, 100 wide-field-of-view video frames are obtained.

[0100] This application employs a multimodal motion detection fusion strategy, combining GMM background modeling, optical flow, and deep learning object detection. A dynamic programming algorithm for stitching lines based on motion trajectory prediction pre-predicts the paths of moving objects. A comprehensive loss function is constructed, incorporating spatial coherence, saliency, moving object penalty, and visual attention constraints. Adaptive fusion strategy selection is implemented for static high-texture regions, dynamic regions, and low-texture regions. An MVC image fusion algorithm based on convolutional pyramids provides efficient numerical computation support. This effectively solves the stitching artifact problem caused by moving objects in dynamic scenes, improving fusion quality.

[0101] Regarding S106, the wide field of view videos are stitched together based on their timestamps.

[0102] Before stitching, each wide-field-of-view video is optimized, and then stitched together. The optimization process is as follows: A comprehensive quality analysis of seamlessly fused wide-field-of-view video frames is performed using multi-dimensional quality assessment technology. The seam visibility assessment algorithm is used to detect the degree of artificial traces at the splicing boundaries, the motion blur detection algorithm is used to assess the sharpness level of dynamic areas, and the color consistency evaluation algorithm is applied to ensure that the color transition between different image areas is natural and smooth. A complete set of fusion quality assessment index system is established, and an assessment report containing various quality indicators is generated to obtain the quality assessment report.

[0103] Next, targeted post-processing optimization techniques are used to repair defects in the quality assessment report. Based on the quality assessment report generated in the first step, targeted post-processing optimization is performed on the quality defects detected in the seamlessly fused wide field-of-view video frames. According to the degree of defects of each indicator in the assessment report, the fusion parameters are dynamically adjusted through an adaptive parameter adjustment mechanism. Local optimization algorithms are used to eliminate detected artifact areas and discontinuities. Image smoothing techniques are used to process areas with unnatural color transitions, resulting in intermediate video frames that have undergone preliminary optimization.

[0104] Then, a consistency optimization operation is performed on the initially optimized intermediate video frames through a degree-cooperative optimization mechanism. A deep collaborative mechanism with the image registration module is adopted. Based on the quality status of the intermediate video frames generated in the second step, the weight allocation and boundary processing strategies in the fusion process are further optimized by sharing the improved SURF feature point set and descriptor F2 output by the improved SURF feature extraction algorithm, the coarse matching point pair set output by the bidirectional matching FLANN coarse matching algorithm, and the precise matching point pair set and homography matrix output by the GMS fine matching algorithm. The feature matching consistency information in the registration stage is used to reweight the poor quality areas. The processing parameters of the dynamic area are optimized based on the motion statistics results of the GMS fine matching algorithm, so as to obtain a high-quality fusion intermediate result that is consistent with the registration process.

[0105] Finally, a multi-round iterative optimization technique is used to integrate and output the high-quality fusion intermediate results. The high-quality fusion intermediate results generated in step three are then integrated and output, ensuring that all quality indicators meet preset thresholds through multiple rounds of optimization. This results in an optimized final video frame with excellent visual consistency and seamless stitching. This video frame achieves optimal levels in seam visibility, motion blur control, and color consistency, providing high-quality input data for subsequent real-time video stitching applications. The purpose of this step is to enrich the information in each wide-field-of-view video frame.

[0106] This application's embodiments employ a multi-dimensional quality assessment system, including indicators such as seam visibility, motion blur, and color consistency; an adaptive parameter adjustment and local optimization mechanism based on the quality assessment report; deep collaborative optimization with the image registration module to fully utilize feature information from the registration stage; and multi-round iterative optimization to ensure that each quality indicator reaches a preset threshold. It provides end-to-end quality control from keyframe extraction to final output, ensuring high-quality output results.

[0107] Finally, the optimized wide-field-of-view video (the final optimized video frames) is stitched together, and the stitching method is as follows: The system uses a timestamp synchronization mechanism to perform time alignment on the optimized final video frame sequence from different shooting devices. Each video frame carries precise timestamp information. The system maintains a time window buffer to collect frame data from each video stream within the same time window. The system aligns and caches multiple video frames in chronological order to ensure that the multiple video frames involved in the stitching are consistent in the time dimension. For frames with timestamp discrepancies, the system uses interpolation algorithms or nearest neighbor frame replacement strategies for time alignment to obtain a time-synchronized set of multiple video frames.

[0108] Next, the system performs a geometric transformation operation on the time-synchronized multi-channel video frame set object A9 using panoramic projection processing technology. After completing the time synchronization, the system extracts the time-aligned multi-channel optimized final video frame set from the time window buffer. Based on the preset shooting device spatial topology and camera parameters, the system uses the intrinsic and extrinsic parameter matrices of each shooting device to uniformly project the multi-channel video frames into a common panoramic coordinate system, thus obtaining a uniformly projected panoramic video frame.

[0109] Then, the unified projection panoramic video frame object B9 is subjected to stitching fusion operation through GPU parallel acceleration processing technology. The GPU parallel computing accelerates the efficient image geometric transformation, reprojection mapping and seamless blending processing. In this process, the system makes full use of the temporal alignment characteristics of each frame to ensure that moving objects at the stitching boundary maintain spatiotemporal consistency and avoid ghosting or jumping phenomena, thus obtaining the panoramic video frame after stitching fusion processing.

[0110] Finally, multi-threaded stream encapsulation technology is used to perform real-time streaming format encoding on the panoramic video frames after stitching fusion. The time-aligned and stitched panoramic video frames are serialized and encapsulated according to a preset frame rate. Multi-threaded technology enables continuous reading and streaming format encoding of video frames. The system adopts a producer-consumer model, where one thread is responsible for obtaining panoramic frames from the stitching processing module and encoding and compressing them, while another thread is responsible for encapsulating the encoded data into real-time video stream formats (such as RTMP, HLS, etc.). Finally, a panoramic video stream that can be played in real time or further processed is generated, ensuring the continuity and real-time nature of the output, resulting in a real-time panoramic video stream.

[0111] This application's embodiments utilize a timestamp-based time synchronization and cache management mechanism to ensure temporal consistency across multiple video frames; leverage GPU parallel computing to accelerate image geometric transformation and reprojection mapping processing; employ a producer-consumer model multi-threaded real-time stream encapsulation technology; and employ seam blending technology to ensure spatiotemporal consistency of moving objects at stitching boundaries. This achieves high-quality real-time panoramic integration of multiple video streams, meeting the real-time requirements of practical applications.

[0112] like Figure 2 As shown, based on the same inventive concept as the above-mentioned multi-channel video splicing method, this application embodiment also provides a multi-channel video splicing device, which includes a processing unit 21 and a splicing unit 22.

[0113] Processing unit 21 is used to: parse at least one video to be processed to obtain a set of video frames; Processing unit 21 is further configured to: determine M keyframe image pairs in the video frame set; wherein each keyframe image pair includes a reference image and a matching image, the timestamp of each reference image is earlier than the timestamp of the corresponding matching image; the time difference between the timestamp of each reference image and the timestamp of the corresponding matching image is less than a set time threshold; and the distance between the source shooting device of each reference image and the source shooting device of the corresponding matching image is less than a set distance threshold. The processing unit 21 is also used to: extract feature points from M keyframe image pairs to obtain a set of feature points and a set of descriptors consisting of descriptors that correspond one-to-one with the feature points in the set of feature points; Processing unit 21 is further configured to: generate a stitching line matching part or all of the keyframe image pairs based on the feature point set and the descriptor subset; Processing unit 21 is also configured to: apply each stitch to determine the wide field of view video of the corresponding keyframe image pair; The stitching unit 22 is used to stitch together the various wide-field-of-view videos based on their timestamps.

[0114] In one optional implementation, the processing unit 21 is specifically used for: Sort the video frames in the video frame set according to time and calculate the similarity between adjacent video frames; Based on the obtained similarity, each video frame in the video frame set is filtered to obtain the first video frame set; The time uniformity adjustment is performed on each video frame in the first video frame set to obtain the second video frame set; wherein the difference in the number of video frames in adjacent time periods in the second video frame set is less than a set number threshold. Based on the timestamps of each video frame in the second video frame set and the location of the source shooting device, M keyframe image pairs are determined.

[0115] In one optional implementation, the processing unit 21 is specifically used for: The directional gradient histogram information is fused into the M keyframe image pairs to obtain the processed M keyframe image pairs; Determine the extraction region, extract P feature points from the processed M keyframe image pairs within the extraction region, and calculate P descriptors corresponding to the P feature points; P feature points are processed into a grid for uniform distribution to obtain Q feature points; wherein, the Q feature points are located within the region formed by the defined grid. Recalculate Q new descriptors corresponding to Q feature points; where the Q feature points constitute a feature point set, and the Q new descriptors constitute a descriptor set.

[0116] In one optional implementation, the processing unit 21 is specifically used for: Based on the descriptive subset, feature points in the feature point set are paired to obtain a set of matching point pairs; Based on the pre-defined correspondence between feature points and keyframe images, determine the reference keyframe image pairs corresponding to the matching point pair set; For each reference keyframe image pair, the first information of the moving objects included in the reference keyframe image pair is determined by calculating the historical Gaussian distribution features of each pixel; the second information of the moving objects included in the reference keyframe image pair is determined by calculating the motion vector of each pixel; and the third information is determined by classifying the moving objects included in the reference keyframe image pair. Here, the first information represents the identifier of the moving object, the second information represents the motion of the moving object, and the third information represents the motion of different categories of moving objects. By fusing the first, second, and third information, a binarized foreground marker image and a reliability assessment map are obtained. By applying a binarized foreground marker image and a reliability assessment map, the suture line for matching the reference keyframe image is determined.

[0117] In one optional implementation, the processing unit 21 is specifically used for: The binary foreground marker image and the reliability assessment map are queried to determine whether the stitching line used in the previous frame reference keyframe image pair intersects with the foreground region of the current frame. If so, the motion trend of the object is predicted based on the motion information and velocity vector of the moving object in the previous frame reference keyframe image pair, and the avoidance range is determined based on the motion trend. Kinematic constraints are generated based on the avoidance range. Generate loss functions based on kinematic constraints; The suture line obtained when the loss function has the minimum loss value is the matching suture line.

[0118] In one optional implementation, the processing unit 21 is specifically used for: Based on Q feature points and Q descriptors, a first set of matching feature points for each reference image feature point is determined, and a second set of matching feature points for each feature point to be matched is determined; wherein, the similarity between each feature point in the first set of feature points and the feature points of the reference image is less than a set similarity threshold, and the similarity between each feature point in the second set of feature points and the feature points of the image to be matched is less than a set similarity threshold. In the first set of feature points, determine the target feature points of the target image to be matched for feature point matching of the reference image; In the second set of feature points, determine the target feature points of the target reference image to match the feature points of the image to be matched; Based on the target feature points matched in each reference image and the target feature points matched in each image to be matched, a set of matching point pairs is determined.

[0119] In one optional implementation, the processing unit 21 is specifically used for: Based on the object motion information provided by the stitching path, and the texture features of each region in each pair of keyframe images, the fusion algorithm used for each region is determined. The offset of each keyframe image pair is calculated using a fusion algorithm; The keyframe image pairs are processed based on the offset to obtain the corresponding wide field-of-view video frames.

[0120] The multi-channel video splicing device proposed in this application embodiment adopts the same inventive concept as the above-mentioned multi-channel video splicing method and can achieve the same beneficial effects, so it will not be described again here.

[0121] Based on the same inventive concept as the aforementioned multi-channel video stitching method, this application also provides an electronic device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), server, etc. Figure 3 As shown, the electronic device may include a processor 301 and a memory 302.

[0122] Processor 301 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0123] Memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 302 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0124] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned computer storage medium can be any available medium or data storage device that a computer can access, including but not limited to: mobile storage devices, random access memory (RAM), magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)) and other media capable of storing program code.

[0125] Alternatively, if the integrated units described above in this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes: mobile storage devices, random access memory (RAM), magnetic memory (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.) and other media capable of storing program code.

[0126] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the multi-channel video splicing methods described above. Since the principle by which the above computer program product solves the problem is similar to that of the multi-channel video splicing method, the implementation of the above computer program product can refer to the implementation of the method, and repeated details will not be described again.

[0127] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods of the embodiments of this application and should not be construed as a limitation on the embodiments of this application. Any changes or substitutions that can be easily conceived by those skilled in the art should be covered within the protection scope of the embodiments of this application.

Claims

1. A method for splicing multiple video streams, characterized in that, include: At least two videos to be processed are parsed to obtain a set of video frames; In the video frame set, M keyframe image pairs are determined; where M is an integer greater than 1, each keyframe image pair includes a reference image and a matching image, the timestamp of each reference image is earlier than the timestamp of the corresponding matching image; the time difference between the timestamp of each reference image and the timestamp of the corresponding matching image is less than a set time threshold; the distance between the source shooting device of each reference image and the source shooting device of the corresponding matching image is less than a set distance threshold. The directional gradient histogram information is fused into the M keyframe image pairs to obtain the processed M keyframe image pairs; Determine the extraction region, and extract P feature points from the processed M keyframe image pairs within the extraction region, and calculate P descriptors corresponding to the P feature points; where P is an integer greater than M; The P feature points are processed into a grid for uniform distribution to obtain Q feature points; wherein the Q feature points are located within a region formed by a defined grid; and Q is an integer less than P. Recalculate the Q new descriptors corresponding to the Q feature points; wherein the Q feature points constitute a feature point set, and the Q new descriptors constitute a descriptor set; Based on the descriptive subset, feature points in the feature point set are paired to obtain a set of matching point pairs; Based on the pre-defined correspondence between feature points and keyframe images, a reference keyframe image pair corresponding to the matching point pair set is determined from the M keyframe image pairs. For each pair of reference keyframe images, the following methods are used: first information about the moving objects included in the reference keyframe image pair is determined by calculating the historical Gaussian distribution features of each pixel; second information about the moving objects included in the reference keyframe image pair is determined by calculating the motion vector of each pixel; and third information is determined by classifying the moving objects included in the reference keyframe image pair. The first information represents the identifier of the moving object, the second information represents the motion of the moving object, and the third information represents the motion of different categories of moving objects. By fusing the first information, the second information, and the third information, a binarized foreground marker image and a reliability assessment map are obtained. The binarized foreground marker image and the reliability assessment map are queried to determine whether the stitching line used in the previous frame reference keyframe image pair intersects with the foreground region of the current frame. If so, the motion trend of the object is predicted based on the motion information and velocity vector of the moving object in the previous frame reference keyframe image pair, and the avoidance range is determined based on the motion trend. Kinematic constraints are generated based on the avoidance range. A loss function is generated based on the aforementioned kinematic constraints; The suture line obtained when the loss value of the loss function is minimized is the matching suture line; Apply each stitch to determine the corresponding keyframe image pair in the wide field of view video; Based on the timestamps of each wide field of view video, the wide field of view videos are stitched together.

2. The method according to claim 1, characterized in that, The step of determining M keyframe image pairs in the video frame set includes: The video frames in the video frame set are sorted by time, and the similarity between adjacent video frames is calculated. Based on the obtained similarity, each video frame in the video frame set is filtered to obtain the first video frame set; The time uniformity adjustment is performed on each video frame in the first video frame set to obtain a second video frame set; wherein the difference in the number of video frames in adjacent time periods in the second video frame set is less than a set number threshold. Based on the timestamp of each video frame in the second video frame set and the location of the source shooting device, M keyframe image pairs are determined.

3. The method according to claim 1, characterized in that, The step of pairing feature points in the feature point set based on the descriptive subset to obtain a set of matching point pairs includes: Based on the Q feature points and the Q new descriptors, a first set of matching feature points for each reference image is determined, and a second set of matching feature points for each feature point to be matched is determined; wherein, the similarity between each feature point in the first set and the feature points of the reference image is less than a set similarity threshold, and the similarity between each feature point in the second set and the feature points of the image to be matched is less than the set similarity threshold. In the first set of feature points, the target feature points of the target image to be matched are determined to be matched with the feature points of the reference image; In the second set of feature points, the target feature points of the target reference image to which the feature points of the image to be matched are determined; Based on the target feature points matched in each reference image and the target feature points matched in each image to be matched, a set of matching point pairs is determined.

4. The method according to any one of claims 1 to 3, characterized in that, The wide-field-of-view video that uses each stitch to determine the corresponding keyframe image pair includes: Based on the object motion information provided by the suture line, and the texture features of each region in each keyframe image pair, the fusion algorithm used for each region is determined. The fusion algorithm is applied to calculate the offset of each keyframe image pair; The keyframe image pairs are processed based on the offset to obtain the corresponding wide field-of-view video frames.

5. A multi-channel video splicing device, characterized in that, include: The processing unit is used to: parse at least two videos to be processed to obtain a set of video frames; The processing unit is further configured to: determine M keyframe image pairs in the video frame set; wherein each keyframe image pair includes a reference image and a matching image, the timestamp of each reference image is earlier than the timestamp of the corresponding matching image; the time difference between the timestamp of each reference image and the timestamp of the corresponding matching image is less than a set time threshold; and the distance between the source shooting device of each reference image and the source shooting device of the corresponding matching image is less than a set distance threshold. The processing unit is further configured to: fuse directional gradient histogram information into the M keyframe image pairs to obtain processed M keyframe image pairs; determine an extraction region, and extract P feature points from the processed M keyframe image pairs within the extraction region, and calculate P descriptors corresponding to the P feature points; wherein P is an integer greater than M; perform gridded uniform processing on the P feature points to obtain Q feature points; wherein the Q feature points are located within a region formed by a set grid; wherein Q is an integer less than P; recalculate Q new descriptors corresponding to the Q feature points; wherein the Q feature points constitute a feature point set, and the Q new descriptors constitute a descriptor set; The processing unit is further configured to: pair feature points in the feature point set based on the descriptive subset to obtain a matching point pair set; determine a reference keyframe image pair corresponding to the matching point pair set among the M keyframe image pairs according to a pre-defined correspondence between feature points and keyframe images; for each reference keyframe image pair, determine first information of the moving object included in the reference keyframe image pair by calculating the historical Gaussian distribution features of each pixel, determine second information of the moving object included in the reference keyframe image pair by calculating the motion vector of each pixel, and determine third information by classifying the moving objects included in the reference keyframe image pair; wherein, the first information represents the identifier of the moving object, and the second information represents... The motion of the moving object is described, and the third information represents the motion of different categories of moving objects. The first information, the second information, and the third information are fused to obtain a binarized foreground marker image and a reliability assessment map. The binarized foreground marker image and the reliability assessment map are queried to determine whether the stitching line used in the previous frame reference keyframe image pair intersects with the foreground region of the current frame. If so, the motion trend of the object is predicted based on the motion information and velocity vector of the moving object in the previous frame reference keyframe image pair, and the avoidance range is determined based on the motion trend. Kinematic constraints are generated based on the avoidance range. A loss function is generated based on the kinematic constraints. The stitching line obtained when the loss value of the loss function is minimized is the matching stitching line. The processing unit is also used to: apply each stitch to determine the wide field-of-view video of the corresponding keyframe image pair; The stitching unit is used to stitch together the various wide-field-of-view videos based on their timestamps.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 4.

Images