Video stitching method, device, equipment, medium and product

Abstract

The invention discloses a video stitching method, device and equipment, a medium and a product, and relates to the technical field of computer vision. Acquiring multiple paths of videos; converting the multi-channel video to obtain at least one image frame set; the image frame set is image frames corresponding to multiple paths of videos at the same moment; determining whether loading of a preset splicing template is completed, and if loading fails, generating a second splicing template based on the image frame set; and splicing the image frames in the image frame set according to the second splicing template to obtain a target spliced image frame. By adopting the technical scheme, the problem of how to update the splicing template of the multi-channel video is solved, and real-time splicing is realized; the corresponding splicing template is generated according to each image frame set, so that scene change is dynamically adapted, the splicing precision is improved, low-delay and high-precision panoramic video splicing under the dynamic multi-video condition is realized, and the monitoring coverage range and the imaging quality are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video stitching method, apparatus, device, medium, and product. Background Technology

[0002] In cutting-edge fields such as smart cities, autonomous driving, industrial automation, and intelligent security, the demand for real-time, high-precision monitoring and situational awareness of large-scale and complex scenarios is becoming increasingly urgent. Multi-camera collaborative work, through video stitching technology, seamlessly integrates their images into a unified, ultra-wide-angle panoramic view, which has become a key technical path to expand the monitoring range and improve the efficiency of information acquisition.

[0003] Existing multi-channel video stitching technologies typically calculate the geometric transformation relationships between cameras through pre-calibration, followed by geometric correction and fusion. While this can achieve relatively stable stitching results while maintaining real-time performance, in practical deployments, this method not only relies on fixed camera deployments but also suffers from a significant drop in stitching quality and lacks adaptability if the cameras move slightly or the viewing angle changes. Another multi-channel video stitching technology treats each set of synchronized video frames as an independent static image stitching problem, performing feature point detection, matching, and transformation matrix estimation frame by frame. However, in dynamic scenes, factors such as targets crossing the stitching area, camera shake, and changes in lighting and parallax often produce obvious visual defects such as stitching seams, misalignment, deformation, and ghosting, severely affecting the reliability of monitoring and perception. Moreover, the huge computational redundancy and performance overhead, high-frequency feature calculations, and complex iterative optimizations make the latency of processing a single frame far exceed real-time requirements. Summary of the Invention

[0004] This invention provides a video stitching method, apparatus, device, medium, and product, which solves the problem of how to update the stitching template of multiple video streams to achieve real-time stitching; by generating corresponding stitching templates based on each set of image frames, it dynamically adapts to scene changes and improves stitching accuracy, thereby achieving low-latency, high-precision panoramic video stitching in dynamic multi-video situations, and improving monitoring coverage and imaging quality.

[0005] According to one aspect of the present invention, a video stitching method is provided, comprising: Acquire multiple video streams; The multiple video streams are converted to obtain at least one set of image frames; the set of image frames consists of image frames corresponding to the multiple video streams at the same time. Determine whether the preset stitching template has been loaded. If loading fails, generate a second stitching template based on the image frame set. The image frames in the image frame set are stitched together according to the second stitching template to obtain the target stitched image frame.

[0006] According to another aspect of the present invention, a video stitching device is provided, comprising: The acquisition module is used to acquire multiple video streams. A conversion module is used to convert the multiple video streams to obtain at least one set of image frames; the set of image frames consists of image frames corresponding to the multiple video streams at the same time. The splicing template generation module is used to determine whether the preset splicing template has been loaded. If loading fails, a second splicing template is generated based on the image frame set. The stitching module is used to stitch the image frames in the image frame set according to the second stitching template to obtain the target stitched image frame.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the video stitching method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video stitching method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the video stitching method according to any embodiment of the present invention.

[0010] The technical solution of this invention solves the problem of how to update the stitching template of multiple videos by generating corresponding stitching templates according to each set of image frames, thereby achieving real-time stitching; dynamically adapting to scene changes and improving stitching accuracy, achieving low-latency, high-precision panoramic video stitching in dynamic multi-video situations, and improving monitoring coverage and imaging quality.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a video stitching method provided according to an embodiment of the present invention; Figure 2 This is a flowchart of a video stitching method provided according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a video splicing device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the video splicing method of this invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product or device.

[0016] Furthermore, it should be noted that the information collected in the technical solution of this invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures have been taken, and public order and good morals are not violated. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] Figure 1This invention provides a flowchart of a video stitching method, applicable to various video stitching scenarios, particularly real-time stitching of videos from multiple cameras working collaboratively. The method can be executed by a video stitching device, which can be implemented in hardware and / or software and can be configured in a server. Figure 1 As shown, the method includes: S110, Acquire multiple video streams.

[0018] Among them, multiple videos are multiple videos that need to be stitched together, that is, multiple videos collected under the collaborative work of multiple cameras.

[0019] Specifically, determine the interface for acquiring multiple video streams, and acquire the videos to be stitched from multiple cameras based on the interface.

[0020] S120. Convert the multiple video streams to obtain at least one set of image frames; the set of image frames consists of the image frames corresponding to the multiple video streams at the same time.

[0021] The conversion process involves transforming video into a corresponding sequence of image frames, which can be sorted chronologically. The image frames are obtained from the conversion of each video stream. The image frame set consists of image frames corresponding to multiple video streams at the same time.

[0022] Specifically, multiple video streams are converted to obtain corresponding frame sequences; then, according to the timestamps of the frame sequences, the image frames corresponding to the multiple video streams at the same time are associated to obtain an image frame set; finally, the image frame sets are sorted according to the timestamps to obtain at least one image frame set corresponding to the multiple video streams.

[0023] Optionally, the multiple video streams are converted to obtain at least one set of image frames, including: The multiple video streams are converted separately to obtain the corresponding video frame sequences; The video frame sequence is preprocessed to obtain a grayscale frame sequence; The grayscale frame sequence is denoised and enhanced to obtain the image frame sequence; By associating the image frames in the image frame sequence corresponding to multiple video streams according to their timestamps, at least one set of image frames can be obtained.

[0024] Among them, the video frame sequence is obtained by segmenting the video to obtain the original image frames corresponding to each time moment and sorting them according to the timestamp; the grayscale frame sequence is obtained by converting each original image frame in the original frame sequence to grayscale; the denoising algorithm can be Gaussian filtering; the enhancement processing can be histogram equalization; and the image frame sequence is obtained by processing the grayscale frame sequence.

[0025] Specifically, multiple video streams are converted separately to obtain corresponding video frames. These video frames are then sorted according to their timestamps to obtain a sequence of video frames corresponding to the multiple video streams. The video frame sequence is then subjected to distortion correction and grayscale transformation to obtain a grayscale frame sequence. The grayscale frame sequence is then denoised and enhanced using Gaussian filtering and histogram equalization to obtain a corresponding image frame sequence. Finally, the image frames in the image frame sequence corresponding to the multiple video streams are associated according to their timestamps to obtain at least one set of image frames.

[0026] Understandably, by associating the corresponding image frame sequences of multiple video streams according to their timestamps, we can obtain the image frames that need to be stitched together at the same time. By associating and splitting the image frame sequences, we can obtain a set of image frames, thereby improving the efficiency of subsequent image stitching and further ensuring the accuracy of subsequent image stitching at the same time.

[0027] S130. Determine whether the preset splicing template has been loaded. If loading fails, generate a second splicing template based on the image frame set.

[0028] The preset splicing template is pre-built. During the iterative update of the splicing template, the new splicing template can be used as the preset splicing template; the second splicing template is the newly generated splicing template.

[0029] Specifically, when stitching together a set of image frames, a pre-stored stitching template is loaded. If loading fails, a second stitching template is generated based on the set of image frames.

[0030] S140. The image frames in the image frame set are stitched together according to the second stitching template to obtain the target stitched image frame.

[0031] Specifically, the image frames in the image frame set are stitched together according to the updated second stitching template to obtain the target stitched image frame, which is then output.

[0032] In an optional embodiment of the present invention, the stitching template includes a globally optimal transformation matrix and a global fusion mask. The global fusion mask is obtained by defining a fusion band of a preset width on both sides of the stitching seam and calculating the vertical distance from the pixels of the image frame to the stitching seam within the fusion band. The image frame set is stitched according to the stitching template, specifically by aligning each image frame in the image frame set, and performing weighted fusion on the images on both sides of the preset fusion band of the stitching seam based on the fusion weight values ​​in the brightness channel and color channel, respectively. The color consistency of the fusion result is adjusted, and the fused image is used as the stitched image.

[0033] Optional, also includes: If loading is successful, the image frames in the image frame set are stitched together according to the preset stitching template to obtain the initial stitched image frame; An anomaly detection algorithm is used to detect anomalies in the initial stitched image frames to obtain abnormal values. The abnormal values ​​are compared with preset thresholds to obtain the evaluation results; If the evaluation result is normal, the initial stitched image frame will be used as the target stitched image frame.

[0034] The initial stitched image frame is the image frame obtained by stitching together a set of image frames; the stitching detection algorithm is trained using deep learning methods and can employ the mean squared error loss function; the anomaly value is the anomaly score obtained by anomaly detection of the initial stitched image frame, which can be determined based on the size and number of anomaly regions and the confidence level within the region; the preset threshold is a pre-set anomaly threshold; and the target stitched image frame is the image frame that can be output.

[0035] Specifically, an initial stitched image frame is obtained by stitching together a set of image frames from the same moment according to a preset stitching template. A stitching detection algorithm is then used to detect this initial stitched image frame, obtaining its anomaly score. This anomaly score is compared to a preset anomaly threshold. If the anomaly score is less than the threshold, the stitching effect is considered good, and the initial stitched image frame is cropped and rectangularized to serve as the target stitched image frame. The preset stitching template is then used to stitch together the next set of image frames. It should be noted that technicians can modify the anomaly threshold according to specific effect requirements and the needs of different scenarios.

[0036] In an optional embodiment of the present invention, if the abnormal score is greater than or equal to a preset abnormal threshold, it indicates that the stitching effect is not good and the evaluation result is abnormal. Then, the stitching template update process is triggered, that is, a second stitching template is generated based on the image frame set and used for image frame stitching in the image frame set.

[0037] Understandably, the introduction of a stitching template update mechanism allows for real-time stitching by directly calling the generated stitching template in the absence of anomalies, ensuring high efficiency and low latency. When an anomaly is detected in the stitching evaluation result, the stitching template update process is triggered, thereby dynamically adapting to scene changes and improving stitching accuracy. This enables low-latency, high-precision panoramic video stitching in dynamic multi-video scenarios, improving monitoring coverage and imaging quality.

[0038] In an optional embodiment of the present invention, the stitching detection algorithm is trained and predicted using a deep learning method; the training samples are stitched images of various stitched images in different scenes that have problems such as stitching seams, ghosting, and artifacts, as well as corresponding label values, the label values ​​being mainly the overall quality score; the training samples are trained using a deep learning algorithm with a mean squared error loss function to obtain the stitching detection algorithm.

[0039] Understandably, algorithms designed and trained based on deep learning for stitching quality detection are used to detect stitching defects in the stitched images, thereby dynamically adapting to scene changes and improving stitching accuracy. This avoids a large number of multiple calculations, improves the real-time performance of the entire system, and balances the real-time performance of the algorithm with the final video stitching effect.

[0040] This invention provides an embodiment that acquires multiple video streams; converts the multiple video streams to obtain at least one set of image frames; the set of image frames consists of image frames corresponding to multiple video streams at the same time; determines whether a preset stitching template has been loaded; if loading fails, a second stitching template is generated based on the set of image frames; and stitches the image frames in the set of image frames according to the second stitching template to obtain the target stitched image frame. This technical solution solves the problem of how to update the stitching template for multiple video streams to achieve real-time stitching; by generating corresponding stitching templates based on each set of image frames, it dynamically adapts to scene changes and improves stitching accuracy, achieving low-latency, high-precision panoramic video stitching in dynamic multi-video situations, thereby improving monitoring coverage and imaging quality.

[0041] Figure 2 This is a flowchart illustrating a video stitching method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment supplements the method for generating the second stitching template. It should be noted that for parts not detailed in this embodiment, please refer to the relevant descriptions in other embodiments, such as... Figure 2 As shown, the method includes: S210, Acquire multiple video streams.

[0042] S220. Convert the multiple video streams to obtain at least one set of image frames; the set of image frames consists of the image frames corresponding to the multiple video streams at the same time.

[0043] S230. Determine whether the preset stitching template has been loaded. If loading fails, process each image frame in the image frame set using the registration algorithm to obtain the target homography transformation matrix.

[0044] The registration algorithm can include ORB (Oriented FAST and Rotated BRIEF, a feature extraction and description combining orientation and rotation) and RANSAC (RANdom Sampling Consensus). The target homography transformation matrix indicates the initial geometric transformation relationship between image frames, which is used to register one image to the coordinate system of another image to complete the initial image alignment.

[0045] Specifically, when stitching together a set of image frames, a pre-stored stitching template is loaded. If loading fails, a registration algorithm is used to register the image frames in the set. Based on the registered image frames, the target homography transformation matrix in the set of image frames is obtained so as to complete the initial image alignment based on the initial geometric transformation relationship between the image frames.

[0046] Optionally, a registration algorithm is used to process each image frame in the image frame set to obtain the target homography transformation matrix, including: Feature extraction is performed on each image frame in the image frame set to obtain the feature points corresponding to each image frame; The feature points are matched using a matching algorithm to obtain at least one matching point pair; Iteratively extract at least one pair of matching points to obtain at least one set of matching points; At least one candidate homography transformation matrix is ​​obtained by calculating each of at least one set of matching points; The target homography transformation matrix is ​​determined from the candidate homography transformation matrices based on the number of interior points in the candidate homography transformation matrices.

[0047] In this process, feature extraction can be performed using the ORB algorithm programmed in CUDA; feature points are generated by binary encoding of pixels at specific locations to produce a feature descriptor of a specific length; the matching algorithm can be a brute-force matching algorithm; the matching point pairs are obtained by finding the corresponding feature points in the image frame based on the brute-force matching algorithm and the Hamming distance between each feature point; the matching point set can be randomly selected matching points; the candidate homography transformation matrix is ​​the geometric relationship obtained by transforming each matching point in the matching point set; and the target homography transformation matrix is ​​the geometric transformation matrix between image frames.

[0048] Specifically, the ORB algorithm is used to extract features from each image frame in the image frame set to obtain the feature points corresponding to each image frame; and the Hamming distance between the feature points in each image frame is further determined; for each feature point in an image frame, a brute-force matching algorithm is used to match the feature point with the smallest Hamming distance in another image frame as a matching point pair; in the obtained set of matching point pairs, the RANSAC algorithm is introduced to randomly select 4 pairs of matching points from the matching points as a matching point set, and the corresponding homography matrix is ​​calculated, and the number of interior points in the candidate homography matrix is ​​counted; through multiple iterations, the candidate homography matrix with the largest number of interior points is selected as the target homography matrix.

[0049] Understandably, CUDA programming is used to accelerate GPU (Graphics Processing Unit) processing, improving the real-time performance and processing efficiency of the algorithm. To avoid noise and mismatches affecting the results by directly obtaining the homography matrix of the matching points, the RANSAC algorithm is introduced to perform robust estimation of the matching point set, achieving global matching of image frames.

[0050] S240. The target homography matrix is ​​iteratively optimized using a global optimization algorithm to obtain the globally optimal transformation matrix.

[0051] The global optimization algorithm can be a pose graph optimization algorithm; the global transformation matrix is ​​the transformation matrix between image frames.

[0052] Specifically, the target homography matrix is ​​iteratively optimized using a pose graph optimization algorithm to output the optimized camera pose and calculate the globally optimal transformation matrix.

[0053] Optionally, the target homography matrix can be iteratively optimized using a global optimization algorithm to obtain the globally optimal transformation matrix, including: Select a reference image frame from the image frame set and use the pose of the reference image frame as the unit pose; Based on the target homography transformation matrix, the initial poses of the remaining image frames in the image frame set are calculated sequentially. Construct a global cost matrix based on the unit pose, initial pose, and target homography transformation matrix; An iterative optimization algorithm is used to iteratively optimize the global cost matrix to obtain the global transformation matrix.

[0054] Among them, the reference image frame is an image frame with relatively rich features; the unit pose is the pose corresponding to the reference image frame; the initial pose is calculated by transforming the other image frames through the target homography transformation matrix; the global cost matrix is ​​the pose node, and the target homography transformation matrix is ​​the edge constraint connecting the corresponding pose node; the cost matrix of the obtained edge constraint error; the iterative optimization algorithm is a nonlinear least squares iterative optimization algorithm.

[0055] Specifically, a feature-rich image frame is selected from the image frame set as the reference image frame, and its pose is taken as the unit pose. Based on the target homography transformation matrix, the initial poses of all other image frames are calculated sequentially through transformation propagation. Each image frame is defined as a pose node in the graph, and the target homography transformation matrix is ​​defined as the edge constraint connecting the corresponding pose node. A global cost function based on the errors of all edge constraints is constructed, where the error represents the difference between the currently estimated pose node and the relative relationship described by the initial transformation matrix. A nonlinear least squares iterative optimization algorithm is used to optimize the parameters of all pose nodes until the global cost function converges to the minimum value, and the optimized camera pose is output as the global transformation matrix.

[0056] Understandably, by further optimizing the target homography matrix through the pose optimization algorithm, a global transformation matrix is ​​obtained, which enhances the robustness to small changes in camera pose and interference from complex scenes. This makes it widely applicable to application scenarios that require high precision, real-time performance, and stability, such as smart city monitoring, intelligent security, and industrial vision inspection.

[0057] S250. The image cut method is used to process each adjacent image frame in the image frame set to obtain the splicing seam.

[0058] Among them, the graph cut method is to segment and traverse each pair of adjacent image frames in the image frame set; the stitching seam is used to stitch the image frames together.

[0059] Specifically, the graph cut method is used to segment each pair of adjacent image frames in the image frame set, and the segmented regions are traversed to obtain the optimal stitching seam for subsequent image stitching. For example, if the image frame set is obtained by processing videos captured by three cameras, then from left to right, they are camera A, camera B, and camera C. When stitching image frames at the same time, the image frame corresponding to camera A is A1, the image frame corresponding to camera B is B1, and the image frame corresponding to camera C is C1. Then, image frames A1 and B1 need to be stitched together, and image frames B1 and C1 need to be stitched together. At this time, the graph cut method is used to obtain the stitching seam when stitching image frames A1 and B1, and the graph cut method is used to obtain the stitching seam when stitching image frames B1 and C1.

[0060] Optionally, a graph cut method is used to process each adjacent image frame in the image frame set to obtain the stitching seam, including: Determine the overlapping region of each adjacent image frame in the image frame set; Superpixel segmentation is performed on the overlapping region to obtain superpixel units; An energy cost function is defined based on the difference in color and texture between adjacent superpixel units, and a weight map is constructed based on the energy cost function. The minimum cut algorithm is used to segment the weighted graph, and the minimum energy cost of the segmented weighted graph is minimized. The boundary corresponding to the cut edge path with the minimum energy cost is defined as the splicing seam.

[0061] Among them, the overlapping region is the overlapping area of ​​adjacent image frames; the superpixel unit is an atomic region with visual significance formed by aggregating adjacent pixels with similar color, texture, brightness and other characteristics; color and texture correspond to pixels; the weighted graph is a graph with weights constructed by aggregating superpixel units into nodes and constructing the color and texture differences between superpixel units, with each edge defining a weight; the stitching seam is the line that stitches adjacent image frames together.

[0062] Specifically, the overlapping region of each adjacent image frame is determined, and the overlapping region is segmented into corresponding superpixel units. Pixels are aggregated into nodes, and a weighted graph is constructed based on the color and texture differences between superpixels. A weight is defined for each edge in the graph, and this weight is used as the cost of cutting this edge, defined as the energy cost function. A weighted graph is constructed based on the energy cost function, such as the ST graph. The minimum cut algorithm can be used to segment the ST graph to generate cut edges. The minimum energy cost value corresponding to the cut edge is calculated, and the boundary of the cut edge path associated with the minimum energy cost value is defined as the seam.

[0063] Understandably, by defining an energy cost function and combining it with the minimum cut algorithm to find the seams in the image frame set, the tearing and ghosting phenomena when moving targets cross the seams are significantly reduced, thus improving the overall viewing experience and analytical value of the video.

[0064] S260. Generate a global fusion mask based on the splicing seam, and use the global optimal transformation matrix and the global fusion mask as the second splicing template.

[0065] The global fusion mask is calculated based on the seams.

[0066] Specifically, a fusion band of preset width is defined on both sides of the stitching seam, and a global fusion mask is calculated within the fusion band based on the vertical distance from the pixel of the image frame to the stitching seam. The global optimal transformation matrix and the global fusion mask are used as the second stitching template.

[0067] S270. The image frames in the image frame set are stitched together according to the second stitching template to obtain the target stitched image frame.

[0068] This invention, through global optimization of the transformation matrix and calculation of the optimal stitching seam, can bypass dynamic objects and high-texture areas, reducing ghosting and stitching seam issues. Furthermore, by regenerating the stitching template through steps such as feature matching, global optimization, and calculation of the optimal stitching seam, it dynamically adapts to scene changes and improves stitching accuracy. This enables low-latency, high-precision panoramic video stitching in dynamic multi-video scenarios, improving monitoring coverage and imaging quality.

[0069] Figure 3 This is a schematic diagram of a video stitching device provided in an embodiment of the present invention. The present invention is applicable to video stitching, particularly to real-time video stitching in multi-camera collaborative operation. This video stitching device can be implemented in hardware and / or software and can be configured in a server. Figure 3 As shown, the video splicing device 300 includes an acquisition module 310, a conversion module 320, a splicing template generation module 330, and a splicing module 340. Acquisition module 310 is used to acquire multiple video streams; The conversion module 320 is used to convert multiple video streams to obtain at least one set of image frames; the set of image frames consists of image frames corresponding to multiple video streams at the same time. The splicing template generation module 330 is used to determine whether the preset splicing template has been loaded. If loading fails, a second splicing template is generated based on the image frame set. The stitching module 340 is used to stitch the image frames in the image frame set according to the second stitching template to obtain the target stitched image frame.

[0070] This invention provides an embodiment that acquires multiple video streams; converts the multiple video streams to obtain at least one set of image frames; the set of image frames consists of image frames corresponding to multiple video streams at the same time; determines whether a preset stitching template has been loaded; if loading fails, a second stitching template is generated based on the set of image frames; and stitches the image frames in the set of image frames according to the second stitching template to obtain the target stitched image frame. This technical solution solves the problem of how to update the stitching template for multiple video streams to achieve real-time stitching; by generating corresponding stitching templates based on each set of image frames, it dynamically adapts to scene changes and improves stitching accuracy, achieving low-latency, high-precision panoramic video stitching in dynamic multi-video situations, thereby improving monitoring coverage and imaging quality.

[0071] Optionally, the video splicing device 300 also includes an update module, which, if the loading is successful, splices the image frames in the image frame set according to the preset splicing template to obtain the initial spliced ​​image frame. An anomaly detection algorithm is used to detect anomalies in the initial stitched image frames to obtain abnormal values. The abnormal values ​​are compared with preset thresholds to obtain the evaluation results; If the evaluation result is normal, the initial stitched image frame will be used as the target stitched image frame.

[0072] Optionally, the splicing template generation module 330 includes a registration unit, a global optimization unit, a splicing seam determination unit, and a splicing template generation unit; The registration unit is used to process each image frame in the image frame set through a registration algorithm to obtain the target homography transformation matrix; The global optimization unit is used to iteratively optimize the target homography matrix using a global optimization algorithm to obtain the globally optimal transformation matrix. The seam determination unit is used to process each adjacent image frame in the image frame set using the graph cut method to obtain the seam. The stitching template generation unit is used to generate a global fusion mask based on the stitching seam, and uses the global optimal transformation matrix and the global fusion mask as the second stitching template.

[0073] Optionally, the registration unit is also used to extract features from each image frame in the image frame set to obtain the feature points corresponding to each image frame; The feature points are matched using a matching algorithm to obtain at least one matching point pair; Iteratively extract at least one pair of matching points to obtain at least one set of matching points; At least one candidate homography transformation matrix is ​​obtained by calculating each of at least one set of matching points; The target homography transformation matrix is ​​determined from the candidate homography transformation matrices based on the number of interior points in the candidate homography transformation matrices.

[0074] Optionally, the global optimization unit is also used to select a reference image frame from the set of image frames and use the pose of the reference image frame as the unit pose; Based on the homography transformation matrix, the initial poses of the remaining image frames in the image frame set are calculated sequentially. Construct a global cost matrix based on the unit pose, initial pose, and target homography transformation matrix; An iterative optimization algorithm is used to iteratively optimize the global cost matrix to obtain the global transformation matrix.

[0075] Optionally, the seam determination unit is also used to determine the overlapping area of ​​each adjacent image frame in the image frame set; Superpixel segmentation is performed on the overlapping region to obtain superpixel units; An energy cost function is defined based on the difference in color and texture between adjacent superpixel units, and a weight map is constructed based on the energy cost function. The minimum cut algorithm is used to segment the weighted graph to obtain the minimum energy cost of the weighted graph after the cut. The boundary corresponding to the cut edge path with the minimum energy cost is defined as the splicing seam.

[0076] Optionally, the conversion module 320 is also used to convert multiple video streams separately to obtain corresponding video frame sequences; The video frame sequence is preprocessed to obtain a grayscale frame sequence; Denoising and enhancement processes are applied to the grayscale frame sequence to obtain at least one image frame; The image frames from multiple video streams are sorted and associated according to time to obtain at least one set of image frames.

[0077] The video splicing device provided in the embodiments of the present invention can execute the video splicing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0078] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0079] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0080] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0081] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0082] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video stitching methods.

[0083] In some embodiments, the video stitching method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video stitching method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video stitching method by any other suitable means (e.g., by means of firmware).

[0084] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0085] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0088] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0089] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system. This addresses the shortcomings of traditional physical hosts and dedicated virtual services, such as high management difficulty and weak business scalability.

[0090] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video stitching method, characterized in that, include: Acquire multiple video streams; The multiple video streams are converted to obtain at least one set of image frames; the set of image frames consists of image frames corresponding to the multiple video streams at the same time. Determine whether the preset stitching template has been loaded. If loading fails, generate a second stitching template based on the image frame set. The image frames in the image frame set are stitched together according to the second stitching template to obtain the target stitched image frame.

2. The method according to claim 1, characterized in that, Also includes: If loading is successful, the image frames in the image frame set are stitched together according to the preset stitching template to obtain the initial stitched image frame; An anomaly detection algorithm is used to detect anomalies in the initial stitched image frame to obtain abnormal values. The abnormal values ​​are compared with a preset threshold to obtain the evaluation result; If the evaluation result is normal, then the initial stitched image frame is used as the target stitched image frame.

3. The method according to claim 1, characterized in that, If loading fails, a second stitching template is generated based on the image frame set, including: The image frames in the image frame set are processed by a registration algorithm to obtain the target homography transformation matrix; The target homography matrix is ​​iteratively optimized using a global optimization algorithm to obtain the globally optimal transformation matrix; The image frame set is processed using the graph cut method to obtain the stitching seam; A global fusion mask is generated based on the stitching seam, and the global optimal transformation matrix and the global fusion mask are used as the second stitching template.

4. The method according to claim 3, characterized in that, The step of processing each image frame in the image frame set using a registration algorithm to obtain the target homography transformation matrix includes: Feature extraction is performed on each image frame in the image frame set to obtain the feature points corresponding to each image frame; The feature points are matched using a matching algorithm to obtain at least one pair of matching points; Iterative extraction is performed on the at least one pair of matching points to obtain at least one set of matching points; At least one candidate homography transformation matrix is ​​obtained by calculating each of the at least one set of matching points; The target homography transformation matrix is ​​determined from the candidate homography transformation matrices based on the number of interior points in the candidate homography transformation matrices.

5. The method according to claim 3, characterized in that, The step of iteratively optimizing the target homography matrix using a global optimization algorithm to obtain the globally optimal transformation matrix includes: A reference image frame is selected from the set of image frames, and the pose of the reference image frame is used as the unit pose. Based on the homography transformation matrix, the initial poses of the remaining image frames in the image frame set are calculated sequentially. A global cost matrix is ​​constructed based on the unit pose, the initial pose, and the target homography transformation matrix; The global cost matrix is ​​iteratively optimized using an iterative optimization algorithm to obtain the global transformation matrix.

6. The method according to claim 3, characterized in that, The step of processing each adjacent image frame in the image frame set using the graph cut method to obtain the stitching seam includes: Determine the overlapping region of each adjacent image frame in the image frame set; The overlapping region is segmented into superpixel units. An energy cost function is defined based on the difference in color and texture between adjacent superpixel units, and a weight map is constructed based on the energy cost function. The weighted graph is segmented using the minimum cut algorithm to obtain the minimum energy cost of the weighted graph after the cut. The boundary corresponding to the cut edge path to which the minimum energy cost belongs is defined as the splicing seam.

7. The method according to claim 1, characterized in that, The conversion of the multiple video streams to obtain at least one set of image frames includes: The multiple video streams are converted separately to obtain the corresponding video frame sequences; The video frame sequence is preprocessed to obtain a grayscale frame sequence; The grayscale frame sequence is denoised and enhanced to obtain at least one image frame; The image frames of the multiple video streams are sorted and associated according to time to obtain at least one set of image frames.

8. A video splicing device, characterized in that, include: The acquisition module is used to acquire multiple video streams. A conversion module is used to convert the multiple video streams to obtain at least one set of image frames; the set of image frames consists of image frames corresponding to the multiple video streams at the same time. The splicing template generation module is used to determine whether the preset splicing template has been loaded. If loading fails, a second splicing template is generated based on the image frame set. The stitching module is used to stitch the image frames in the image frame set according to the second stitching template to obtain the target stitched image frame.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video stitching method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video stitching method according to any one of claims 1-7.

Images