Method, device, electronic device and storage medium for video splicing in dynamic scenes

Abstract

The embodiments of the present invention relate to the field of video processing and disclose a method, device, electronic device and storage medium for video stitching in dynamic scenes. The method includes determining a first-frame registration image from each frame of images to be stitched extracted from a video stream from a multi-camera, using the transformation matrix used by the first-frame registration image in the registration process as the initial transformation matrix, and its stitching line as the initial stitching line. An initial background model is established based on the first-frame registration image. For subsequent frames to be stitched, when the background model detects that the current video frame image contains a moving object, the transformation matrix is ​​updated based on the current frame to be stitched to re-register and determine a new stitching line, and the background model is updated. Otherwise, the transformation matrix, background model and stitching line of the previous frame are maintained unchanged, and real-time fusion is performed based on the stitching line of each frame to be stitched to obtain a stitched video. The method meets the stability and real-time requirements of panoramic video stitching of mobile unmanned carriers.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video processing, and in particular to a method, device, electronic device, and storage medium for video splicing in dynamic scenes. Background Art

[0002] Real-time panoramic video surveillance systems, as a wider field of view extension of a single camera, can be applied to smart IoT fields such as smart transportation, smart venues, and smart industry. The biggest difference between video stitching and image stitching is the presence of moving objects in the video, especially when the camera is also in motion. This greatly increases the complexity of video stitching. In a mobile environment, the camera and the object being photographed are in relative motion, and the scale and lighting of the images captured by the camera are constantly changing.

[0003] Current image stitching algorithms are mostly focused on fixed video surveillance areas, and are unable to meet the current requirements for stability and real-time panoramic video stitching of unmanned vehicles in motion. Summary of the Invention

[0004] The purpose of the present invention is to at least provide a method, device, electronic device and storage medium for video stitching in dynamic scenes, which can at least solve the problem that current image stitching algorithms are difficult to meet the needs of real-time video stitching in dynamic scenes, and at least achieve the effect of stable and real-time video stitching in dynamic scenes.

[0005] To solve the above technical problems, at least one embodiment of the present application provides a method for video stitching in a dynamic scene, including:

[0006] Extracting a plurality of images to be stitched in each frame based on a video stream from a multi-camera, wherein two adjacent cameras in the multi-camera have overlapping shooting areas, and the images to be stitched from the corresponding two adjacent cameras have overlapping areas;

[0007] Determining a first-frame registration image among all images to be stitched, using a transformation matrix used in a registration process of the first-frame registration image as an initial transformation matrix, using a stitching line calculated based on an overlapping area of ​​the first-frame registration image as an initial stitching line, and establishing an initial background model based on the first-frame registration image, wherein the transformation matrix is ​​calculated based on feature points of an overlapping area of ​​adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on a Vibe algorithm, which is used to determine a key frame image having a moving target in a subsequent frame of images to be stitched;

[0008] For subsequent frames of images to be stitched, when the background model detects that the current video frame image is the key frame image, feature information of adjacent images to be stitched of the current frame is extracted to update the transformation matrix, the images to be stitched of the current frame are registered based on the updated transformation matrix, a new stitching line is re-determined within the overlapping area of ​​the registered aligned images of the current frame, and the background model is updated based on the registered aligned images of the current frame to determine whether the next frame of image to be stitched is a key frame image; otherwise, the transformation matrix, the background model, and the stitching line of the previous frame are maintained unchanged, and the images to be stitched of the current frame are processed according to the transformation matrix and the stitching line of the previous frame;

[0009] The images to be stitched are fused in real time based on the stitching lines of each frame to obtain a stitched video.

[0010] At least one embodiment of the present application further provides a device for video splicing in a dynamic scene, including:

[0011] A module for extracting images to be stitched is configured to extract a plurality of images to be stitched in each frame based on a video stream from a multi-camera, wherein two adjacent cameras in the multi-camera have overlapping shooting areas, and the images to be stitched from the corresponding two adjacent cameras have overlapping areas;

[0012] An initialization module is configured to determine a first-frame registration image among all images to be stitched, use a transformation matrix used by the first-frame registration image in the registration process as an initial transformation matrix, use a stitching line calculated based on an overlapping area of ​​the first-frame registration image as an initial stitching line, and establish an initial background model based on the first-frame registration image, wherein the transformation matrix is ​​calculated based on feature points of an overlapping area of ​​adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on a Vibe algorithm, which is configured to determine key frame images having a moving target in subsequent frames of images to be stitched;

[0013] an updating module configured to, for a subsequent frame image to be stitched, extract feature information of adjacent images to be stitched of the current frame to update the transformation matrix when the background model detects that the current video frame image is the key frame image; register the current frame image to be stitched based on the updated transformation matrix; re-determine a new stitching line within an overlapping area of ​​the registered aligned images of the current frame; and update the background model based on the registered aligned images of the current frame to determine whether the next frame image to be stitched is a key frame image; otherwise, maintain the transformation matrix, the background model, and the stitching line of the previous frame unchanged, and process the current frame image to be stitched according to the transformation matrix and the stitching line of the previous frame;

[0014] The fusion module is used to fuse the images to be stitched in real time based on the stitching lines of each frame of the image to be stitched to obtain a stitched video.

[0015] At least one embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned method of video stitching in dynamic scenes.

[0016] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for video splicing in a dynamic scene.

[0017] The embodiments of the present application provide a method, apparatus, electronic device, and storage medium for video stitching in dynamic scenes. These methods determine the first frame of the registered image and use the transformation matrix used in its registration process as the initial transformation matrix. The stitching line calculated based on the overlapping area is used as the initial stitching line, and an initial background model is established. This initialization method based on the first frame provides a foundation for processing subsequent frames, giving the entire video stitching process a reasonable starting point and avoiding the need to calculate the transformation matrix and stitching line from scratch for each frame, thereby improving processing efficiency. The background model accurately determines keyframe images with moving objects in the images to be stitched in subsequent frames. In dynamic scenes, the presence of moving objects can cause changes in image content, which traditional stitching methods may not be able to effectively handle. However, the present application detects keyframes and only performs operations such as updating the transformation matrix, re-determining the stitching line, and updating the background model for the keyframes, thereby avoiding complex calculations for each frame and improving processing speed. When a keyframe is detected, feature information of the adjacent images to be stitched in the current frame is extracted to update the transformation matrix. The images to be stitched in the current frame are registered based on the updated transformation matrix, and a new stitching line is re-determined. At the same time, the background model is updated based on the aligned image of the current frame after registration to determine whether the next frame to be spliced ​​is a key frame image. This dynamic update mechanism can adapt to changes in the scene in a timely manner to ensure the accuracy and consistency of splicing; for non-key frames, the transformation matrix, background model and stitching line of the previous frame are maintained unchanged, and the current frame to be spliced ​​is processed according to the transformation matrix and stitching line of the previous frame. This method avoids frequent parameter updates due to minor changes or noise in the scene, ensures the stability of the splicing process, reduces splicing errors that may be caused by frequent parameter changes, and greatly reduces the amount of calculation. Therefore, the present application effectively improves the efficiency, accuracy and stability of real-time video splicing in dynamic scenes by adapting to the key frame processing mechanism of dynamic scenes, and can provide high-quality spliced ​​videos.

[0018] In some optional embodiments, extracting multiple images to be stitched in each frame based on the video stream from the multi-camera includes:

[0019] Based on the video stream of each camera in the multi-camera, a set of original image sequences is correspondingly extracted;

[0020] Each set of original image sequences is preprocessed using a cylindrical projection method, wherein the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each set of original image sequences and the photographed object.

[0021] In this embodiment, cylindrical projection can be used to convert planar images captured by different cameras into cylindrical images, making the stitched image more consistent with human visual habits and reducing the perception of distortion. The cylindrical projection parameters are dynamically adjusted based on the distance between the camera and the object being photographed, corresponding to each set of original image sequences. Different shooting distances result in different image viewing angles and degrees of distortion. Dynamic adjustment of the cylindrical projection parameters can better adapt to these changes, further improving the quality of the projected image.

[0022] In some optional embodiments, re-determining a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes:

[0023] Performing illumination compensation on the registered image of the current frame to obtain an illumination compensated image;

[0024] A stitching line is determined within the overlapping region of the illumination-compensated images.

[0025] In this embodiment, illumination compensation is performed on the registered, aligned images of the current frame to produce an illumination-compensated image, which can bring the illumination conditions in the overlapping area closer to uniformity. Determining stitching lines within the overlapping area of ​​the illumination-compensated image and performing stitching effectively eliminates stitching artifacts caused by illumination differences, resulting in a more natural and smooth transition in the stitched image.

[0026] In some optional embodiments, re-determining a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes:

[0027] Applying the optical flow detection algorithm to insert the horizontal and vertical displacement parameters into the aligned image of the current frame after registration to perform displacement compensation and obtain a displacement compensated image;

[0028] The stitching line is determined in an overlapping region of the displacement-compensated images.

[0029] In this embodiment, when the background undergoes drastic global changes during camera movement, optical flow detection is used to further stabilize the video frames and eliminate the displacement differences between adjacent images to be stitched, thereby reducing the stitching error caused by the displacement and improving the stitching accuracy.

[0030] In some optional embodiments, the process of determining the stitching line includes: defining an energy function based on the grayscale difference between the two displacement-compensated images to be stitched;

[0031] In the overlapping region, the stitching line is obtained by dynamic programming by minimizing the cumulative energy.

[0032] In this embodiment, the dynamic programming method is used to search for the seam line with the minimum energy in the overlapping area, which can effectively improve the ghosting and blurring phenomena when the moving object passes through the seam line.

[0033] In some optional embodiments, the energy function is calculated as follows:

[0034] E(x,y)=|I1(x,y)-I2(x+Δ x (x,y),y+Δ y (x,y))|

[0035] Wherein, I1(x, y) and I2(x, y) represent the grayscale values ​​of the two displacement compensation images, Δ x (x,y) and Δ y (x, y) are the horizontal and vertical displacements obtained by the optical flow detection algorithm at point (x, y);

[0036] The calculation formula of the cumulative energy is as follows:

[0037] C(x,y)=E(x,y)+min{C(x-1,y-1),C(x,y-1),C(x+1,y-1)};

[0038] Where E(x,y) is the energy at the current pixel, C(x-1,y-1), C(x,y-1), and C(x+1,y-1) are the cumulative energies of the pixels to the upper left, directly above, and upper right of the current pixel, respectively.

[0039] In some optional embodiments, the real-time fusing of the images to be stitched based on the stitching lines of each frame of the images to be stitched includes:

[0040] Multi-band fusion is performed on the images to be stitched based on the stitching lines of each frame of the images to be stitched.

[0041] In this embodiment, traditional stitching methods may produce obvious stitching artifacts at the stitching line, such as sudden brightness changes and color discontinuities. Multi-band fusion can adopt different fusion strategies for different frequency sub-bands based on the location of the stitching line, making the transition between the stitching area and the surrounding image more natural.

[0042] In some optional embodiments, performing multi-band fusion on the images to be stitched based on the stitching lines of each frame of image includes:

[0043] At each pyramid level, the Laplacian coefficients of the two images are fused in a weighted manner:

[0044]

[0045] in, and are the Laplacian pyramids of the two images respectively, is the original smooth transition weight, Φ(·) is the function adjusted according to the motion amplitude;

[0046] From the fused Laplacian pyramid Reconstruct the fused image, and the recursive calculation formula is as follows:

[0047]

[0048] Among them, F i is the image at the i-th layer of the fused Laplacian pyramid, F N-1 is the top image of the fused Laplacian pyramid, Expand(F i+1 ) is the process of expanding the image of the i+1th layer to the i-th layer.

[0049] In this embodiment, is the original smooth transition weight, which plays a role in smoothing the transition and making the fusion boundary more natural. Φ(·) is a function that adjusts according to the motion amplitude. When the motion amplitude is large, the smoothing effect of the weight is reduced to expand the transition area and make the image fusion more natural.

[0050] In some optional embodiments, the method uses CUDA for accelerated processing during execution.

[0051] In this embodiment, real-time video stitching involves a large amount of image data processing, such as feature extraction, image registration, pixel fusion, and other operations. By processing these operations in parallel through CUDA, the processing speed of video stitching can be significantly improved to meet the real-time requirements in mobile scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0053] Figure 1 This is the process of a method for video splicing in a dynamic scene provided by an embodiment of the present application. Figure 1 ;

[0054] Figure 2 It is a flowchart of the existing feature point extraction algorithm;

[0055] Figure 3 is a schematic diagram of overlapping areas of adjacent images to be stitched provided by an embodiment of the present application;

[0056] Figure 4 This is the flow chart of the existing Vibe algorithm for detecting foreground points;

[0057] Figure 5 This is a flow chart of an improved Vibe algorithm that introduces SURF feature point matching, provided by an embodiment of the present application;

[0058] Figure 6 This is the process of a method for video splicing in a dynamic scene provided by an embodiment of the present application. Figure 2 ;

[0059] Figure 7 This is a schematic diagram of the structure of a multi-camera provided by an embodiment of the present application;

[0060] Figure 8 3 is a structural diagram of a device for video splicing in a dynamic scene provided by another embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0062] To facilitate understanding of the embodiments of the present application, relevant content about existing video splicing is first introduced here.

[0063] Prior art 1: A four-camera panoramic stitching method, which uses four cameras to collect image information in the same space and imports the image information into a CMOS image sensor; uses the CMOS image sensor to convert the image information light signal into a digital image signal, and then inputs it into a DSP digital signal processing chip for image information preprocessing; uses the SIFT algorithm to extract key points of the digital image information in the DSP digital signal processing chip collected from different angles by the four cameras, and matches the panoramic image information in the four cameras based on the key point extraction to obtain a first stitched panoramic image output by the matching output; identifies dynamic targets by calculating the pixel difference between the first stitched panoramic image and the second stitched panoramic image of the adjacent frame; uses the background modeling method to construct a panoramic image background, and integrates the dynamic target as the foreground into the constructed panoramic image background to stitch out a dynamic image.

[0064] Among the above methods, one is to collect image information in the same space through four cameras, which is limited to the stitching of dynamic images in fixed situations, and does not mention how to achieve real-time panoramic video stitching in a mobile environment; the other is to extract key points of image information through the SIFT algorithm. This algorithm is complex to calculate and time-consuming, and is not applicable to panoramic video stitching in mobile situations.

[0065] Prior art 2: A panoramic stitching method and system based on a pan-tilt device: the method includes: acquiring an image containing a target, and determining the starting point and the ending point of the panoramic stitching within the rectangular boundary frame of the image containing the target; acquiring a capture movement trajectory based on the starting point and the ending point of the panoramic stitching and the total number of times the rectangular field of view frame of the imaging device moves in the horizontal and vertical directions respectively; controlling the movement of the pan-tilt device so that the rectangular field of view frame of the imaging device starts from the starting position of the movement and moves along the capture movement trajectory until it reaches the end position of the movement, and when the rectangular field of view frame is at the starting position of the movement, each time it moves, and at the end position of the movement, a corresponding image is acquired by the imaging device connected to the pan-tilt device; and stitching the images acquired by all imaging devices to obtain a panoramic stitching image.

[0066] This method mentions acquiring an image containing a target, and determining the starting point and the ending point of the panoramic stitching within the rectangular bounding box of the image containing the target. In the case of movement, the area for acquiring the image is determined by the target, and the field of view is limited; secondly, when the rectangular field of view box is located at the starting position of the movement, each time it moves, and at the end position of the movement, the corresponding image is acquired by the imaging device connected to the pan-tilt head, and the images acquired by all the imaging devices are stitched together to obtain a panoramic stitched image. Acquiring an image once per movement is limited to the stitching of panoramic images, and does not have the real-time nature of video stitching, and cannot perform real-time panoramic monitoring of emergencies in mobile situations. Thirdly, the images are captured at regular intervals according to the automatically planned capture path, and stitched together to ultimately obtain the complete features of the larger target. Compared with identifying and marking targets of interest in real-time panoramic videos, this method does not have advantages.

[0067] In this regard, the technical problem to be solved by this application is: in a mobile environment, the camera and the photographed object are in relative motion, and the scale, lighting, etc. of the image captured by the camera are constantly changing. If the 360-degree panoramic video surveillance system is combined with an unmanned carrier, how to ensure the stability and real-time nature of the video frame splicing.

[0068] In order to solve the above technical problems, the present invention proposes a method for video splicing in dynamic scenes. The implementation details of the method for video splicing in dynamic scenes in this embodiment are specifically described below. The following content is only the implementation details provided for easy understanding and is not necessary for implementing this solution.

[0069] Example 1:

[0070] The method for video splicing in dynamic scenes of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 Shown, including:

[0071] Step 110: extracting a plurality of images to be stitched in each frame based on the video stream from the multi-camera, wherein two adjacent cameras in the multi-camera have overlapping shooting areas, and the images to be stitched from the corresponding two adjacent cameras have overlapping areas;

[0072] In this embodiment, the multi-camera array can be a camera array arranged in a circular or linear manner, with adjacent cameras having overlapping fields of view. In dynamic scenes, the overlapping areas provide correlation information between adjacent images and are the key to image stitching. Image data is extracted frame by frame from the video stream, and for each frame, multiple images to be stitched are obtained based on the number and layout of the cameras. For example, if there are three cameras, three images to be stitched are obtained for each frame. These images to be stitched have overlapping areas between adjacent cameras, and these overlapping areas provide the basis for subsequent image registration and stitching.

[0073] Step 120: Determine a first-frame registered image among all images to be stitched, use a transformation matrix used in the registration process of the first-frame registered image as an initial transformation matrix, use a stitching line calculated based on an overlapping area of ​​the first-frame registered image as an initial stitching line, and establish an initial background model based on the first-frame registered image, wherein the transformation matrix is ​​calculated based on feature points in an overlapping area of ​​adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on a Vibe algorithm, which is used to determine key frame images with moving targets in subsequent frames of images to be stitched;

[0074] In this embodiment, the first-frame registration image is determined among all the images to be spliced. The image to be spliced ​​of the initial frame of the video stream can be selected for registration as the first-frame registration image. Alternatively, the image to be spliced ​​of a subsequent frame can be selected for registration as the first-frame registration image. Usually, a frame of a static scene (without significant moving targets) is selected as the initial registration reference to improve the accuracy of the transformation matrix, stitching line, and background model. When the initial frame of the video stream has no motion interference, the image to be spliced ​​of the initial frame of the video stream can be selected for registration. If there is a moving target in the initial frame, the first static frame can be searched in the subsequent frames as the first-frame registration image.

[0075] In the registration process of the first-frame registration image, the transformation matrix is ​​first calculated based on the feature points of the overlapping areas of adjacent images to be stitched from adjacent cameras. Commonly used feature point extraction algorithms include SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features). Through feature point matching, the corresponding feature point pairs in the overlapping areas of adjacent images are found, and then these feature point pairs are used to calculate the transformation matrix. The transformation matrix describes the geometric transformation relationship between adjacent images, such as translation, rotation, scaling, etc. In the overlapping area of ​​the first-frame registration image, the stitching line is calculated based on the image's color, gradient and other information. The purpose of the stitching line is to find an optimal path so that the stitched image transitions naturally at the stitching line, reducing the visual discontinuity.

[0076] During the execution of the traditional feature point extraction algorithm, feature points are extracted and described for all areas of the stitched image. The process is as follows: Figure 2 This embodiment improves the feature point extraction algorithm by controlling the feature point detection threshold S in response to feature changes in the surrounding scene captured in a mobile environment and combining the overlapping area ratio relationship of adjacent images in a panoramic camera frame.

[0077] Specifically, the overlapping area of ​​adjacent images to be stitched can be as follows: Figure 3 As shown. The overlapping range of the images captured by adjacent cameras accounts for more than 30% of a single captured image, and there is a certain error in the physical installation position of the camera. Therefore, the detection threshold is set to 30% in the feature point extraction algorithm (referring to the detection area accounting for 30% of a single image). For the left view image, only the 30% overlapping area on the right is detected; for the right view image, only the 30% overlapping area on the left is detected; if there are fewer feature points in the overlapping area of ​​the video frame collected at a certain moment during the movement, the threshold is relaxed to the entire overlapping area. By controlling the detection range of the overlapping area between adjacent images, on the one hand, the relative error of the overlapping area caused by the physical placement method between adjacent cameras can be improved, the detected overlapping range can be reduced, and the execution efficiency of the algorithm can be improved; on the other hand, the detection and description of a large number of redundant feature points can be avoided, and the interference caused by mismatched points in non-stitched areas can be reduced.

[0078] An initial background model is established based on the first-frame registered image. This background model includes a background subtraction model based on the Vibe (Visual Background Extraction) algorithm. The Vibe algorithm analyzes image sequences to establish a background model, effectively detecting foreground moving objects. In the initial stage, the first-frame registered image is used to initialize the Vibe model's parameters, such as the sample set and matching threshold. The background model is then used to identify key frames containing moving objects in subsequent frames, allowing for dynamic updates of the transformation matrix and seam lines.

[0079] Step 130: for subsequent frames of images to be stitched, when the background model detects that the current video frame image is the key frame image, extract feature information of adjacent images to be stitched of the current frame to update the transformation matrix, register the current frame of images to be stitched based on the updated transformation matrix, re-determine a new stitching line within the overlapping area of ​​the registered aligned images of the current frame, and update the background model based on the registered aligned images of the current frame to determine whether the next frame of images to be stitched is a key frame image; otherwise, maintain the transformation matrix, the background model, and the stitching line of the previous frame unchanged, and process the current frame of images to be stitched according to the transformation matrix and stitching line of the previous frame;

[0080] In this embodiment, the background model is used to detect subsequent frames to be stitched. The background model determines whether the current frame is a key frame based on the difference between the current frame and the background model. If the current frame contains moving objects and the difference between the current frame and the background model is significant, the current frame is determined to be a key frame. For example, in a surveillance scene, if a pedestrian or vehicle enters the frame, the background model will detect these moving objects and determine that the current frame is a key frame.

[0081] When it is detected that the current frame image is a key frame image, the following processing is performed:

[0082] (1) Extract feature information of the adjacent images to be stitched in the current frame. You can also use feature point extraction algorithms such as SIFT and SURF to obtain feature points in the overlapping area of ​​adjacent images.

[0083] (2) Update the transformation matrix based on the extracted feature information. Since the moving objects in the scene may cause the geometric relationship between images to change, the transformation matrix needs to be recalculated to ensure the accuracy of image registration.

[0084] (3) The images to be stitched in the current frame are registered according to the updated transformation matrix, and adjacent images are aligned to the same coordinate system.

[0085] (4) Re-determine the new stitching line in the overlapping area of ​​the aligned images of the current frame after registration. Since the image content has changed, the original stitching line may no longer be applicable, so the stitching line needs to be recalculated to ensure the quality of the stitched image.

[0086] (5) Update the background model based on the aligned image of the current frame after registration. The information of the current frame is integrated into the background model to more accurately detect whether the next frame is a key frame image.

[0087] Specifically, the Vibe algorithm first uses background subtraction to detect significant background changes. When the background undergoes significant changes, the algorithm discards the original background model and recreates a new background model using the first frame image (i.e., the keyframe image) after the changes. The Vibe algorithm's foreground object detection process compares all pixels in each frame image with those in its background model, calculates the Euclidean distance between pixels, and then uses a set threshold to determine whether the current pixel belongs to the foreground object or the background. After the classification is complete, the foreground points in the image are counted to obtain information about the moving object, and the process is accelerated using CUDA.

[0088] like Figure 4 The figure shows the flow chart of the existing Vibe algorithm for detecting foreground points. There are three situations in which the Vibe algorithm updates the background model:

[0089] 1. For a foreground pixel, if it is judged as a foreground point N times in a row, it will be updated to a background pixel to solve the problem of a moving target staying in the foreground for a long time;

[0090] Second, for each pixel background model there is The probability of is updated;

[0091] 3. When each pixel is updated The probability of propagating the update to the neighboring pixels.

[0092] like Figure 5 The figure shows a flowchart of the improved Vibe algorithm for SURF feature point matching in this embodiment. After obtaining a binary image of the foreground area through background subtraction using the Vibe algorithm, it is possible to determine whether the stitching line calculated in the previous stitching frame can be used for the current stitching frame. If any pixels on the stitching line calculated in the previous stitching frame fall within the foreground area, the stitching line needs to be recalculated for the current stitching frame. Otherwise, the stitching line calculated in the previous stitching frame can continue to be used for the current stitching frame.

[0093] It is understood that during the image fusion stage, the biggest difference between video fusion and image fusion technology lies in the presence of real-time moving objects in the video. Therefore, when determining the stitching line, moving objects must be avoided to prevent misalignment, ghosting, blurring, and other phenomena in the fused image. However, if conventional image fusion methods are used for video fusion with camera motion, these misalignment, ghosting, and blurring will recur when a moving object passes through the stitching line. Calculating the stitching line for each frame will also not meet real-time requirements. Therefore, this embodiment first uses background subtraction to detect moving objects in the video frames using selected keyframes. When a moving object passes through the stitching line, the stitching line is dynamically updated. If the current frame is detected as a non-keyframe image, meaning that the background model detects that the current frame differs little from the background model and there are no obvious moving objects, the transformation matrix, background model, and stitching line of the previous frame are maintained unchanged. The current frame to be stitched is processed according to the transformation matrix and stitching line of the previous frame, and image registration and stitching are performed directly. Feature point detection and perspective transformation matrix calculation are performed on selected key frames, without having to do it for every frame, which greatly reduces the amount of calculation and improves processing efficiency.

[0094] Step 140 : The images to be stitched are fused in real time based on the stitching lines of each frame of the image to be stitched to obtain a stitched video.

[0095] In this embodiment, for each frame of the image to be stitched, adjacent images are fused based on the calculated stitching line. On one side of the stitching line, the color value of one image is selected; on the other side of the stitching line, the color value of the other image is selected. A gradient transition can be used near the stitching line to reduce stitching artifacts.

[0096] The images to be stitched are fused frame by frame to achieve real-time video stitching. The fused image sequence is encoded according to a video format and the stitched video is output. During the real-time fusion process, it is important to balance processing speed and image quality. Optimized algorithms and hardware acceleration technologies can be used to improve processing efficiency and ensure the real-time and stability of video stitching.

[0097] The first approach is to stitch two images together. Directly transforming one image into the coordinate system of the other through perspective transformation can cause distortion at the edges of the panoramic image, as perspective transformation stretches or compresses different parts of the image to varying degrees depending on the viewpoint. This can also contribute to distortion in the stitched image. For stitching multiple images together, selecting a reference image and transforming the other images into that coordinate system through perspective transformation can increase the cumulative error. Adjustments to the image stitching strategy (stitching images together) and projection model (such as cylindrical projection) are necessary.

[0098] In summary, the method for video stitching in dynamic scenes of this embodiment determines the first frame of the registered image and uses the transformation matrix used in the registration process as the initial transformation matrix. The stitching line calculated based on the overlapping area is used as the initial stitching line, and an initial background model is established. This initialization method based on the first frame provides a foundation for the processing of subsequent frames, giving the entire video stitching process a reasonable starting point, avoiding the need to calculate the transformation matrix and stitching line from scratch for each frame, and improving processing efficiency. The background model can accurately determine the key frame images with moving objects in the subsequent frames to be stitched. In dynamic scenes, the presence of moving objects can cause changes in image content, which may not be effectively handled by traditional stitching methods. However, the present application detects key frames and only performs operations such as updating the transformation matrix, re-determining the stitching line, and updating the background model for these key frames, avoiding complex calculations for each frame and improving processing speed. When a key frame is detected, feature information of the adjacent images to be stitched in the current frame is extracted to update the transformation matrix. The images to be stitched in the current frame are registered based on the updated transformation matrix, and a new stitching line is re-determined. At the same time, the background model is updated based on the aligned image of the current frame after registration to determine whether the next frame to be spliced ​​is a key frame image. This dynamic update mechanism can adapt to changes in the scene in a timely manner to ensure the accuracy and consistency of splicing; for non-key frames, the transformation matrix, background model and stitching line of the previous frame are maintained unchanged, and the current frame to be spliced ​​is processed according to the transformation matrix and stitching line of the previous frame. This method avoids frequent parameter updates due to minor changes or noise in the scene, ensures the stability of the splicing process, reduces splicing errors that may be caused by frequent parameter changes, and greatly reduces the amount of calculation. Therefore, the present application effectively improves the efficiency, accuracy and stability of real-time video splicing in dynamic scenes by adapting to the key frame processing mechanism of dynamic scenes, and can provide high-quality spliced ​​videos.

[0099] In some optional embodiments, the images to be stitched may be stitched together using an image-to-stitched-image approach and / or a stitched-image-to-stitched-image approach. The image-to-stitched-image approach involves selecting a reference image from all the images to be stitched and then transforming the remaining non-reference images into that coordinate system. When stitching multiple images, each perspective transformation is based on a locally optimal match between the current image and the reference image and may not necessarily be globally optimal.

[0100] Image-to-image stitching involves dividing the image to be stitched into multiple sub-images based on their matching degree, fusing these sub-images individually, and finally stitching the fused sub-images together to form a complete panoramic image. This method helps reduce the cumulative error generated during the stitching process. To address the distortion issues associated with stitching multiple images, a combination of the two aforementioned stitching strategies can be used.

[0101] To address the problem of eliminating distortion when stitching multiple images, in some optional embodiments, all adjacent images to be stitched can be registered to obtain feature point information. Using the feature point information, the transformation matrix between all adjacent images is calculated, and then the global transformation matrix H from each input image to the reference image is obtained according to the following formula:

[0102]

[0103] Here, H[i] represents the transformation matrix obtained from each set of adjacent images, and k represents the middle index in the sequence. The transformation matrix from the input image to the reference image is obtained by statistically analyzing the matrix set. Due to the transitivity of matrices, the transformation matrix from any image to the reference image can be obtained. The LM algorithm is then used to adjust the transformation matrix to further reduce the cumulative matching error, completing the image registration process. This reduces the distortion of the final stitched image, while maintaining the required accuracy of image registration.

[0104] In some optional embodiments, the method of extracting multiple images to be stitched in each frame based on the video stream from the multi-camera includes: extracting a corresponding set of original image sequences based on the video stream of each camera in the multi-camera; and preprocessing each set of original image sequences using a cylindrical projection method, wherein the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each set of original image sequences and the photographed object.

[0105] In the specific implementation, after extracting multiple groups of original image sequences from multiple video streams, cylindrical projection is used to preprocess the original images of each camera to reduce the degree of image distortion during perspective transformation.

[0106] During movement, the camera adaptively adjusts the focal length of the optimal cylindrical projection based on the distance to the object being photographed. The optimal adjustment parameters are determined by the initial measurement values. CUDA is used to accelerate the processing of single-frame images, significantly reducing processing time and effectively ensuring the real-time performance of video stitching.

[0107] The cylindrical projection process includes the following:

[0108] (1) Let the pixel coordinates of the original image be (x, y), where x∈[0, W], y∈[0, H], W and H are the image width and height respectively. The normalized coordinates with the image center as the origin are:

[0109]

[0110] (2) Set the camera focal length to f, and the projection coordinates of each pixel (X c ,Y c ,Z c ) is given by the following formula:

[0111] Z c =f

[0112] Among them, X c The input image x′ is projected onto the cylinder through the arc tangent function, Y c Through triangular relationship calculation, the vertical proportion is kept consistent.

[0113] (3) The new pixel position after projection is given by the following formula:

[0114]

[0115] (4) Finally, bilinear interpolation is used to calculate the color value of the target pixel. The mapping relationship between the projected image and the original image is as follows: I′(x″,y″)=I(x,y)

[0116] In this embodiment, cylindrical projection can be used to convert planar images captured by different cameras into cylindrical images, making the stitched image more consistent with human visual habits and reducing the perception of distortion. The cylindrical projection parameters are dynamically adjusted based on the distance between the camera and the object being photographed, corresponding to each set of original image sequences. Different shooting distances result in different image viewing angles and degrees of distortion. Dynamic adjustment of the cylindrical projection parameters can better adapt to these changes, further improving the quality of the projected image.

[0117] In some optional embodiments, re-determining a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes: performing illumination compensation on the aligned images of the current frame after registration to obtain an illumination compensated image; and determining the stitching line in the overlapping area of ​​the illumination compensated image.

[0118] In this embodiment, when multiple cameras are moving and capturing images, due to factors such as weather variations and varying 360-degree horizontal visibility, even data captured at the same moment can experience exposure differences due to factors such as the camera's viewing angle or autofocus. Directly stitching such a set of data would result in noticeable differences in brightness and color within the stitched image, and would also create noticeable seam lines, affecting the quality of the video stitching. Therefore, after image registration, illumination compensation should be performed first, followed by image fusion, to obtain a panoramic stitched image with a natural transition.

[0119] When performing illumination compensation, a block-based image compensation method can be used. This method divides the image into blocks and calculates the compensation coefficient for each block. By building a light intensity error function model, the illumination compensation gain matrix is ​​solved. Finally, the gain coefficients in the matrix are used to perform illumination compensation on the corresponding areas in the source image.

[0120] In some optional embodiments, re-determining a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes: applying an optical flow detection algorithm to insert horizontal and vertical displacement parameters into the aligned images of the current frame after registration to perform displacement compensation, thereby obtaining a displacement compensated image; and determining the stitching line in the overlapping area of ​​the displacement compensated image.

[0121] In this embodiment, an optical flow detection algorithm can be used to further stabilize video frames in situations where camera motion causes dramatic global background changes. After performing illumination compensation or registration, the optical flow field is used to analyze the motion information of each pixel. Horizontal and vertical displacement parameters are then inserted into the aligned image of the current frame to compensate for the displacement. This eliminates the displacement differences between adjacent images to be stitched, thereby reducing stitching errors caused by displacement and improving stitching accuracy.

[0122] In some optional embodiments, the process of determining the stitching line includes: defining an energy function based on the grayscale difference between the two displacement-compensated images to be stitched; and obtaining the stitching line by dynamic programming in the overlapping area by minimizing the cumulative energy.

[0123] In this embodiment, the dynamic programming method is used to search for the seam line with the minimum energy in the overlapping area, which can effectively improve the ghosting and blurring phenomena when the moving object passes through the seam line.

[0124] In some optional embodiments, the energy function is calculated as follows:

[0125] E(x,y)=|I1(x,y)-I2(x+Δ x (x,y),y+Δ y (x,y))|

[0126] Wherein, I1(x, y) and I2(x, y) represent the grayscale values ​​of the two displacement compensation images, Δ x (x,y) and Δ y (x, y) are the horizontal and vertical displacements obtained by the optical flow detection algorithm at point (x, y);

[0127] The calculation formula of the cumulative energy is as follows:

[0128] C(x,y)=E(x,y)+min{C(x-1,y-1),C(x,y-1),C(x+1,y-1)};

[0129] Where E(x,y) is the energy at the current pixel, C(x-1,y-1), C(x,y-1), and C(x+1,y-1) are the cumulative energies of the pixels to the upper left, directly above, and upper right of the current pixel, respectively.

[0130] In this embodiment, the energy function E(x,y) measures the grayscale difference between the two images to be stitched at the point (x,y), taking into account the displacement obtained by the optical flow detection algorithm. The cumulative energy is set to the energy of the current pixel plus the minimum cumulative energy of the pixels to the upper left, directly above, or to the upper right. This function finds the optimal stitching path during the image stitching process, effectively reducing stitching errors and improving stitching accuracy and quality.

[0131] In some optional embodiments, the real-time fusing of the images to be stitched based on the stitching line of each frame of the images to be stitched includes: performing multi-band fusing of the images to be stitched based on the stitching line of each frame of the images to be stitched.

[0132] Compared to other fusion algorithms, the multi-band fusion algorithm in this embodiment utilizes multi-scale image information, enabling smooth transitions in overlapping areas and effectively avoiding noticeable seams and ghosting. The complex computational process is accelerated using CUDA to match the speed of other fusion algorithms. In this embodiment, pixel information from corresponding images on both sides of the seam line is dynamically updated, and improved multi-band fusion is employed to achieve a visually consistent seamless panoramic image.

[0133] Specifically, the multi-band fusion of the images to be stitched based on the stitching lines of each frame includes: at each pyramid level, fusing the Laplace coefficients of the two images in a weighted manner:

[0134]

[0135] in, and are the Laplacian pyramids of the two images respectively, is the original smooth transition weight, Φ(·) is the function adjusted according to the motion amplitude;

[0136] From the fused Laplacian pyramid Reconstruct the fused image, and the recursive calculation formula is as follows:

[0137]

[0138] Among them, F i is the image at the i-th layer of the fused Laplacian pyramid, F N-1 is the top image of the fused Laplacian pyramid, Expand(F i+1 ) is the process of expanding the image of the i+1th layer to the i-th layer.

[0139] In this embodiment, the Laplacian pyramid effectively preserves high-frequency image details, ensuring the fused image remains clear and sharp. By weighting the Laplacian coefficients of the two images, the contribution of each image to the final fusion result can be adjusted as needed, achieving a more natural transition. The smooth transition weight function Φ(·) can be adjusted based on the motion amplitude, providing greater flexibility to adapt to different image content and motion conditions. This helps reduce artifacts and abrupt changes at the splicing point, resulting in a more natural fusion result.

[0140] Example 2:

[0141] Based on the above embodiments, this embodiment provides a specific application example. The method of video splicing in dynamic scenes provided by this embodiment can be applied to a real-time panoramic monitoring system carried by an unmanned carrier, making it intelligent. When the unmanned carrier is manipulated to enter a dangerous area, real-time full-field detection can be performed. For example, installing a real-time panoramic monitoring system on a ship sailing in the ocean will enhance the breadth and speed of ship identification, providing strong support for maritime safety and law enforcement operations. Figure 6 FIG. 1 is a schematic diagram of the overall process of the method for video splicing in a dynamic scene in this embodiment.

[0142] Specifically, firstly, in terms of image acquisition, a ring camera array is formed by using multiple industrial camera modules to collect horizontal 360-degree real-time video streams with overlapping areas. Figure 7 Figure 2 shows the schematic diagram of the multi-camera structure in this embodiment. Since the viewing angle of common industrial cameras is between 90 and 120 degrees, this embodiment uses a circular array of eight industrial camera modules to capture 360-degree images in different directions. This ensures that the overlap between adjacent camera images exceeds 30% of a single captured image, ensuring that the panoramic image can be aligned and stitched.

[0143] In preprocessing the first frame of the multi-camera image, cylindrical projection is used to reduce the degree of distortion during image transformation. An improved SURF registration algorithm is used to align the images within the overlapping region of the images, stitching them together from image to stitched image. Perspective transformation is then used to transform the cylindrical images from different perspectives onto a reference plane. Before fusion, an illumination gain compensation algorithm is used to eliminate exposure differences between the images captured by the multi-camera. A fusion algorithm combining an improved optimal stitching line and a multi-band fusion algorithm is then used to avoid ghosting and blurring when moving objects cross the stitching line. This allows for horizontal 360-degree panoramic video stitching in mobile environments.

[0144] After acquiring the first frame from the multi-camera system, cylindrical projection is used to preprocess the raw images from each camera to reduce image distortion during perspective transformation. During the camera movement process, the camera adaptively adjusts the focal length of the optimal cylindrical projection based on the distance to the object being photographed. The optimal adjustment parameters are determined by the initial measurements. This single-frame image processing step is accelerated using CUDA, reducing processing time to 0.05% of the original time, effectively ensuring real-time video stitching.

[0145] The cylindrical projection process is as follows:

[0146] (1) Let the pixel coordinates of the original image be (x, y), where x∈[0, W], y∈[0, H], W and H are the image width and height respectively. The normalized coordinates with the image center as the origin are:

[0147]

[0148] (2) Set the camera focal length to f, and the projection coordinates of each pixel (X c ,Y c ,Z c ) is given by the following formula:

[0149] Z c =f

[0150] Among them, X c The input image x′ is projected onto the cylinder through the arc tangent function, Y c Through triangular relationship calculation, the vertical proportion is kept consistent.

[0151] (3) The new pixel position after projection is given by the following formula:

[0152]

[0153] (4) Finally, bilinear interpolation is used to calculate the color value of the target pixel. The mapping relationship between the projected image and the original image is as follows: I′(x″,y″)=I(x,y).

[0154] During the image registration phase, a threshold-based SURF algorithm is used to extract and match features within overlapping image regions. Because the overlap between the images captured by the eight adjacent industrial cameras exceeds 30% of a single captured image, there is a certain degree of error in the physical placement of the cameras. Therefore, a detection threshold of 30% is set in the algorithm, meaning that the detection area covers 30% of a single image. For the left-view image, only the right 30% of the overlapping region is detected; for the right-view image, only the left 30% of the overlapping region is detected. If the number of feature points in the overlapping region of a video frame captured at a certain moment during the movement is relatively small, the threshold is relaxed to include the entire overlapping region. By controlling the detection range of the overlapping region between adjacent images, the relative error in the overlapping region caused by the physical placement of adjacent cameras can be improved, reducing the detected overlap range and improving the algorithm's execution efficiency. Furthermore, the detection and description of a large number of redundant feature points can be avoided, reducing interference caused by mismatched points in non-joined areas.

[0155] Before image fusion, when multiple industrial cameras are moving and capturing images, weather conditions and 360-degree horizontal visibility vary. Even if data is captured at the same moment, exposure differences can occur between the video data due to factors such as the camera's viewing angle or autofocus. Directly stitching such data will result in noticeable differences in brightness and color in the stitched image, and noticeable seam lines will appear, affecting the quality of the video stitching. Therefore, after image registration, illumination compensation should be performed before image fusion to obtain a panoramic stitched image with a natural transition.

[0156] In this embodiment, a block-based image compensation method is used to divide the image into blocks and then calculate the compensation coefficient for each image block. By establishing a light intensity error function model, the illumination compensation gain matrix is ​​solved, and finally the gain coefficients in the matrix are used to perform illumination compensation on the corresponding areas in the source image. In the case where the camera motion causes drastic global changes in the background, motion detection based on optical flow can be introduced to further stabilize the video frame. After compensation or alignment, the optical flow field is used to analyze the motion information of each pixel. By comparing the difference between the local motion vector and the overall (global) motion vector, areas inconsistent with the background motion can be identified, and then the real moving objects can be detected.

[0157] In the image fusion stage, the background subtraction method is first used to detect moving objects in the video frames based on the selected key frames. That is, when a moving object passes through the optimal stitching line, the optimal stitching line is dynamically updated. In the solution of the energy function, motion estimation methods such as optical flow are used to insert horizontal and vertical displacement parameters for displacement compensation. Finally, the improved multi-band fusion is used to decompose and reconstruct the image to obtain a panoramic stitching image.

[0158] Based on the searched optimal stitching line, the steps of fusing the spliced ​​video frame images using the improved multi-band fusion algorithm are as follows:

[0159] (1) In order to avoid moving objects as much as possible in the overlapping area, a dynamic programming method based on energy function is used to find the optimal stitching line. The energy function is defined as:

[0160] E(x,y)=|I1(x,y)-I2(x+Δ x (x,y),y+Δ y (x,y))|

[0161] Among them, I1(x,y) and I2(x,y) represent the grayscale or color value of the stitching image and the image to be stitched respectively. The smaller the energy, the closer the two images are at that position and the more suitable they are for stitching. Here, in order to compensate for the image displacement caused by camera motion, Δ x (x,y) and Δ y (x,y) are the horizontal and vertical displacements obtained at point (x,y) by motion estimation methods such as optical flow.

[0162] (2) To ensure the minimum energy accumulation from top to bottom, we need to find an optimal path extending from the upper boundary to the lower boundary of the overlapping area. Using the updated energy function, we construct the cumulative energy function C(x,y), and its recursive relationship is:

[0163] C(x,y)=E(x,y)+min{C(x-1,y-1),C(x,y-1),C(x+1,y-1)}

[0164] Among them, the boundary conditions need to be properly initialized according to the boundaries of the overlapping area.

[0165] (3) Starting from the bottom of the overlapping area, select the pixel with the smallest energy value, and trace back along the minimum path saved during dynamic programming recursion to obtain the final optimal stitching line.

[0166] (4) Perform multi-band fusion. At each pyramid level, in order to better cope with camera motion, the Laplacian coefficients of the two images are updated by weighted fusion:

[0167]

[0168] in, is the original smooth transition weight, which plays a role in smoothing the transition and making the fusion boundary more natural. Φ(·) is a function that adjusts according to the motion amplitude. When the motion amplitude is large, the smoothing effect of the weight is reduced to expand the transition area and make the image fusion more natural.

[0169] (5) From the fused Laplace pyramid Reconstruct the fused image, the recursive formula is:

[0170]

[0171] During the image fusion process, background subtraction based on the Vibe algorithm is first performed. When a significant background change is detected, the Vibe algorithm discards the original background model and recreates a new background model using the first frame after the change. The Vibe algorithm's foreground object detection process compares all pixels in each frame with those in the background model, calculates the Euclidean distance between pixels, and then uses a set threshold to determine whether the current pixel belongs to the foreground object or background. After the classification is completed, the foreground point count is used to obtain information about moving targets, and CUDA is used for accelerated processing.

[0172] The Vibe algorithm updates the background model in three ways. First, if a foreground pixel is identified as a foreground point N times consecutively, it is updated to a background pixel. This addresses the issue of moving objects remaining in the foreground for extended periods of time. Second, the background model for each pixel has a probability of 1 / φ being updated. Third, when each pixel is updated, there is a probability of 1 / φ that the update will be propagated to neighboring pixels.

[0173] The background subtraction method based on the Vibe algorithm is aimed at dynamic backgrounds in moving situations. On the premise of performing SURF feature point detection and perspective transformation matrix update on video key frames to achieve background alignment, the Vibe algorithm is introduced to update the background model, which can better solve the problem of dynamic backgrounds.

[0174] After obtaining a binary image of the foreground area using the Vibe algorithm's background subtraction method, we can determine whether the current frame can continue to use the optimal stitching line calculated in the previous frame. If any pixels on the optimal stitching line calculated in the previous frame fall within the foreground area, the optimal stitching line must be recalculated for the current frame, meaning it is updated. Otherwise, the current frame can continue to use the optimal stitching line calculated in the previous frame.

[0175] The Vibe algorithm only needs to construct and initialize a background model in the first frame of the video. The next frame in the fusion process can then be used for moving object detection. Building the background model is the first step. The second step, based on this established background model, is to determine foreground targets. This involves comparing all pixels in each frame of the image fusion process with those in the background model. A threshold is used to determine whether the current pixel belongs to the foreground or background. Background subtraction based on the Vibe algorithm primarily focuses on foreground target detection.

[0176] The Vibe algorithm's foreground object detection requires processing every frame during the image fusion process. However, SURF feature detection and homography transformation during registration only require processing of key frames, as this involves updating the optimal seam line.

[0177] In actual stitching, the core algorithm of video stitching technology is still the image stitching algorithm. Generally, the better algorithm in image stitching is complex and time-consuming. It is obviously unrealistic to apply the image stitching algorithm to every frame of the panoramic video, and it cannot meet the real-time requirements in practical applications. This embodiment is based on the technical solution of static image stitching and proposes a real-time video stitching method, which provides technical support for establishing an intelligent carrier panoramic video stitching system.

[0178] This embodiment proposes a video stitching method for dynamic scenes in mobile environments. It encompasses four components: image acquisition, image registration, image illumination gain compensation, and image fusion. Through algorithmic processing of video keyframes and a physically mounted servo pan / tilt system, it can be used in unmanned real-time panoramic video surveillance systems. To ensure real-time video stitching, CUDA programming is used to optimize the stitching algorithm for GPUs, significantly improving program execution speed.

[0179] Example 3:

[0180] Another embodiment of the present application relates to a device for video splicing in a dynamic scene. The implementation details of the device for video splicing in a dynamic scene of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this solution. The schematic diagram of the device for video splicing in a dynamic scene of this embodiment can be as follows: Figure 8 As shown, it includes a to-be-joined image extraction module 810 , an initialization module 820 , an update module 830 and a fusion module 840 .

[0181] The image extraction module 810 is configured to extract a plurality of images to be stitched from each frame of a video stream from a multi-camera system, wherein two adjacent cameras in the multi-camera system have overlapping shooting areas, and the images to be stitched from the two adjacent cameras have overlapping areas.

[0182] An initialization module 820 is configured to determine a first-frame registration image among all images to be stitched, use the transformation matrix used by the first-frame registration image during the registration process as the initial transformation matrix, use a stitching line calculated based on the overlapping area of ​​the first-frame registration image as the initial stitching line, and establish an initial background model based on the first-frame registration image, wherein the transformation matrix is ​​calculated based on feature points in the overlapping area of ​​adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on the Vibe algorithm, which is used to determine key frame images with moving targets in subsequent frames of images to be stitched;

[0183] An updating module 830 is configured to, for a subsequent frame image to be stitched, extract feature information of adjacent images to be stitched of the current frame to update the transformation matrix when the background model detects that the current video frame image is the key frame image; register the current frame image to be stitched based on the updated transformation matrix; re-determine a new stitching line within the overlapping area of ​​the registered aligned images of the current frame; and update the background model based on the registered aligned images of the current frame to determine whether the next frame image to be stitched is a key frame image; otherwise, maintain the transformation matrix, background model, and stitching line of the previous frame unchanged, and process the current frame image to be stitched according to the transformation matrix and stitching line of the previous frame;

[0184] The fusion module 840 is configured to fuse the images to be spliced ​​in real time based on the stitching lines of each frame of the images to be spliced ​​to obtain a spliced ​​video.

[0185] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0186] In some optional embodiments, the module for extracting images to be stitched includes:

[0187] An original image extraction unit is used to extract a set of original image sequences based on the video stream of each camera in the multi-camera;

[0188] The preprocessing unit is used to preprocess each group of original image sequences by using a cylindrical projection method, wherein the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each group of original image sequences and the photographed object.

[0189] In some optional embodiments, the update module includes:

[0190] an illumination compensation unit, configured to perform illumination compensation on the registered image of the current frame to obtain an illumination compensated image;

[0191] The stitching line determining unit is configured to determine a stitching line in an overlapping area of ​​the illumination-compensated images.

[0192] In some optional embodiments, the update module further includes:

[0193] a displacement compensation unit, configured to apply an optical flow detection algorithm to insert horizontal and vertical displacement parameters into the aligned image of the current frame after registration to perform displacement compensation, thereby obtaining a displacement-compensated image;

[0194] The stitching line determining unit is further configured to determine the stitching line in the overlapping area of ​​the displacement-compensated images.

[0195] In some optional embodiments, the real-time fusing of the images to be stitched based on the stitching lines of each frame of the images to be stitched includes:

[0196] Multi-band fusion is performed on the images to be stitched based on the stitching lines of each frame of the images to be stitched.

[0197] In some optional embodiments, the fusion module includes:

[0198] The coefficient fusion unit is used to fuse the Laplacian coefficients of the two images in a weighted manner at each pyramid level:

[0199]

[0200] in, and are the Laplacian pyramids of the two images respectively, is the original smooth transition weight, Φ(·) is the function adjusted according to the motion amplitude;

[0201] The fused image reconstruction unit is used to reconstruct the image from the fused Laplacian pyramid. Reconstruct the fused image, and the recursive calculation formula is as follows:

[0202]

[0203] Among them, F i is the image at the i-th layer of the fused Laplacian pyramid, F N-1 is the top image of the fused Laplacian pyramid, Expand(F i+1 ) is the process of expanding the image of the i+1th layer to the i-th layer.

[0204] In some optional embodiments, the method further includes:

[0205] The acceleration module is used to use CUDA to perform acceleration processing during the execution of the method.

[0206] Example 4:

[0207] Another embodiment of the present application relates to an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for video stitching in dynamic scenes in the above-mentioned embodiments.

[0208] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0209] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0210] Embodiment 5:

[0211] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0212] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0213] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for video splicing in a dynamic scene, characterized in that: include: Extracting a plurality of images to be stitched in each frame based on a video stream from a multi-camera, wherein two adjacent cameras in the multi-camera have overlapping shooting areas, and the images to be stitched from the corresponding two adjacent cameras have overlapping areas; Determining a first-frame registration image among all images to be stitched, using a transformation matrix used in a registration process of the first-frame registration image as an initial transformation matrix, using a stitching line calculated based on an overlapping area of ​​the first-frame registration image as an initial stitching line, and establishing an initial background model based on the first-frame registration image, wherein the transformation matrix is ​​calculated based on feature points of an overlapping area of ​​adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on a Vibe algorithm, which is used to determine a key frame image having a moving target in a subsequent frame of images to be stitched; For subsequent frames of images to be stitched, when the background model detects that the current video frame image is the key frame image, feature information of adjacent images to be stitched of the current frame is extracted to update the transformation matrix, the images to be stitched of the current frame are registered based on the updated transformation matrix, a new stitching line is re-determined within the overlapping area of ​​the registered aligned images of the current frame, and the background model is updated based on the registered aligned images of the current frame to determine whether the next frame of image to be stitched is the key frame image; otherwise, the transformation matrix, the background model, and the stitching line of the previous frame are maintained unchanged, and the images to be stitched of the current frame are processed according to the transformation matrix and the stitching line of the previous frame; The images to be stitched are fused in real time based on the stitching lines of each frame to obtain a stitched video.

2. The method for video splicing in dynamic scenes according to claim 1, characterized in that: The method extracts multiple images to be stitched in each frame based on the video stream from the multi-camera, including: Based on the video stream of each camera in the multi-camera, a set of original image sequences is correspondingly extracted; Each set of original image sequences is preprocessed using a cylindrical projection method, wherein the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each set of original image sequences and the photographed object.

3. The method for video splicing in dynamic scenes according to claim 1, characterized in that: The re-determining of a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes: Performing illumination compensation on the registered image of the current frame to obtain an illumination compensated image; A stitching line is determined within the overlapping region of the illumination-compensated images.

4. The method for video splicing in a dynamic scene according to claim 1, characterized in that: The re-determining of a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes: Applying the optical flow detection algorithm to insert the horizontal and vertical displacement parameters into the aligned image of the current frame after registration to perform displacement compensation and obtain a displacement compensated image; The stitching line is determined in an overlapping region of the displacement-compensated images.

5. The method for video splicing in dynamic scenes according to claim 4, characterized in that: The process of determining the stitching line includes: defining an energy function based on the grayscale difference between the two displacement-compensated images to be stitched; In the overlapping region, the stitching line is obtained by dynamic programming by minimizing the cumulative energy.

6. The method for video splicing in dynamic scenes according to claim 5, characterized in that: The calculation formula of the energy function is as follows: in, and Respectively represent the grayscale values ​​of the two displacement-compensated images, and At points The horizontal and vertical displacements are obtained by the optical flow detection algorithm; The calculation formula of the cumulative energy is as follows: ； in, is the energy at the current pixel, are the accumulated energies at the pixels to the upper left, directly above, and upper right of the current pixel, respectively.

7. The method for video splicing in a dynamic scene according to any one of claims 1 to 6, characterized in that: The real-time fusion of the images to be stitched based on the stitching lines of each frame of the images to be stitched includes: Multi-band fusion is performed on the images to be stitched based on the stitching lines of each frame of the images to be stitched.

8. The method for video splicing in dynamic scenes according to claim 7, characterized in that: The multi-band fusion of the images to be stitched based on the stitching line of each frame of the images to be stitched includes: At each pyramid level, the Laplacian coefficients of the two images are fused in a weighted manner: in, and are the Laplacian pyramids of the two images respectively, is the original smooth transition weight, is a function that adjusts according to the amplitude of the motion; From the fused Laplacian pyramid Reconstruct the fused image, and the recursive calculation formula is as follows: ， in, is the image at the i-th layer of the fused Laplacian pyramid, is the top image of the fused Laplacian pyramid, It is the process of expanding the image of the i+1th layer to the i-th layer.

9. The method for video splicing in a dynamic scene according to any one of claims 1 to 6, characterized in that: The method adopts CUDA to perform accelerated processing during execution.

10. A device for video splicing in dynamic scenes, characterized in that: include: A module for extracting images to be stitched is configured to extract a plurality of images to be stitched in each frame based on a video stream from a multi-camera, wherein two adjacent cameras in the multi-camera have overlapping shooting areas, and the images to be stitched from the corresponding two adjacent cameras have overlapping areas; An initialization module is configured to determine a first-frame registration image among all images to be stitched, use a transformation matrix used by the first-frame registration image in the registration process as an initial transformation matrix, use a stitching line calculated based on an overlapping area of ​​the first-frame registration image as an initial stitching line, and establish an initial background model based on the first-frame registration image, wherein the transformation matrix is ​​calculated based on feature points of an overlapping area of ​​adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on a Vibe algorithm, which is configured to determine key frame images having a moving target in subsequent frames of images to be stitched; an updating module configured to, for a subsequent frame image to be stitched, extract feature information of adjacent images to be stitched of the current frame to update the transformation matrix when the background model detects that the current video frame image is the key frame image; register the current frame image to be stitched based on the updated transformation matrix; re-determine a new stitching line within an overlapping area of ​​the registered aligned images of the current frame; and update the background model based on the registered aligned images of the current frame to determine whether the next frame image to be stitched is a key frame image; otherwise, maintain the transformation matrix, the background model, and the stitching line of the previous frame unchanged, and process the current frame image to be stitched according to the transformation matrix and the stitching line of the previous frame; The fusion module is used to fuse the images to be stitched in real time based on the stitching lines of each frame of the image to be stitched to obtain a stitched video.

11. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for video stitching in a dynamic scene as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for video stitching in a dynamic scene according to any one of claims 1 to 9 is implemented.

Images