Video stitching method and device in dynamic scene, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the field of video processing, and discloses a video stitching method and device in a dynamic scene, electronic equipment and a storage medium, and the method comprises the steps: determining a first frame of registration image in each frame of to-be-stitched image extracted from a video stream from a multi-view camera, taking a transformation matrix used in the registration process of the first frame of registration image as an initial transformation matrix, taking a suture line of the transformation matrix as an initial suture line, establishing an initial background model based on the first frame of registration image, and for subsequent frames of to-be-spliced images, when the background model detects that a current video frame image contains a moving object, carrying out registration on the subsequent frames of to-be-spliced images; and updating the transformation matrix based on the current frame of the to-be-spliced image so as to carry out registration again to determine a new suture line and update the background model, otherwise, maintaining the transformation matrix, the background model and the suture line of the previous frame unchanged, and carrying out real-time fusion based on the suture line of each frame of the to-be-spliced image to obtain a spliced video. And the requirements of stability and real-time performance of panoramic video stitching of the mobile unmanned carrier are met.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video processing, and particularly to a method, device, electronic device and storage medium for video stitching in a dynamic scene. Background Art

[0002] As an extension of the wider field of view of a single camera image, a real-time panoramic video monitoring system can be applied to intelligent transportation, intelligent venues, intelligent industries and other intelligent Internet of Things fields. The biggest difference between video stitching and image stitching is that there are moving objects in the video. Especially when the camera is also in motion, this will greatly increase the complexity of video stitching. In a mobile environment, the camera and the objects being photographed are in relative motion, and the scale, illumination, etc. of the images collected by the camera are constantly changing.

[0003] Most of the current image stitching algorithms focus on the fixed video monitoring field and are difficult to meet the requirements of the stability and real-time performance of panoramic video stitching of unmanned carriers in motion. Summary of the Invention

[0004] The purpose of the present invention is to provide at least a method, device, electronic device and storage medium for video stitching in a dynamic scene, which can at least solve the problem that the current image stitching algorithms are difficult to meet the requirements of real-time video stitching in a dynamic scene, and can at least achieve the effect of stably and real-time stitching videos in a dynamic scene.

[0005] To solve the above technical problems, at least one embodiment of the present application provides a method for video stitching in a dynamic scene, including:

[0006] Based on the video streams from a multi-camera, multiple images to be stitched in each frame are extracted, wherein there is an overlapping shooting area between two adjacent cameras in the multi-camera, and the images to be stitched corresponding to the adjacent two cameras have overlapping areas;

[0007] Determine the first frame registration image among all the images to be stitched, use the transformation matrix used by the first frame registration image during the registration process as the initial transformation matrix, use the stitching line calculated based on the overlapping area of the first frame registration image as the initial stitching line, and establish an initial background model based on the first frame registration image, wherein the transformation matrix is calculated based on the feature points of the overlapping areas of the adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on the Vibe algorithm for determining key frame images with moving targets in the subsequent frame images to be stitched;

[0008] For the subsequent frames of the images to be stitched, when the background model detects that the current video frame image is the key frame image, extract the feature information of the adjacent images to be stitched in the current frame to update the transformation matrix, register the image to be stitched in the current frame based on the updated transformation matrix, re-determine a new stitching line within the overlapping region of the registered aligned image in the current frame, and update the background model based on the registered aligned image in the current frame to determine whether the next frame of the image to be stitched is the key frame image. Otherwise, keep the transformation matrix, the background model, and the stitching line of the previous frame unchanged, and process the image to be stitched in the current frame according to the transformation matrix and the stitching line of the previous frame;

[0009] Perform real-time fusion on the images to be stitched based on the stitching lines of each frame of the images to be stitched to obtain the stitched video.

[0010] At least one embodiment of the present application further provides a device for video stitching in a dynamic scenario, including:

[0011] An image extraction module to be stitched, configured to extract multiple images to be stitched in each frame based on the video stream from a multi-camera, where there is an overlapping shooting area between two adjacent cameras in the multi-camera, and the images to be stitched corresponding to the adjacent two cameras have an overlapping area;

[0012] An initialization module, configured to determine the first registered image among all the images to be stitched, use the transformation matrix used by the first registered image during the registration process as the initial transformation matrix, use the stitching line calculated based on the overlapping area of the first registered image as the initial stitching line, and establish an initial background model based on the first registered image, where the transformation matrix is calculated based on the feature points of the overlapping area of the adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on the Vibe algorithm for determining the key frame image with a moving target in the subsequent frames of the images to be stitched;

[0013] An update module, configured to, for the subsequent frames of the images to be stitched, when the background model detects that the current video frame image is the key frame image, extract the feature information of the adjacent images to be stitched in the current frame to update the transformation matrix, register the image to be stitched in the current frame based on the updated transformation matrix, re-determine a new stitching line within the overlapping region of the registered aligned image in the current frame, and update the background model based on the registered aligned image in the current frame to determine whether the next frame of the image to be stitched is the key frame image. Otherwise, keep the transformation matrix, the background model, and the stitching line of the previous frame unchanged, and process the image to be stitched in the current frame according to the transformation matrix and the stitching line of the previous frame;

[0014] A fusion module for performing real-time fusion on the images to be stitched based on the stitching lines of each frame of the images to be stitched, and obtaining the stitched video.

[0015] At least one embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for video stitching in a dynamic scenario described above.

[0016] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, and the computer program, when executed by a processor, implements the method for video stitching in a dynamic scenario described above.

[0017] The method, apparatus, electronic device, and storage medium for video stitching in a dynamic scenario provided by the embodiments of the present application determine the first-frame registered image, use the transformation matrix used in its registration process as the initial transformation matrix, use the stitching line calculated based on its overlapping area as the initial stitching line, and establish an initial background model. This initialization method based on the first frame provides a basis for the processing of subsequent frames, enables the entire video stitching process to have a reasonable starting point, avoids calculating the transformation matrix and stitching line from scratch for each frame, and improves the processing efficiency; the background model can accurately determine the key-frame images with moving objects in the subsequent frames of the images to be stitched. In a dynamic scenario, the appearance of moving objects will cause changes in the image content, and traditional stitching methods may not be able to effectively handle it. However, the present application only performs operations such as updating the transformation matrix, re-determining the stitching line, and updating the background model on the key frames by detecting the key frames, avoiding complex calculations for each frame and improving the processing speed; when a key frame is detected, the feature information of the adjacent images to be stitched in the current frame is extracted to update the transformation matrix, and the images to be stitched in the current frame are registered based on the updated transformation matrix, and a new stitching line is re-determined. At the same time, the background model is updated based on the registered aligned images in the current frame to determine whether the next frame of the images to be stitched is a key-frame image. This dynamic update mechanism can timely adapt to the changes in the scenario and ensure the accuracy and coherence of the stitching; for non-key frames, the transformation matrix, background model, and stitching line of the previous frame are maintained unchanged, and the images to be stitched in the current frame are processed according to the transformation matrix and stitching line of the previous frame. This method avoids frequent parameter updates due to minor changes or noise in the scenario, ensures the stability of the stitching process, reduces stitching errors that may be caused by frequent parameter changes, and greatly reduces the computational complexity. Therefore, the present application effectively improves the efficiency, accuracy, and stability of real-time video stitching in a dynamic scenario through the key-frame processing mechanism that adapts to the dynamic scenario, and can provide high-quality stitched videos.

[0018] In some alternative embodiments, extracting multiple images to be stitched in each frame based on the video stream from a multi-camera includes:

[0019] Correspondingly extracting a set of original image sequences based on the video stream of each camera in the multi-camera;

[0020] Preprocessing each set of original image sequences by using a cylindrical projection method, wherein the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each set of original image sequences and the object being photographed.

[0021] In this embodiment, through cylindrical projection, the planar images captured by different cameras can be converted into cylindrical images, making the stitched image more in line with the visual habits of the human eye and reducing the sense of distortion after stitching. The parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each set of original image sequences and the object being photographed. Different shooting distances will result in different perspectives and degrees of distortion of the images. By dynamically adjusting the parameters of the cylindrical projection, these changes can be better adapted, and the quality of the projected image can be further improved.

[0022] In some alternative embodiments, re-determining a new stitching line within the overlapping region of the currently aligned images in the current frame after registration includes:

[0023] Performing illumination compensation on the currently aligned images in the current frame after registration to obtain an illumination-compensated image;

[0024] Determining the stitching line within the overlapping region of the illumination-compensated image.

[0025] In this embodiment, by performing illumination compensation on the currently aligned images in the current frame after registration to obtain an illumination-compensated image, the illumination conditions in the overlapping region can be made to tend to be consistent. Determining the stitching line within the overlapping region of the illumination-compensated image and performing stitching can effectively eliminate the stitching marks caused by illumination differences, making the transition of the stitched image more natural and smooth.

[0026] In some alternative embodiments, re-determining a new stitching line within the overlapping region of the currently aligned images in the current frame after registration includes:

[0027] Applying an optical flow detection algorithm to insert horizontal and vertical displacement parameters in the currently aligned images in the current frame after registration for displacement compensation to obtain a displacement-compensated image;

[0028] Determining the stitching line within the overlapping region of the displacement-compensated image.

[0029] In this embodiment, when there is a significant global change in the background during the movement of the camera, the optical flow detection is further used to stabilize the video frame image, eliminate the displacement difference between adjacent images to be stitched, thereby reducing the stitching error caused by displacement and improving the stitching accuracy.

[0030] In some alternative embodiments, the process of determining the stitching line includes: defining an energy function based on the gray difference between two displacement compensation images to be stitched;

[0031] In the overlapping area, the stitching line is obtained by dynamic programming with the minimum cumulative energy.

[0032] In this embodiment, by searching for the stitching line with the minimum energy in the overlapping area through dynamic programming, the ghosting and blurring phenomena when a moving object passes through the stitching line can be effectively improved.

[0033] In some alternative embodiments, the calculation formula of the energy function is as follows:

[0034] E(x,y) = |I1(x,y) - I2(x + Δ x (x,y), y + Δ y (x,y))|

[0035] where I1(x,y) and I2(x,y) respectively represent the gray values of the two displacement compensation images, and Δ x (x,y) and Δ y (x,y) are the horizontal and vertical displacements obtained by the optical flow detection algorithm at the point (x,y) respectively;

[0036] The calculation formula of the cumulative energy is as follows:

[0037] C(x,y) = E(x,y) + min{C(x - 1,y - 1), C(x,y - 1), C(x + 1,y - 1)};

[0038] where E(x,y) is the energy at the current pixel point, and C(x - 1,y - 1), C(x,y - 1), and C(x + 1,y - 1) are the cumulative energies at the pixel points in the upper left, directly above, and upper right of the current pixel point respectively.

[0039] In some alternative embodiments, the real-time fusion of the images to be stitched based on the stitching line of each frame of the images to be stitched includes:

[0040] Performing multi-band fusion on the images to be stitched based on the stitching line of each frame of the images to be stitched.

[0041] In this embodiment, traditional stitching methods may produce obvious stitching marks at the stitching lines, such as brightness mutations, color discontinuities, etc. Multi-band fusion can adopt different fusion strategies on different frequency sub-bands according to the position of the stitching line, making the transition between the stitched area and the surrounding images more natural.

[0042] In some alternative embodiments, performing multi-band fusion on the images to be stitched based on the stitching line of each frame of image includes:

[0043] At each pyramid level, fuse the Laplacian coefficients of the two images by weighting:

[0044]

[0045] where and are the Laplacian pyramids of the two images respectively, is the original smooth transition weight, and Φ(·) is a function adjusted according to the motion amplitude;

[0046] From the fused Laplacian pyramid reconstruct the fused image, and the recurrence calculation formula is as follows:

[0047]

[0048] where F i is the image of the fused Laplacian pyramid at the i-th layer, F N-1 is the top-layer image of the fused Laplacian pyramid, and Expand(F i+1 ) is the process of expanding the image of the (i + 1)-th layer to the i-th layer.

[0049] In this embodiment, is the original smooth transition weight, which plays a role in smooth transition, making the fusion boundary more natural. Φ(·) is a function adjusted according to the motion amplitude. When the motion amplitude is large, the smooth effect of the weight is reduced to expand the transition area and make the image fusion more natural.

[0050] In some alternative embodiments, CUDA is used for acceleration processing during the execution of the method.

[0051] In this embodiment, a large amount of image data processing is involved in video real-time stitching, such as feature extraction, image registration, pixel fusion, etc. By parallelizing these operations through CUDA, the processing speed of video stitching can be significantly improved to meet the real-time requirements in mobile scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] One or more embodiments are illustrated by way of example in the accompanying drawings, and such illustrations do not constitute a limitation on the embodiments.

[0053] Figure 1 is the flow of the method for video stitching in a dynamic scenario provided by an embodiment of the present application Figure 1 ;

[0054] Figure 2 is the flowchart of the existing feature point extraction algorithm;

[0055] Figure 3 is the schematic diagram of the overlapping area of adjacent images to be stitched provided by an embodiment of the present application;

[0056] Figure 4 is the flowchart of the existing Vibe algorithm for detecting foreground points;

[0057] Figure 5 is the flowchart of the improved Vibe algorithm introducing SURF feature point matching provided by an embodiment of the present application;

[0058] Figure 6 is the flow of the method for video stitching in a dynamic scenario provided by an embodiment of the present application Figure 2 ;

[0059] Figure 7 is the schematic diagram of the structure of a multi-camera provided by an embodiment of the present application;

[0060] Figure 8 is the schematic diagram of the structure of the device for video stitching in a dynamic scenario provided by another embodiment of the present application. Detailed implementation manners

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that in the embodiments of the present application, many technical details are provided to help readers better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions required to be protected by the present application can still be implemented. The following division of each embodiment is for convenience of description and should not constitute any limitation on the specific implementation manner of the present application. Each embodiment can be combined and cross-referenced with each other on the premise of not being contradictory.

[0062] To facilitate the understanding of the embodiments of the present application, the relevant content of the existing video stitching is introduced here first.

[0063] Prior Art 1: A panoramic stitching method for a four-eye camera, which collects image information in the same space through the four-eye camera and imports the image information into a CMOS image sensor; after converting the optical signal of the image information into a digital image signal by the CMOS image sensor, it is input into a DSP digital signal processing chip for preprocessing of the image information; the SIFT algorithm is used to extract key points from the digital image information in the DSP digital signal processing chip collected at different angles by the four-eye camera, and based on the key point extraction, the panoramic image information in the four-eye camera is matched to obtain the first stitched panoramic image output by the matching; by calculating the pixel difference between the first stitched panoramic image and the second stitched panoramic image of the adjacent frame, the dynamic target is identified; the background of the panoramic image is constructed by the background modeling method, and the dynamic target is fused into the constructed panoramic image background as the foreground to stitch the dynamic image.

[0064] In the above method, first, the image information in the same space is collected through the four-eye camera, which is limited to only realizing the stitching of dynamic images in a fixed situation, and there is no mention of how to realize real-time panoramic video stitching in a mobile environment; second, the SIFT algorithm is used to extract key points of the image information. This algorithm is computationally complex and time-consuming, and is not suitable for panoramic video stitching in a mobile situation.

[0065] Prior Art 2: A panoramic stitching method and system based on a pan-tilt device: The method includes: obtaining an image containing a target, and determining a starting point and an ending point of panoramic stitching within the rectangular bounding box of the image containing the target; obtaining a capture movement trajectory according to the starting point, ending point of the panoramic stitching and the total number of movements of the rectangular field of view frame of the imaging device in the horizontal and vertical directions; controlling the movement of the pan-tilt so that the rectangular field of view frame of the imaging device starts from the movement starting position and moves along the capture movement trajectory until it reaches the movement ending position, and when the rectangular field of view frame is at the movement starting position, each time it moves, and at the movement ending position, the corresponding image is obtained through the imaging device connected to the pan-tilt; stitching all the images obtained by the imaging device to obtain a panoramic stitched image.

[0066] In this method, it is mentioned to obtain an image containing the target, and determine the starting point and ending point of panoramic stitching within the rectangular bounding box of the image containing the target. In the case of movement, the area for obtaining the image is determined by the target, and the field of view is limited. Second, when the rectangular field-of-view frame is at the starting position of movement, each time it moves, and at the ending position of movement, images corresponding to the imaging devices connected to the pan-tilt head are obtained, and the images obtained by all imaging devices are stitched together to obtain a panoramic stitched image. An image is obtained each time it moves, which is only limited to the stitching of panoramic images, does not have the real-time performance of video stitching, and cannot perform real-time panoramic monitoring on emergencies in the case of movement. Third, images are captured regularly according to the automatically planned image capture path and stitched together, and finally the complete features of a larger target are obtained. Compared with identifying and marking the target of interest in a real-time panoramic video, this method does not have an advantage.

[0067] In view of this, the technical problem to be solved by this application is: in a mobile environment, the camera and the object being photographed are in relative motion, and the scale, illumination, etc. of the images collected by the camera are constantly changing. If a 360-degree panoramic video monitoring system is combined with an unmanned vehicle, how to ensure the stability and real-time performance of video frame stitching.

[0068] To solve the above technical problem, the present invention proposes a method for video stitching in a dynamic scene. The implementation details of the method for video stitching in a dynamic scene in this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing this solution.

[0069] Embodiment 1:

[0070] The method for video stitching in a dynamic scene in this embodiment can be applied to an electronic device with communication, computing, and data storage capabilities. Its specific process can be as Figure 1 shown, including:

[0071] Step 110, based on the video stream from a multi-camera, extract multiple images to be stitched in each frame. Among them, there is an overlapping shooting area between two adjacent cameras in the multi-camera, and the images to be stitched corresponding to the two adjacent cameras have an overlapping area;

[0072] In this embodiment, the multi-camera can be a camera array arranged in a ring or linearly, and there is an overlapping area in the field of view angles of adjacent cameras. In a dynamic scene, the overlapping area provides the correlation information between adjacent images, which is the key to realizing image stitching. Image data is extracted frame by frame from the video stream. For each frame, multiple images to be stitched are obtained according to the number and layout of the cameras. For example, if there are three cameras, then three images to be stitched will be obtained for each frame. These images to be stitched have overlapping areas between adjacent cameras, and these overlapping areas provide the basis for subsequent image registration and stitching.

[0073] Step 120, determine the first-frame registered image among all the images to be stitched, use the transformation matrix used by the first-frame registered image in the registration process as the initial transformation matrix, use the stitching line calculated based on the overlapping area of the first-frame registered image as the initial stitching line, and establish an initial background model based on the first-frame registered image, where the transformation matrix is calculated based on the feature points of the overlapping areas of adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on the Vibe algorithm for determining the key-frame images with moving objects in the subsequent frames of images to be stitched;

[0074] In this embodiment, to determine the first-frame registered image among all the images to be stitched, it can be to select the image to be stitched in the initial frame of the video stream for registration as the first-frame registered image. It can also be to select the image to be stitched in a subsequent frame for registration as the first-frame registered image. Generally, a static scene (without significant moving objects) is selected as the initial registration reference to improve the accuracy of the transformation matrix, stitching line, and background model. When there is no motion interference in the initial frame of the video stream, the image to be stitched in the initial frame of the video stream can be selected for registration. If there are moving objects in the initial frame, the first static frame can be searched for in the subsequent frames as the first-frame registered image.

[0075] In the registration process of the first-frame registered image, first, based on the feature points of the overlapping areas of adjacent images to be stitched from adjacent cameras, calculate the transformation matrix. Commonly used feature point extraction algorithms include SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), etc. By feature point matching, find the corresponding feature point pairs in the overlapping areas of adjacent images, and then use these feature point pairs to calculate the transformation matrix. This transformation matrix describes the geometric transformation relationship between adjacent images, such as translation, rotation, scaling, etc. In the overlapping area of the first-frame registered image, calculate the stitching line according to information such as the color and gradient of the image. The purpose of the stitching line is to find an optimal path so that the stitched image has a natural transition at the stitching line and reduces the visual discontinuity.

[0076] During the execution of traditional feature point extraction algorithms, feature points are extracted and described for all regions of the images to be stitched, and its process is asFigure 2 As shown. In this embodiment, in view of the feature changes of the surrounding scenes captured in the mobile environment, in combination with the proportional relationship of the overlapping regions of adjacent images in the panoramic camera framework, the feature point extraction algorithm is improved by controlling the feature point detection threshold S.

[0077] Specifically, the overlapping regions of adjacent images to be stitched can be as Figure 3 shown. The overlapping range of the images captured by adjacent cameras all accounts for more than 30% of a single captured image, and there are certain errors in the physical installation positions of the cameras. Therefore, the detection threshold is set to 30% in the feature point extraction algorithm (referring to that the detection region accounts for 30% of a single frame). For the left-viewpoint image, only the right 30% of the overlapping region is detected; for the right-viewpoint image, only the left 30% of the overlapping region is detected; if there are fewer feature points in the overlapping region of the video frame captured at a certain moment during movement, the threshold is relaxed to the entire overlapping region. By controlling the detection range of the overlapping regions between adjacent images, on the one hand, the relative error of the overlapping regions caused by the physical placement method between adjacent cameras can be improved, the detected overlapping range can be reduced, and the execution efficiency of the algorithm can be enhanced; on the other hand, the detection and description of a large number of redundant feature points can be avoided, and the interference caused by mis-matched points in non-stitched regions can be reduced.

[0078] An initial background model is established based on the first-frame registered image, where the background model includes a background subtraction model based on the Vibe (Visual Background Extractor) algorithm. The Vibe algorithm analyzes the image sequence to establish the background model and can effectively detect foreground moving objects. In the initial stage, the parameters of the Vibe model, such as the sample set, matching threshold, etc., are initialized using the first-frame registered image. The background model is used to determine the key-frame images with moving objects in subsequent frames, so as to dynamically update the transformation matrix and the stitching line.

[0079] Step 130, for the subsequent frames of images to be stitched, when the background model detects that the current video frame image is the key-frame image, extract the feature information of the adjacent images to be stitched in the current frame to update the transformation matrix, register the image to be stitched in the current frame based on the updated transformation matrix, re-determine a new stitching line in the overlapping region of the registered and aligned image in the current frame, and update the background model based on the registered and aligned image in the current frame to determine whether the next frame of image to be stitched is a key-frame image; otherwise, keep the transformation matrix, the background model, and the stitching line of the previous frame unchanged, and process the image to be stitched in the current frame according to the transformation matrix and the stitching line of the previous frame;

[0080] In this embodiment, for the subsequent frame of the image to be stitched, it is detected using the background model. The background model will determine whether the current frame is a key-frame image based on the difference between the current frame image and the background model. If there are moving objects in the current frame image and the difference from the background model is large, then the current frame is determined to be a key-frame image. For example, in a surveillance scenario, if a pedestrian or a vehicle enters the frame, the background model will detect these moving objects and thus determine that the current frame is a key frame.

[0081] When it is detected that the current frame image is a key-frame image, the following processing is performed:

[0082] (1) Extract the feature information of the adjacent images to be stitched in the current frame. The feature point extraction algorithms such as SIFT and SURF can also be used to obtain the feature points in the overlapping area of adjacent images.

[0083] (2) Update the transformation matrix based on the extracted feature information. Since the moving objects in the scene may cause changes in the geometric relationship between images, it is necessary to recalculate the transformation matrix to ensure the accuracy of image registration.

[0084] (3) Register the image to be stitched in the current frame according to the updated transformation matrix, and align the adjacent images in the same coordinate system.

[0085] (4) Redetermine the new stitching line in the overlapping area of the aligned images in the current frame after registration. Since the image content has changed, the original stitching line may no longer be applicable, so it is necessary to recalculate the stitching line to ensure the quality of the stitched image.

[0086] (5) Update the background model based on the aligned images in the current frame after registration. Incorporate the information of the current frame into the background model to more accurately detect whether the next frame is a key-frame image.

[0087] Specifically, first, through the background subtraction method based on the Vibe algorithm, when it is detected that there are obvious changes in the background, the Vibe algorithm will choose to discard the original background model and use the first frame image (i.e., the key-frame image) after the change to re-establish a new background model. The foreground object detection process of the Vibe algorithm is to compare each pixel point in each frame image with the pixel points in its background model in turn, calculate the Euclidean distance between pixel points, and then judge whether the current pixel point belongs to the foreground object or the background through a set threshold. After classifying the image, the foreground points can be counted to obtain the information of the moving object, and CUDA is used for acceleration processing.

[0088] As Figure 4 shown, it is the flowchart of the existing Vibe algorithm for detecting foreground points. There are three situations for the Vibe algorithm to update the background model:

[0089] 1. For foreground pixels, if a pixel is continuously determined to be a foreground pixel for N times, then update this foreground pixel to a background pixel to solve the problem of long-term retention of moving objects in the foreground;

[0090] 2. For the background model of each pixel, there is a probability of being updated;

[0091] 3. When each pixel is updated, there is a probability of propagating the update to neighboring pixels.

[0092] As Figure 5 shown, the figure is a flowchart of an improved Vibe algorithm that introduces SURF feature point matching in this embodiment. After obtaining the binary image of the foreground area through the background subtraction method of the Vibe algorithm, it can be determined whether the stitching line calculated in the previous frame of the image to be stitched can be used for the current frame of the image to be stitched. If there are pixels on the stitching line calculated in the previous frame that fall into the foreground area, then the stitching line needs to be recalculated for the current frame of the image to be stitched. Otherwise, the current frame of the image to be stitched can continue to use the stitching line calculated in the previous frame.

[0093] It can be understood that in the image fusion stage, the biggest difference between video fusion technology and image fusion technology is that there are real-time moving objects in the video. Therefore, when determining the stitching line, it is necessary to avoid moving objects to prevent phenomena such as misalignment, ghosting, and blurring in the fused image. However, when performing video fusion in the case of camera movement, if the existing method in image fusion is still used for fusion, when a moving object passes through the stitching line, the above-mentioned misalignment, ghosting, and blurring phenomena will occur again. If the stitching line is calculated for each frame of the image, real-time performance cannot be satisfied. Therefore, in this embodiment, for the selected key frame images, first, the moving objects existing in the video frame are detected through the background subtraction method, and the stitching line is dynamically updated when a moving object passes through the stitching line. When it is detected that the current frame image is a non-key frame image, that is, the background model detects that the difference between the current frame and the background model is small and there is no obvious moving target, then the transformation matrix, background model, and stitching line of the previous frame are maintained unchanged. The current frame of the image to be stitched is processed according to the transformation matrix and stitching line of the previous frame, and image registration and stitching are directly performed. Feature point detection and perspective transformation matrix calculation are performed on the selected key frames, and it is not necessary to do it for each frame, thus greatly reducing the calculation amount and improving the processing efficiency.

[0094] Step 140, perform real-time fusion on the images to be stitched based on the stitching line of each frame of the image to be stitched to obtain the stitched video.

[0095] In this embodiment, for each frame of the images to be stitched, the adjacent images are fused according to the calculated stitching line. On one side of the stitching line, the color values of one image are selected; on the other side of the stitching line, the color values of the other image are selected; near the stitching line, a gradient method can be used for transition to reduce the stitching traces.

[0096] The images to be stitched are fused frame by frame to achieve real-time video stitching. The fused image sequence is encoded in video format and the stitched video is output. During the real-time fusion process, the balance between processing speed and image quality needs to be considered. Optimized algorithms and hardware acceleration technologies can be used to improve the processing efficiency and ensure the real-time performance and stability of video stitching.

[0097] The first method that can be adopted is that for the stitching of two images, if one image is directly transformed to the coordinate system of the other image through perspective transformation, since perspective transformation will stretch or compress different parts of the image to different degrees according to different viewpoints, it will cause stretching distortion in the edge picture of the panoramic image, which is also one of the reasons for the distortion of the stitching picture; for the stitching of multiple images, selecting a reference image and transforming other images to this coordinate system through perspective transformation may lead to an increase in cumulative error. It is necessary to adjust the image stitching strategy (from the stitched image to the stitched image) and the projection model (such as using cylindrical projection).

[0098] In summary, for the method of video stitching in a dynamic scenario according to this embodiment, by determining the first-frame registered image, using the transformation matrix used in its registration process as the initial transformation matrix, using the stitching line calculated based on its overlapping area as the initial stitching line, and establishing an initial background model. This initialization method based on the first frame provides a basis for the processing of subsequent frames, enabling a reasonable starting point for the entire video stitching process, avoiding calculating the transformation matrix and stitching line from scratch for each frame, and improving the processing efficiency; through the background model, key frame images with moving objects in the subsequent frames to be stitched can be accurately determined. In a dynamic scenario, the appearance of moving objects will cause changes in the image content, and traditional stitching methods may not be able to effectively handle it. However, in this application, by detecting key frames, only operations such as updating the transformation matrix, re-determining the stitching line, and updating the background model are performed on key frames, avoiding complex calculations for each frame and improving the processing speed; when a key frame is detected, the feature information of the adjacent images to be stitched in the current frame is extracted to update the transformation matrix, the images to be stitched in the current frame are registered based on the updated transformation matrix, and a new stitching line is re-determined. At the same time, the background model is updated based on the registered aligned images in the current frame to determine whether the next frame to be stitched is a key frame image. This dynamic update mechanism can timely adapt to changes in the scenario, ensuring the accuracy and coherence of stitching; for non-key frames, the transformation matrix, background model, and stitching line of the previous frame are maintained unchanged, and the images to be stitched in the current frame are processed according to the transformation matrix and stitching line of the previous frame. This method avoids frequent parameter updates due to minor changes or noise in the scenario, ensures the stability of the stitching process, reduces stitching errors that may be caused by frequent parameter changes, and greatly reduces the computational amount. Therefore, through the key frame processing mechanism that adapts to dynamic scenarios, this application effectively improves the efficiency, accuracy, and stability of real-time video stitching in dynamic scenarios and can provide high-quality stitched videos.

[0099] In some alternative embodiments, when stitching each frame of the images to be stitched, the image-to-stitching-image method and / or the stitching-image-to-stitching-image method can be adopted. The image-to-stitching-image method means selecting a reference image from all the images to be stitched, and then transforming the remaining non-reference images into this coordinate system. When performing stitching of multiple images, since each perspective transformation is based on the local optimal match between the current image and the reference image, it may not be the global optimal.

[0100] The stitching-image-to-stitching-image method means dividing the images to be stitched into multiple stitching sub-images according to the matching degree, then respectively fusing these stitching sub-images, and finally stitching the fused stitching sub-images into a complete panoramic image. This method helps to reduce the cumulative error generated during the stitching process. For the problem of image distortion in the stitching of multiple images, the above two stitching strategies can be combined to handle it.

[0101] For the problem of eliminating stitching distortion in multiple images, in some alternative embodiments, it may also be to register all adjacent images to be stitched to obtain feature point information. After calculating the transformation matrix between all adjacent images using the feature point information, the global transformation matrix H from each input image to the reference image is obtained according to the following calculation formula:

[0102]

[0103] where H[i] represents the transformation matrix obtained from each group of adjacent images, and k represents the middle index of the sequence. The transformation matrix from the input image to the reference image is obtained by statistical matrix set. Due to the transitivity of the matrix, the transformation matrix from any image to the reference image can be obtained, and then the L-M algorithm is used to adjust the transformation matrix to further reduce the cumulative matching error. The image registration process is completed. The distortion degree of the final stitched image is reduced from the accuracy requirement of image registration.

[0104] In some alternative embodiments, extracting multiple images to be stitched in each frame based on the video stream from a multi-camera includes: corresponding to each camera's video stream in the multi-camera, extracting a set of original image sequences; preprocessing each set of original image sequences by using a cylindrical projection method, where the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each set of original image sequences and the object being photographed.

[0105] In a specific implementation, after extracting multiple sets of original image sequences from multiple video streams, cylindrical projection is used to preprocess the original images of each camera to reduce the distortion degree of the images during perspective transformation.

[0106] During the movement, the camera can adaptively adjust the focal length of the best cylindrical projection according to the distance of the object being photographed, and the best adjustment parameter is given by the initial measurement value. The processing process of a single-frame image is accelerated by CUDA, and the time can be greatly shortened, effectively ensuring the real-time performance of video stitching.

[0107] The cylindrical projection process includes the following:

[0108] (1) Let the pixel coordinates of the original image be (x, y), where x ∈ [0, W], y ∈ [0, H], and W and H are the image width and height respectively. The normalized coordinates with the image center as the origin are:

[0109]

[0110] (2) Set the camera focal length as f, and the projection coordinates (X c , Y c , Z c ) of each pixel point are given by the following formula:

[0111] Z c = f

[0112] wherein, X c is projected from x' of the input image onto the cylinder through the arctangent function, and Y c is calculated through trigonometric relations to keep the longitudinal ratio consistent.

[0113] (3) The new pixel position after projection is given by the following formula:

[0114]

[0115] (4) Finally, the color value of the target pixel is calculated using bilinear interpolation. The mapping relationship between the projected image and the original image is as follows: I'(x'', y'') = I(x, y)

[0116] In this embodiment, through cylindrical projection, the planar images captured by different cameras can be converted into cylindrical images, making the stitched image more in line with the visual habits of the human eye and reducing the distortion feeling after stitching. The parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each group of original image sequences and the object being photographed. Different shooting distances will result in different viewing angles and distortion degrees of the images. By dynamically adjusting the parameters of the cylindrical projection, these changes can be better adapted, and the quality of the projected image can be further improved.

[0117] In some optional embodiments, determining a new stitching line within the overlapping region of the currently registered and aligned image of the current frame includes: performing light compensation on the currently registered and aligned image of the current frame to obtain a light-compensated image; determining the stitching line within the overlapping region of the light-compensated image.

[0118] In this embodiment, when the multi-view camera is shooting during movement, due to the diversity of weather and the difference in visibility within the horizontal 360-degree range, even for the data collected at the same moment, there will be exposure differences between the video data due to reasons such as the viewing angle of the camera or autofocus. If such a set of data is directly stitched, it will cause obvious differences in brightness and color in the stitched image, and obvious stitching lines will also appear, affecting the effect of video stitching. Therefore, after image registration, light compensation should be performed first, and then image fusion should be performed to obtain a panoramic stitched image with natural transition.

[0119] When performing light compensation, a block-based image compensation method can be used. After dividing the image into blocks, the compensation coefficient of each image block is calculated. By establishing a light intensity error function model, the light compensation gain matrix is solved, and finally, the corresponding region in the source image is light-compensated using the gain coefficient in the matrix.

[0120] In some alternative embodiments, determining a new suture line within the overlapping region of the currently registered and aligned image of the current frame includes: applying an optical flow detection algorithm to insert horizontal and vertical displacement parameters in the currently registered and aligned image of the current frame for displacement compensation to obtain a displacement-compensated image; determining the suture line within the overlapping region of the displacement-compensated image.

[0121] In this embodiment, for the situation where the background changes drastically globally due to camera movement, an optical flow detection algorithm can be optionally introduced to further stabilize the video frame image. After performing illumination compensation or registration, the optical flow field is used to analyze the motion information of each pixel, and horizontal and vertical displacement parameters are inserted in the currently registered and aligned image of the current frame for displacement compensation, eliminating the displacement difference between adjacent images to be stitched, thereby reducing the stitching error caused by displacement and improving the stitching accuracy.

[0122] In some alternative embodiments, the process of determining the suture line includes: defining an energy function based on the gray-scale difference between two displacement-compensated images to be stitched; within the overlapping region, obtaining the suture line through dynamic programming with the minimum cumulative energy.

[0123] In this embodiment, by searching for the suture line with the minimum energy within the overlapping region through dynamic programming, the ghosting and blurring phenomena when a moving object passes through the suture line can be effectively improved.

[0124] In some alternative embodiments, the calculation formula of the energy function is as follows:

[0125] E(x,y) = |I1(x,y) - I2(x + Δ x (x,y), y + Δ y (x,y))|

[0126] where I1(x,y) and I2(x,y) respectively represent the gray-scale values of the two displacement-compensated images, and Δ x (x,y) and Δ y (x,y) are the horizontal and vertical displacements obtained through the optical flow detection algorithm at the point (x,y) respectively;

[0127] The calculation formula of the cumulative energy is as follows:

[0128] C(x,y) = E(x,y) + min{C(x - 1,y - 1), C(x,y - 1), C(x + 1,y - 1)};

[0129] where E(x,y) is the energy at the current pixel point, and C(x - 1,y - 1), C(x,y - 1), and C(x + 1,y - 1) are the cumulative energies at the pixel points in the upper left, directly above, and upper right of the current pixel point respectively.

[0130] In this embodiment, the energy function E(x, y) measures the grayscale difference between two images to be stitched at the point (x, y), taking into account the displacement obtained through the optical flow detection algorithm. The cumulative energy is set to be the energy at the current pixel point plus the minimum cumulative energy of its upper left, directly above, or upper right, so as to find the optimal stitching path during the image stitching process, effectively reducing the stitching error and improving the stitching accuracy and effect.

[0131] In some alternative embodiments, the real-time fusion of the images to be stitched based on the stitching line of each frame of the images to be stitched includes: performing multi-band fusion on the images to be stitched based on the stitching line of each frame of the images to be stitched.

[0132] In this embodiment, compared with other fusion algorithms, the multi-band fusion algorithm utilizes the multi-scale information of the images, enabling the images to achieve a smooth transition in the overlapping area, effectively avoiding obvious stitching seams and ghosting phenomena. The complex calculation process is accelerated using CUDA to achieve the operation speed of other fusion algorithms. In this embodiment, based on the dynamic update of the stitching line, pixel information of the corresponding images is taken on both sides of the stitching line, and an improved multi-band fusion is adopted to achieve a seamless stitched panoramic image with visual consistency.

[0133] Specifically, the multi-band fusion of the images to be stitched based on the stitching line of each frame of the image includes: at each pyramid level, fusing the Laplacian coefficients of the two images in a weighted manner:

[0134]

[0135] where and are the Laplacian pyramids of the two images respectively, is the original smooth transition weight, and Φ(·) is a function adjusted according to the motion amplitude;

[0136] From the fused Laplacian pyramid reconstruct the fused image, and the recurrence calculation formula is as follows:

[0137]

[0138] where F i is the image of the i-th layer of the fused Laplacian pyramid, F N-1 is the top-layer image of the fused Laplacian pyramid, and Expand(F i+1 ) is the process of expanding the image of the (i + 1)-th layer to the i-th layer.

[0139] In this embodiment, the Laplacian pyramid can effectively retain the high-frequency details of the image, ensuring that the fused image remains clear and sharp. By fusing the Laplacian coefficients of the two images in a weighted manner, the contribution of each image to the final fused result can be adjusted as needed, achieving a more natural transition. Using the smooth transition weight function Φ(·) can be adjusted according to the motion amplitude, providing greater flexibility to adapt to different image contents and motion conditions, helping to reduce artifacts and abrupt changes at the stitching, and making the fused result more natural.

[0140] Embodiment 2:

[0141] Based on the above embodiment, this embodiment provides a specific application example. The method for video stitching in a dynamic scene provided in this embodiment can be applied to a real-time panoramic monitoring system carried on an unmanned vehicle, making it intelligent. When the unmanned vehicle is maneuvered into a certain dangerous area, real-time all-round detection can be carried out. For example, when the real-time panoramic monitoring system is carried on a ship sailing in the ocean, it will enhance the breadth and speed of ship identification, providing strong support for maritime safety and law enforcement operations. As Figure 6 shown, it is a schematic diagram of the overall process of the method for video stitching in a dynamic scene in this embodiment.

[0142] Specifically, first, in terms of image acquisition, multiple industrial camera modules are used to form a circular camera array for collecting real-time video streams with a 360-degree horizontal overlap area. As Figure 7 shown, it is a schematic diagram of the structure of the multi-camera in this embodiment. Since the viewing angle range of ordinary industrial cameras is 90 degrees to 120 degrees, in this embodiment, 8 industrial camera modules are used to form a circular array for obtaining images in different directions around 360 degrees, ensuring that the overlapping range of the images captured by adjacent cameras accounts for more than 30% of a single captured image, so as to ensure the alignable stitching of the panoramic image.

[0143] In the preprocessing of the first-frame image of the multi-camera, cylindrical projection is used to reduce the distortion degree during image transformation; in the overlapping area of the image, an improved SURF registration algorithm is used for image registration in the stitching manner from the image to the stitched image, and then the cylindrical images from different perspectives are transformed to the reference plane by perspective transformation; before fusion, first, the illumination gain compensation algorithm is used to eliminate the exposure difference of the images captured by the multi-camera, and then a fusion algorithm combining an improved optimal stitching line and a multi-band fusion algorithm is used to avoid ghosting and blurring phenomena when moving objects pass through the stitching line. The stitching of the horizontal 360-degree panoramic video in a mobile environment is realized.

[0144] After the first frame of the multi-camera is obtained, cylindrical projection is used to preprocess the original images of each camera, reducing the distortion degree of the images during perspective transformation. During the movement, the camera needs to adaptively adjust the focal length of the best cylindrical projection according to the distance of the object being photographed, and the best adjustment parameters are given by the initial measurement values. The processing of a single-frame image in this step is accelerated by CUDA, and the time can be shortened to 0.05% of the original, effectively ensuring the real-time performance of video stitching.

[0145] The process of cylindrical projection is as follows:

[0146] (1) Let the pixel coordinates of the original image be (x, y), where x ∈ [0, W], y ∈ [0, H], and W and H are the image width and height respectively. The normalized coordinates with the image center as the origin are:

[0147]

[0148] (2) Set the camera focal length to f, and the projection coordinates (X c , Y c , Z c ) of each pixel are given by the following formula:

[0149] Z c = f

[0150] Among them, X c is projected from x' of the input image to the cylinder through the arctangent function, and Y c is calculated through trigonometric relations to keep the longitudinal ratio consistent.

[0151] (3) The new pixel position after projection is given by the following formula:

[0152]

[0153] (4) Finally, bilinear interpolation is used to calculate the color value of the target pixel, and the mapping relationship between the projected image and the original image is as follows: I'(x″, y″) = I(x, y).

[0154] In the image registration stage, the SURF algorithm based on threshold is used to extract and match features in the overlapping area of the images. Since the overlapping range of the images captured by eight industrial cameras adjacent to each other accounts for more than 30% of a single captured image, there are certain errors in the physical installation positions of the cameras. Therefore, the detection threshold is set to 30% in the algorithm, which means the detection area accounts for 30% of a single image. For the left-viewpoint image, only the 30% overlapping area on the right is detected; for the right-viewpoint image, only the 30% overlapping area on the left is detected; if there are fewer feature points in the overlapping area of the video frames captured at a certain moment during movement, the threshold is relaxed to the entire overlapping area. By controlling the detection range of the overlapping area between adjacent images, on the one hand, the relative error of the overlapping area caused by the physical placement method between adjacent cameras can be improved, the detected overlapping range can be reduced, and the execution efficiency of the algorithm can be enhanced; on the other hand, the detection and description of a large number of redundant feature points can be avoided, and the interference caused by mis-matched points in non-stitching areas can be reduced.

[0155] Before image fusion, when multiple industrial cameras are shooting during movement, due to the diversity of weather and the difference in visibility within the horizontal 360-degree range, etc. Even for the data collected at the same moment, due to reasons such as the camera's perspective or autofocus, there will be exposure differences between video data. If such a set of data is directly stitched, it will cause obvious differences in brightness and color in the stitched image, and obvious stitching lines will also appear, affecting the effect of video stitching. Therefore, after image registration, first, light compensation should be performed, and then image fusion should be carried out to obtain a panoramic stitched image with natural transition.

[0156] In this embodiment, a block-based image compensation method is adopted. After the image is divided into blocks, the compensation coefficient of each image block is calculated. By establishing a light intensity error function model, solving the light compensation gain matrix, and finally using the gain coefficients in the matrix to perform light compensation on the corresponding areas in the source image. For the situation where the background changes violently globally due to camera movement, motion detection based on optical flow can be selectively introduced to further stabilize the video frame image. After compensation or registration, the optical flow field is used to analyze the motion information of each pixel. By comparing the difference between the local motion vector and the overall (global) motion vector, the areas inconsistent with the background motion can be identified, and then the real moving objects can be detected.

[0157] In the image fusion stage, for the selected key frames, first, moving objects existing in the video frames are detected by the background subtraction method, that is, when a moving object passes through the optimal stitching line, the optimal stitching line is dynamically updated. In the solution of the energy function, motion estimation methods such as optical flow are used to insert horizontal and vertical displacement parameters for displacement compensation. Finally, the improved multi-band fusion is used to decompose and reconstruct the image to obtain the panoramic stitched image.

[0158] Based on the best stitching line found, the steps of fusing the spliced video frame images using the improved multi-band fusion algorithm are as follows:

[0159] (1) In the overlapping region, in order to avoid moving objects as much as possible, a dynamic programming method based on an energy function is used to find the best stitching line. The energy function is defined as:

[0160] E(x,y) = |I1(x,y) - I2(x + Δ x (x,y), y + Δ y (x,y))|

[0161] where I1(x,y) and I2(x,y) represent the grayscale or color values of the spliced image and the image to be spliced respectively. The smaller the energy, the closer the two images are at this position and the more suitable for stitching. Here, in order to compensate for the image displacement caused by camera movement, Δ x (x,y) and Δ y (x,y) are the horizontal and vertical displacements obtained by motion estimation methods such as optical flow at the point (x,y) respectively.

[0162] (2) To ensure the cumulative minimum energy from top to bottom and thus find an optimal path extending from the upper boundary to the lower boundary of the overlapping region. Using the updated energy function, the cumulative energy function C(x,y) is constructed, and its recurrence relation is:

[0163] C(x,y) = E(x,y) + min{C(x - 1,y - 1), C(x,y - 1), C(x + 1,y - 1)}

[0164] where the boundary conditions need to be appropriately initialized according to the boundary of the overlapping region.

[0165] (3) Starting from the pixel with the minimum energy value at the bottom of the overlapping region, trace back upward along the minimum path saved during the dynamic programming recurrence to obtain the final best stitching line.

[0166] (4) Perform multi-band fusion. At each pyramid level, in order to better handle the camera movement situation, the Laplacian coefficients of the two images are updated by fusing them in a weighted manner as:

[0167]

[0168] where is the original smooth transition weight, which plays a role in smooth transition, making the fusion boundary more natural. Φ(·) is a function adjusted according to the motion amplitude. When the motion amplitude is large, the smooth effect of the weight is reduced to expand the transition region and make the image fusion more natural.

[0169] (5) From the fused Laplacian pyramid Reconstruct the fused image, and the recurrence formula is:

[0170]

[0171] During the image fusion process, first, through the background subtraction method based on the Vibe algorithm, when significant changes are detected in the background, the Vibe algorithm will choose to discard the original background model and use the first frame image after the change to re - establish a new background model. The foreground object detection process of the Vibe algorithm is to compare each pixel point in each frame image with the pixel points in its background model in turn, calculate the Euclidean distance between pixel points, and then judge whether the current pixel point belongs to the foreground object or the background through a set threshold. After classifying the image, counting the foreground points can obtain the information of the moving object, and CUDA is used for acceleration processing.

[0172] There are three situations for the Vibe algorithm to update the background model. First, for foreground pixel points, if the pixel point is continuously judged as a foreground point N times, then the foreground point is updated to a background pixel point to solve the problem of long - term retention of moving objects in the foreground. Second, for the background model of each pixel point, there is a probability of 1 / φ to be updated. Third, when each pixel point is updated, there is a probability of 1 / φ to spread the update to neighboring pixel points.

[0173] The background subtraction method based on the Vibe algorithm, for the dynamic background in the moving situation, while detecting SURF feature points and updating the perspective transformation matrix for the key frames of the video simultaneously to achieve background alignment, introduces the Vibe algorithm to update the background model, which can better solve the problem of dynamic background.

[0174] After obtaining the binary image of the foreground area through the background subtraction method of the Vibe algorithm, it can be judged whether the current frame of the image to be stitched can use the best stitching line calculated in the previous frame of the image to be stitched. If there are pixel points on the best stitching line calculated in the previous frame that fall within the foreground area, then it is necessary to recalculate the best stitching line for the current frame of the image to be stitched, that is, update the best stitching line. Otherwise, the current frame of the image to be stitched can continue to use the best stitching line calculated in the previous frame.

[0175] The Vibe algorithm only needs to construct and initialize the background model in the first frame of the video, and the next frame image in the fusion process can perform moving object detection. The establishment of the background model is the first step, and the second step is a process of judging foreground objects based on the established background model, which involves a process of comparing all pixel points in each frame image in image fusion with the pixel points in the background model. A threshold is used to judge whether the current pixel point belongs to the foreground object or the background. The main function of the background subtraction method based on the vibe algorithm is foreground object detection.

[0176] The foreground object detection of the Vibe algorithm requires processing each frame of the image during the image fusion process. In the registration process, SURF feature detection and homography transformation only need to process key frames because it involves the update of the subsequent optimal stitching line.

[0177] In actual stitching, the algorithm core of video stitching technology is still the image stitching algorithm. Generally, the better algorithms in image stitching are complex and time-consuming. It is obviously unrealistic to apply the image stitching algorithm to each frame of the panoramic video, and it cannot meet the real-time requirement in practical applications. Based on the technical solution of static image stitching, this embodiment proposes a real-time video stitching method, providing technical support for establishing an intelligent vehicle panoramic video stitching system.

[0178] The video stitching method for dynamic scenes proposed in this embodiment, from four parts: image acquisition, image registration, image illumination gain compensation, and image fusion, through the algorithm processing of video key frames and the physically mounted servo pan-tilt, can be used in the real-time panoramic video monitoring system of unmanned vehicles. To ensure the real-time performance of video stitching, CUDA programming is used to complete the GPU optimization of the stitching algorithm, greatly improving the program running speed.

[0179] Embodiment 3:

[0180] Another embodiment of this application relates to a device for video stitching in a dynamic scene. The implementation details of the device for video stitching in a dynamic scene of this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing this solution. The schematic diagram of the device for video stitching in a dynamic scene of this embodiment can be as Figure 8 shown, including a to-be-stitched image extraction module 810, an initialization module 820, an update module 830, and a fusion module 840.

[0181] The to-be-stitched image extraction module 810 is used to extract multiple to-be-stitched images in each frame based on the video stream from a multi-camera, where there is an overlapping shooting area between two adjacent cameras in the multi-camera, and the to-be-stitched images corresponding to the two adjacent cameras have an overlapping area;

[0182] An initialization module 820, configured to determine a first-frame registered image among all images to be stitched, use the transformation matrix used by the first-frame registered image in the registration process as the initial transformation matrix, use the stitching line calculated based on the overlapping area of the first-frame registered image as the initial stitching line, and establish an initial background model based on the first-frame registered image, where the transformation matrix is calculated based on feature points in the overlapping area of adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on the Vibe algorithm for determining key-frame images with moving targets in subsequent frames of images to be stitched;

[0183] An update module 830, configured to, for subsequent frames of images to be stitched, when the background model detects that the current video-frame image is the key-frame image, extract the feature information of adjacent images to be stitched in the current frame to update the transformation matrix, register the current-frame image to be stitched based on the updated transformation matrix, re-determine a new stitching line within the overlapping area of the registered current-frame aligned image, and update the background model based on the registered current-frame aligned image to determine whether the next frame of image to be stitched is the key-frame image; otherwise, keep the transformation matrix, the background model, and the stitching line of the previous frame unchanged, and process the current-frame image to be stitched according to the transformation matrix and the stitching line of the previous frame;

[0184] A fusion module 840, configured to perform real-time fusion on the images to be stitched based on the stitching line of each frame of image to be stitched to obtain a stitched video.

[0185] It should be noted that each module involved in this embodiment is a logical module. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or implemented as a combination of multiple physical units. In addition, to highlight the innovative part of this application, units not closely related to solving the technical problems proposed in this application are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0186] In some optional embodiments, the image extraction module to be stitched includes:

[0187] A raw image extraction unit, configured to correspondingly extract a set of raw image sequences based on the video stream of each camera in the multi-camera;

[0188] A preprocessing unit, configured to preprocess each set of raw image sequences by using a cylindrical projection method, where the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each set of raw image sequences and the object being photographed.

[0189] In some optional embodiments, the update module includes:

[0190] A lighting compensation unit for performing lighting compensation on the registered and aligned current frame image to obtain a lighting compensated image;

[0191] A suture determination unit for determining sutures within the overlapping region of the lighting compensated image.

[0192] In some alternative embodiments, the update module further includes:

[0193] A displacement compensation unit for applying an optical flow detection algorithm to insert horizontal and vertical displacement parameters in the registered and aligned current frame image for displacement compensation to obtain a displacement compensated image;

[0194] The suture determination unit is further configured to determine the suture within the overlapping region of the displacement compensated image.

[0195] In some alternative embodiments, the real-time fusion of the images to be stitched based on the sutures of each frame of the images to be stitched includes:

[0196] Performing multi-band fusion on the images to be stitched based on the sutures of each frame of the images to be stitched.

[0197] In some alternative embodiments, the fusion module includes:

[0198] A coefficient fusion unit for fusing the Laplacian coefficients of two images in a weighted manner at each pyramid level:

[0199]

[0200] Wherein, and are respectively the Laplacian pyramids of the two images, is the original smooth transition weight, and Φ(·) is a function adjusted according to the motion amplitude;

[0201] A fused image reconstruction unit for reconstructing the fused image from the fused Laplacian pyramid The recursive calculation formula is as follows:

[0202]

[0203] Wherein, F i is the image of the fused Laplacian pyramid at the i-th layer, F N-1 is the top layer image of the fused Laplacian pyramid, and Expand(F i+1 ) is the process of expanding the image of the (i + 1)-th layer to the i-th layer.

[0204] In some alternative embodiments, it further includes:

[0205] An acceleration module for accelerating the processing using CUDA during the execution of the method.

[0206] Embodiment Four:

[0207] Another embodiment of the present application relates to an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for video stitching in a dynamic scenario in the above embodiments.

[0208] Wherein, the memory and the processor are connected in a bus manner. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and the memory together. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, etc., which are well known in the art, so they will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be an element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on the transmission medium. The data processed by the processor is transmitted over the wireless medium through the antenna. Further, the antenna also receives data and transmits the data to the processor.

[0209] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor when executing operations.

[0210] Embodiment Five:

[0211] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method embodiments described above are implemented.

[0212] That is, those skilled in the art can understand that all or part of the steps in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program is stored in a storage medium, including several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs, etc., which can store program codes.

[0213] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application. In actual applications, various changes can be made to them in form and details without departing from the spirit and scope of the present application.

Claims

1. A method for video splicing in a dynamic scene, characterized in that: include: Based on the video stream from the multi-camera, a plurality of images to be stitched in each frame are extracted, wherein two adjacent cameras in the multi-camera have a shooting overlap area, and the images to be stitched from the two adjacent cameras have an overlap area; Determine a first-frame registration image among all images to be stitched, use a transformation matrix used by the first-frame registration image in the registration process as an initial transformation matrix, use a stitching line calculated based on an overlapping area of ​​the first-frame registration image as an initial stitching line, and establish an initial background model based on the first-frame registration image, wherein the transformation matrix is ​​calculated based on feature points of overlapping areas of adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on a Vibe algorithm, which is used to determine a key frame image with a moving target in a subsequent frame of images to be stitched; For subsequent frames of images to be stitched, when the background model detects that the current video frame image is the key frame image, feature information of adjacent images to be stitched of the current frame is extracted to update the transformation matrix, the images to be stitched of the current frame are registered based on the updated transformation matrix, a new stitching line is re-determined in the overlapping area of ​​the registered aligned images of the current frame, and the background model is updated based on the registered aligned images of the current frame to determine whether the next frame of images to be stitched is the key frame image, otherwise, the transformation matrix, the background model and the stitching line of the previous frame are maintained unchanged, and the images to be stitched of the current frame are processed according to the transformation matrix and the stitching line of the previous frame; The images to be stitched are fused in real time based on the stitching lines of each frame to obtain a stitched video.

2. The method for video splicing in dynamic scenes according to claim 1, characterized in that: The method extracts multiple images to be stitched in each frame based on the video stream from the multi-camera, including: Based on the video stream of each camera in the multi-camera, a set of original image sequences are correspondingly extracted; Each group of original image sequences is preprocessed by a cylindrical projection method, wherein the parameters of the cylindrical projection are dynamically adjusted based on the distance between the camera corresponding to each group of original image sequences and the photographed object.

3. The method for video splicing in a dynamic scene according to claim 1, characterized in that: The re-determining a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes: Performing illumination compensation on the aligned image of the current frame after registration to obtain an illumination compensated image; A stitch line is determined in an overlapping region of the illumination compensated images.

4. The method for video splicing in a dynamic scene according to claim 1, characterized in that: The re-determining a new stitching line in the overlapping area of ​​the aligned images of the current frame after registration includes: Applying the optical flow detection algorithm to insert the horizontal and vertical displacement parameters into the aligned image of the current frame after registration to perform displacement compensation, and obtain a displacement compensated image; The stitching line is determined in an overlapping region of the displacement-compensated images.

5. The method for video splicing in dynamic scenes according to claim 4, characterized in that: The process of determining the stitching line includes: defining an energy function based on the grayscale difference between the two displacement-compensated images to be stitched; In the overlapping region, the stitching line is obtained by dynamic programming by minimizing the cumulative energy.

6. The method for video splicing in dynamic scenes according to claim 5, characterized in that: The calculation formula of the energy function is as follows: E(x,y)=|I1(x,y)-I2(x+Δ x (x,y),y+Δ y (x,y))| Wherein, I1(x, y) and I2(x, y) represent the grayscale values ​​of the two displacement compensation images respectively, Δ x (x,y) and Δ y (x, y) are the horizontal and vertical displacements obtained by the optical flow detection algorithm at the point (x, y); The calculation formula of the cumulative energy is as follows: C(x,y)=E(x,y)+min{C(x-1,y-1),C(x,y-1),C(x+1,y-1)}; Among them, E(x,y) is the energy at the current pixel, C(x-1,y-1), C(x,y-1), and C(x+1,y-1) are the cumulative energies of the pixels to the upper left, directly above, and upper right of the current pixel, respectively.

7. The method for video stitching in a dynamic scene according to any one of claims 1 to 6, characterized in that: The real-time fusion of the images to be stitched based on the stitching lines of each frame of the images to be stitched includes: The multi-band fusion of the images to be stitched is performed based on the stitching lines of each frame of the images to be stitched.

8. The method for video splicing in dynamic scenes according to claim 7, characterized in that: The multi-band fusion of the image to be stitched based on the stitching line of each frame of image includes: At each pyramid level, the Laplacian coefficients of the two images are fused in a weighted manner: in, and They are the Laplacian pyramids of the two images respectively. is the original smooth transition weight, Φ(·) is the function adjusted according to the motion amplitude; From the fused Laplacian pyramid Reconstruct the fused image, and the recursive calculation formula is as follows: Among them, F i is the image of the fused Laplacian pyramid at the i-th layer, F N-1 is the top image of the fused Laplacian pyramid, Expand(F i+1 ) is the process of expanding the image of the i+1th layer to the ith layer.

9. The method for video splicing in a dynamic scene according to any one of claims 1 to 6, characterized in that: The method adopts CUDA to perform accelerated processing during execution.

10. A device for video splicing in a dynamic scene, characterized in that: include: The image extraction module to be stitched is used to extract a plurality of images to be stitched in each frame based on the video stream from the multi-camera, wherein two adjacent cameras in the multi-camera have a shooting overlap area, and the images to be stitched from the two adjacent cameras have an overlap area; An initialization module, used to determine a first-frame registration image among all images to be stitched, use a transformation matrix used by the first-frame registration image in the registration process as an initial transformation matrix, use a stitching line calculated based on an overlapping area of ​​the first-frame registration image as an initial stitching line, and establish an initial background model based on the first-frame registration image, wherein the transformation matrix is ​​calculated based on feature points of an overlapping area of ​​adjacent images to be stitched from adjacent cameras, and the background model includes a background subtraction model based on a Vibe algorithm, which is used to determine a key frame image with a moving target in a subsequent frame of images to be stitched; An updating module, for extracting feature information of adjacent images to be stitched of the current frame to update the transformation matrix for subsequent frames of images to be stitched, when the background model detects that the current video frame image is the key frame image, registering the images to be stitched of the current frame based on the updated transformation matrix, re-determining a new stitching line in an overlapping area of ​​the registered aligned images of the current frame, and updating the background model based on the registered aligned images of the current frame to determine whether the next frame of images to be stitched is a key frame image, otherwise, maintaining the transformation matrix, the background model and the stitching line of the previous frame unchanged, and processing the images to be stitched of the current frame according to the transformation matrix and the stitching line of the previous frame; The fusion module is used to fuse the images to be spliced ​​in real time based on the stitching lines of each frame of the images to be spliced ​​to obtain a spliced ​​video.

11. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for video stitching in a dynamic scene as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for video stitching in a dynamic scene according to any one of claims 1 to 9 is implemented.

Images