A video fusion algorithm based on dynamic optimal stitching line and improved fade-in and fade-out method

Through dynamic optimal suture and improved video fusion algorithm of in-depth method, the problems of ghosting, blur, splicing seams and transitional non-smoothing caused by moving objects and lighting differences in the prior art are solved, and real-time high resolution is achieved The generation of panoramic video streams improves splicing effect and efficiency.

CN113221665BActive Publication Date: 2024-11-19SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110419556.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-19
Publication Date
2024-11-19
Estimated Expiration
2041-04-19

AI Technical Summary

Technical Problem

Existing video stitching algorithms are difficult to take into account both real-time and visual quality, especially when moving objects have different differences through sutures and lighting, they are prone to ghosting, blurring, splicing seams and non-smooth transitions.

Method used

Using video fusion algorithm based on dynamic best sutures and improved in-depth in-depth method, we can find the best sutures through camera calibration, distortion correction, image registration, spherical projection mapping and dynamic programming algorithms, and combine mixed Gaussian models for background removal , and use an improved in-depth in-out method for image fusion, dynamically update the best sutures to avoid the influence of moving objects.

Benefits of technology

Effectively eliminate ghosting, blurring, splicing seams and lighting differences in stitching images, real-time high-resolution panoramic video stream generation, and improve stitching effect and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113221665B_ABST
    Figure CN113221665B_ABST
Patent Text Reader

Abstract

The invention discloses a video fusion algorithm based on dynamic optimal stitching line and improved fade-in and fade-out method, belonging to the field of video information processing technology. Real-time video streams with overlapping areas are collected by a multi-camera system; feature points of the overlapping areas are extracted and roughly matched and purified, and the intrinsic parameter matrix and rotation matrix of each camera are calculated in combination with the bundle adjustment method; after spherical projection of images between different viewing angles and distortion to the same reference plane, the video images are spliced ​​by the optimal stitching line algorithm, and in the case where ghosting and blurring are generated when a moving object passes through the overlapping area of ​​the image, the optimal stitching line dynamic update algorithm is used to avoid the ghosting and blurring generated when the moving object passes through the stitching line; the improved fade-in and fade-out fusion algorithm is used to obtain a wide field of view and high-resolution panoramic stitching video with good visual effects. The steps are simple, and a splicing image with good visual effects can be quickly obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a video fusion algorithm, in particular to a video fusion algorithm based on a dynamic optimal stitching line and an improved fade-in and fade-out method suitable for a multi-camera system, and belongs to the technical field of video information processing. Background Art

[0002] Vision is the most effective means for humans to obtain information. About 80% of the information humans receive every day is obtained through vision. In today's society, digital cameras and other shooting equipment, photography and video technology have become a necessity for every family, and multimedia information such as video and images play an important role in many fields such as monitoring security, intelligent driving, entertainment, remote sensing, and military. At present, most video shooting equipment on the market can obtain a small field of view, far less than the field of view of the human eye, which makes it difficult to meet people's current needs for large field of view and high-resolution video. Therefore, video splicing technology came into being.

[0003] Video stitching technology is based on image stitching technology. Video stitching is to stitch together the video streams with overlapping areas collected by multiple cameras according to certain registration and fusion technologies to form a panoramic video stream with a large field of view and high resolution. Video stitching technology is widely used in many fields, but most of the current video stitching methods are difficult to meet the growing real-time and visual quality requirements.

[0004] Image stitching mainly includes two aspects: image registration and image fusion. At present, scholars at home and abroad have proposed a variety of different image stitching methods. In terms of image registration, Lowe proposed a feature detection image registration algorithm based on scale-invariant feature transform (SIFT) in 2003. This method can ensure the stability of affine, noise, perspective and other performances without changing the properties of local features of the image (such as brightness, rotation, and scaling), so it has good robustness. However, the calculation of the SIFT algorithm is relatively complex, so the real-time performance is not high, and the ability to extract feature points of targets with smooth edges is weak. In order to improve the defects of the SIFT algorithm, Bay et al. proposed the SURF algorithm in 2006, which improved the extraction and description of feature points, making the matching of feature points more efficient. In 2011, Roblee proposed the ORB image registration method, which combined the FAST corner detection and BRIEF feature point descriptor generation algorithms, greatly improving the speed of image stitching. At present, the ORB algorithm is the fastest stitching algorithm in the world, but because the ORB algorithm is not scale invariant, its application scope is much more limited than that of SURF and SIFT algorithms.

[0005] In terms of image fusion, after image registration, multiple images need to be fused into a visually consistent seamless wide-field image. There have been many studies on image fusion methods at home and abroad. Since simple Alpha fusion and feather fusion cannot handle the seams and ghosting problems that occur during image fusion. In order to solve the ghosting and seams problems that occur during the stitching process, Burt proposed a multi-band fusion method based on image pyramid. Patrick Perez defined the image fusion problem as an image editing problem, and used the Poisson partial differential equation and Dirichlet boundary conditions to solve the problem to obtain a seamless fusion image with good visual consistency. However, since solving the Poisson partial differential equation itself is a variational problem, the time complexity of solving the equation is very high. In order to improve efficiency, Zeev Farbman proposed that the original image editing problem can be converted into a mean coordinate system method that uses boundary values ​​to interpolate the interior of the region, which improves the speed of image fusion while ensuring the image fusion effect. In order to further improve the computational efficiency, Zeev Farbman proposed an algorithm in 2011 that uses a convolutional pyramid to replace the mean coordinate system, which further improves the computational efficiency while ensuring the fusion performance.

[0006] The biggest difference between video stitching technology and image stitching technology is that there will be interference from moving objects in video stitching. In image stitching technology, the use of the best stitching line algorithm can effectively avoid the occurrence of ghosting and blurring in overlapping areas, but in video stitching, when a moving object passes through the best stitching line, it will produce obvious ghosting, blurring and stitching seams. Therefore, for video stitching, it is necessary to update the best stitching line in real time to avoid the passage of moving objects. Most existing video stitching algorithms cannot obtain panoramic videos with good visual effects while ensuring real-time.

[0007] Since cameras with different viewing angles capture the same area with a certain degree of parallax, this will cause blurring and ghosting in the overlapping areas of the stitched image, especially when a moving object passes through the stitching line, this phenomenon will become more obvious. In addition, the video images captured by different cameras may have different lighting differences, which will cause obvious stitching seams and uneven transitions in the overlapping areas of the final stitched image. Currently, most algorithms cannot take into account both real-time performance and stitching image effects at the same time. Summary of the invention

[0008] Technical problem: To address the above technical problems, a video fusion algorithm based on a dynamic optimal stitching line and an improved fade-in and fade-out method is provided, which can effectively solve the problems of ghosting, blurring, stitching seams and uneven transitions caused by moving objects passing through stitching lines and different lighting differences, while taking into account both real-time performance and stitching image effects.

[0009] Technical solution: To achieve the above technical objectives, the present invention provides a video fusion algorithm based on dynamic optimal stitching lines and improved fade-in and fade-out methods. First, a multi-camera system with a fixed position is used to collect video information at the same time to obtain multiple video streams. The video images in the multiple video streams are synchronized, decoded, distorted, aligned, projected, mapped, stitched, and fused, thereby integrating the multiple video streams into a real-time panoramic video stream with a large field of view and high resolution.

[0010] The specific steps are as follows:

[0011] Step 1: Use a multi-camera system to shoot a video containing a high-speed moving object, and calibrate the multi-camera system to obtain the distortion coordinate mapping matrix of the camera lens;

[0012] Step 2: Correct the first frame of each video stream collected by the multi-camera system according to the distortion coordinate mapping matrix and perform image registration, and then combine the bundle adjustment method to obtain the precise intrinsic parameter matrix and rotation matrix of each eyepiece lens;

[0013] Step 3: Select the imaging plane of one of the lenses of the multi-eye camera system as the reference plane, and perform forward spherical mapping and then reverse mapping to the reference plane on the corrected first frame of video image according to the intrinsic parameter matrix and rotation matrix of each eyepiece lens to obtain the projection mapping matrix;

[0014] Step 4: For each subsequent frame image in the video stream, distortion correction and spherical projection mapping are performed according to the distortion coordinate mapping matrix and the projection mapping matrix respectively.

[0015] Step 5: Calculate the overlapping area between the images taken by adjacent camera lenses in the multi-camera system using the homography transformation matrix between adjacent images;

[0016] Step 6: Gaussian blur and downsample the overlapping area images, that is, reduce the amount of calculation by reducing the image scale;

[0017] Step 7: Perform background removal based on the mixed Gaussian model MOG on the overlapping area image after Gaussian blur and downsampling, so as to obtain the segmented foreground image;

[0018] Step 8: Calculate the best stitching line of the overlapping area image of the first frame using the best stitching line algorithm. For each subsequent frame, determine whether it is necessary to re-search the best stitching line based on the foreground image. Determine whether it is necessary to re-search by determining whether there are pixels on the stitching line falling in the foreground image area of ​​the current frame. If so, recalculate the best stitching line of the current frame and update it. Otherwise, continue to use the best stitching line of the previous frame.

[0019] Step 9: If the best seam line is updated, the best seam line algorithm based on dynamic programming is used to search and update the image at a small scale;

[0020] Step 10: Search for the best seam line in the small-scale image and perform linear interpolation to obtain the best seam line at the original scale;

[0021] Step 11: Based on the found optimal stitching line, an improved fade-in and fade-out fusion algorithm is used to fuse multiple images acquired by the multi-camera system at the same time, and finally a complete fused panoramic image is obtained.

[0022] The multi-eye camera system is the Insta360 Pro2 multi-eye camera system, which is spherical in shape. Six fisheye lenses with the same optical properties are evenly distributed on the same horizontal plane around the body. The field of view is much wider than that of the human eye. The videos captured between adjacent lenses have a certain overlapping area, which accounts for about one-third of the video screen.

[0023] The SURF algorithm is used to extract feature points from the first frame of the corrected video image and generate corresponding feature descriptors;

[0024] The nearest neighbor algorithm is used to roughly match the feature points of adjacent corrected images, and then the RANSAC algorithm is used to purify the matched feature point pairs, and finally the homography transformation matrix between the two adjacent images is obtained;

[0025] First, the intrinsic parameter matrix and rotation matrix of each eyepiece lens are estimated according to the obtained homography transformation matrix, and then the precise intrinsic parameter matrix and rotation matrix of the lens are obtained and saved by using the bundle adjustment method;

[0026] Finally, according to the intrinsic parameter matrix and rotation matrix of the lens, the first frame of the corrected video image is first forward spherically mapped and then reversely mapped to the plane where the reference image is located to obtain the projection mapping matrix and save it.

[0027] The optimal seam line algorithm based on dynamic programming is specifically manifested as finding an optimal seam line in the overlapping area of ​​two adjacent images, where the pixels on the seam line are most similar to each other. In order to search for the optimal seam line, a similarity function e(i, j) is first defined to measure the similarity of grayscale, gradient and texture intensity between pixels. The smaller e(i, j) is, the more similar the two images are at the pixel position:

[0028]

[0029] where ξ I (i,j) and It represents the intensity difference and gradient difference of the same pixel position I1(i,j) and I2(i,j) in the overlapping area. w1 and w2 are the weights of the intensity difference and gradient difference, and w1+w2=1. Since human vision is more sensitive to changes in intensity than the intensity itself, the weight w1 is taken as 0.3 and the weight w2 is taken as 0.7.

[0030] Searching for the best stitching line specifically includes: each pixel in the last row of the overlapping area of ​​the image is the end point of a stitching line, let E(x, y) represent the cumulative similarity measure from the stitching line starting point to the pixel (x, y) along the stitching line, and restrict each row of the overlapping area to only one pixel as the point on the final best stitching line. Therefore, the optimization problem has an optimal substructure, and a dynamic programming algorithm can be used to calculate the best stitching line of the overlapping area. The iterative equation is as follows:

[0031]

[0032] Among them, mask is the binary foreground image obtained after the MOG background is eliminated in the overlapping area of ​​the video image, L is the detection range of each row, and the L value is 1.

[0033] The specific method for dynamically updating the optimal stitching line of the image obtained by Gaussian blurring and downsampling the overlapping area of ​​each frame of image in step 9 is:

[0034] 1) Initialization: Each pixel in the last row of the overlapping area corresponds to the end point of a stitching line. The similarity values ​​of the pixels in the first row and columns of the overlapping area are calculated, and the corresponding cumulative similarity measurement values ​​of each position are initialized to the similarity values ​​of each point.

[0035] 2) Expand downward: After the calculation of the current row is completed, expand downward to calculate the cumulative similarity measure of the stitching line until the last row; specifically, the expansion method is to compare the cumulative similarity measure values ​​of the three pixel points in the upper left, vertically above, and upper right of the previous row of the current point in each row, connect the point with the smallest value with the current point, record the propagation direction to the current point, and calculate the cumulative similarity measure value of the current point. In this way, the cumulative similarity measure values ​​of all pixel points in the current row are calculated.

[0036] 3) Backtracking to select the best stitching line: After the expansion is completed, traverse all the cumulative similarity measurement values ​​​​of the last row and find the minimum value. The corresponding pixel point is the end point of the best stitching line. Then, backtrack according to the propagation direction recorded by each pixel until the first row of the overlapping area, and the best stitching line can be determined.

[0037] In step 11, based on the searched optimal stitching line, the specific method of using the improved fade-in and fade-out algorithm to fuse the stitched images is as follows:

[0038] 1) Divide the overlapping area: Based on the best stitching line searched in the overlapping area, the overlapping area is divided into three parts. The first part is the area from the left boundary of the overlapping area to the best stitching line, the second part is the best stitching line, and the third part is from the best stitching line to the right boundary of the overlapping area.

[0039] 2) The three parts of the overlapping area are fused separately: the two areas on the left and right of the best stitching line are still fused according to the existing fade-in and fade-out method, and then the searched best stitching line is added to complete the overall image fusion, which can eventually eliminate the ghosting, blur, stitching seams and uneven transition problems in the overlapping area of ​​the stitched image;

[0040] The idea of ​​the fade-in and fade-out fusion algorithm is to take the weighted sum of the pixel values ​​of the overlapping area of ​​the two images as the final pixel value. This method is convenient and fast, but during the fusion process, the image details will be blurred, and when a moving object moves in the overlapping area, ghosting will appear. The improved fade-in and fade-out algorithm overcomes the above shortcomings. The algorithm is based on the dynamic best stitching line. First, the overlapping area is divided into three parts. The first part is the area from the left boundary of the overlapping area to the best stitching line, the second part is the best stitching line, and the third part is from the best stitching line to the right boundary of the overlapping area. Then, the two sides of the best stitching line are still fused according to the existing fade-in and fade-out method, and the best stitching line is added to complete the overall image fusion, which can eventually eliminate ghosting, blur, stitching seams and lighting differences. The formula of the algorithm is:

[0041]

[0042] Among them, I1 represents the pixel point from the reference image to the left boundary of the overlapping area, I1∩I seam Indicates the left boundary of the overlapping area to the optimal stitching line I seam The pixels between seam ∩I2 represents the pixel point from the best stitching line to the right edge of the overlapping area, I2 represents the pixel point from the right edge of the overlapping area to the image to be stitched, d1+d seam1 =1,0<d1<1,d seam2 +d2=1,0<d2<1, the specific calculation formulas of each coefficient are as follows:

[0043]

[0044] Among them, x i Indicates the horizontal coordinate of the current pixel in the overlapping area, x l Indicates the horizontal coordinate of the left boundary point of the overlapping area, x r Indicates the horizontal coordinate of the right boundary point of the overlapping area, x seam Represents the horizontal coordinate of the point on the optimal suture line.

[0045] Beneficial effects: The present invention completes camera calibration, distortion correction, image registration, lens parameter precision and spherical projection transformation in the stitching preprocessing stage, and saves the distortion coordinate transformation matrix, lens precision parameters and spherical projection mapping transformation matrix needed in the stitching process, and proposes a method for updating the best stitching line based on the foreground area where the moving object is located in the video, that is, if the best stitching line obtained in the previous frame passes through the foreground area of ​​the current frame image, then the best stitching line is searched again for the current frame image, otherwise the current frame image continues to use the best stitching line of the previous frame, thereby greatly reducing the processing time of the stitching system. In the stage of finding the best stitching line and image fusion, the improved fade-in and fade-out method is used in combination with the improved best stitching line method of the present invention to eliminate the stitching seam, and the overlapping areas on both sides of the best stitching line are fused using the original fade-in and fade-out algorithm respectively, and then the best stitching line is added to smooth the overlapping area of ​​the stitching image. The method effectively avoids the problems of ghosting, blurring, stitching seams and uneven transition caused by the presence of moving objects and lighting differences, and obtains a seamlessly fused stitching image with good visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a video fusion algorithm based on a dynamic optimal stitching line and an improved fade-in and fade-out method according to the present invention;

[0047] Figure 2 A flowchart of preprocessing a video image according to the present invention;

[0048] Figure 3 The first frame of original video image in each of the five RTMP video streams shot by the fisheye lens;

[0049] Figure 4 This is the result image after the original video image is distorted and projected on a spherical surface;

[0050] Figure 5 The overlapping area image obtained by Gaussian blur downsampling and the foreground image obtained by MOG background removal;

[0051] Figure 6 An image with a moving object passing through the best fixed stitching line;

[0052] Figure 7 For Figure 6 The image obtained after the best stitching line in is dynamically updated;

[0053] Figure 8 This is the binocular stitching result diagram obtained when the optimal stitching line is fixed;

[0054] Fig. 9This is the binocular stitching result image obtained by combining the improved fade-in and fade-out method with the dynamic update of the best stitching line;

[0055] Fig.10 To use the algorithm in the present invention Figure 2 The panorama is obtained by stitching five images. DETAILED DESCRIPTION

[0056] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings:

[0057] like Figure 1 As shown, the video fusion algorithm based on the dynamic optimal seam line and the improved fade-in and fade-out method of the present invention, in order to avoid the influence of the moving object passing through the seam line and the different illumination differences, firstly, the background removal algorithm based on the mixture of Gaussians (MOG) is used to remove the background of each frame of the video image to obtain the foreground image. Then, according to the corresponding objective function, the optimal seam line algorithm based on dynamic programming is used to obtain the optimal seam line without the moving object passing through, and finally the improved fade-in and fade-out method is used to eliminate the splicing seams generated by the fusion of the optimal seam line.

[0058] The video stitching system used contains real-time video streams with overlapping areas collected by a multi-eye fisheye camera system, but the type of multi-eye camera system is not limited in actual use; because the fisheye lens has severe radial distortion, each frame of video image needs to be corrected before registration, projection and fusion. The registration algorithm uses the SURF algorithm based on feature points, the projection method selects spherical projection according to the position arrangement of multiple lenses, and the fusion algorithm combines the dynamic optimal stitching line algorithm and the improved fade-in and fade-out algorithm.

[0059] The specific steps are as follows:

[0060] Step 1: Splicing preprocessing

[0061] Step 1.1: Use the standard chessboard to calibrate the camera, obtain the distortion coordinate mapping matrix and save it.

[0062] Step 1.2: Correct the first frame of video image in the video stream captured by each lens according to the distortion coordinate mapping matrix.

[0063] Step 1.3: Use the SURF algorithm to extract feature points from the corrected first frame of video image and generate corresponding feature descriptors.

[0064] Step 1.4: First, use the nearest neighbor algorithm to roughly match the feature points of adjacent rectified images, then use the RANSAC algorithm to purify the matched feature point pairs, and finally calculate the homography transformation matrix between the two adjacent images.

[0065] Step 1.5: First, estimate the intrinsic parameter matrix and rotation matrix of each eyepiece lens based on the obtained homography transformation matrix, and then use the bundle adjustment method to obtain the precise intrinsic parameter matrix and rotation matrix of the lens and save them.

[0066] Step 1.6: According to the intrinsic parameter matrix and rotation matrix of the lens, the first frame of the corrected video image is first forward spherically mapped and then reversely mapped to the plane where the reference image is located to obtain the projection mapping matrix and save it.

[0067] Since each video image is taken by the camera at different angles, they are not on the same projection plane. If the overlapping parts are directly stitched together, the visual consistency of the actual scene will be destroyed. Therefore, it is necessary to perform projection transformation on the image before stitching, that is, to distort the image. Suppose the two-dimensional coordinates of a pixel in the image are (x, y), and its corresponding world coordinates are (X, Y, Z), then the relationship between the two is:

[0068]

[0069] Among them, R is the rotation matrix and K is the intrinsic parameter matrix of the camera.

[0070] According to the distribution of each lens in the multi-camera system, the projection method used in the present invention is spherical projection. Assuming that the two-dimensional coordinates of the pixel point (x, y) after spherical projection mapping are (u, v), the forward mapping formula of spherical projection is:

[0071]

[0072] Among them, s represents the scale, which is proportional to the focal length of the camera. The reverse mapping formula of spherical projection is:

[0073]

[0074]

[0075] Step 2: Find the best suture

[0076] Step 2.1: Distortion correction and projection mapping are performed on each frame of video image in the video stream according to the distortion coordinate mapping matrix and the projection mapping matrix.

[0077] Step 2.2: Find the overlapping area of ​​all video images, and repeat 5*5 Gaussian blur and downsampling for each overlapping area image;

[0078] Since there is a lot of redundancy in the data of a high-resolution image, it takes a lot of time to search for the best seam line in a high-resolution image. Using Gaussian blur and downsampling to obtain an image can effectively reduce the time complexity of searching for the best seam line while ensuring a certain accuracy, and effectively eliminate the impact of noise points on the image. Therefore, in general, the best seam line is searched in a small-scale image that reduces the redundancy of image data, and then the best seam line at the original scale of the image is obtained by linear interpolation. Suppose an original image is I, the image obtained after Gaussian blur and downsampling is G, and the 5*5 Gaussian template is w(m,n), then:

[0079]

[0080] Step 2.3: First, use MOG to remove the background of each overlapping area image after downsampling, and then perform an erosion and then dilation operation on it to finally obtain the foreground area of ​​the current image;

[0081] Step 2.4: Determine whether there are any pixels on the best stitching line searched out in the previous frame image that fall in the foreground area of ​​the current frame. If so, update the best stitching line, that is, recalculate the best stitching line of the current frame image. Otherwise, continue to use the best stitching line of the previous frame image.

[0082] Step 2.5: If the best seam line needs to be updated, the best seam line algorithm based on dynamic programming can be used for search and update.

[0083] Step 2.6: Search for the best seam line in the small-scale image and perform linear interpolation to obtain the best seam line at the scale of the original image.

[0084] The optimal seam line algorithm based on dynamic programming is specifically to find an optimal seam line in the overlapping area of ​​two adjacent images, where the pixels on the seam line are most similar to each other. In order to search for the optimal seam line, a similarity function e(i,j) is usually defined to measure the similarity of grayscale, gradient and texture intensity between pixels. The smaller e(i,j), the more similar the two images are at this pixel position:

[0085]

[0086] where ξ I (i,j) and represents the intensity difference and gradient difference of the same pixel position I1(i,j) and I2(i,j) in the overlapping area, w1 and w2 are the weights of the intensity difference and gradient difference, and w1+w2=1. Since human vision is more sensitive to intensity changes than intensity itself, in the present invention, w1 is taken as 0.3 and w2 is taken as 0.7, which is feasible in most cases.

[0087] Searching for the best stitching line is essentially an optimization problem. Assume that E(x, y) represents the cumulative similarity measure from the stitching line starting point to the pixel point (x, y) along the stitching line. Since the overlapping area is restricted to taking only one pixel point in each row as the point on the final best stitching line, the optimization problem has an optimal substructure. The dynamic programming algorithm can be used to calculate the best stitching line in the overlapping area. The iterative equation is as follows:

[0088]

[0089] Among them, mask is the binary foreground image obtained after the MOG background is eliminated in the overlapping area of ​​the video image, L is the detection range of each row, and the L value is generally 1. The algorithm can be divided into the following three steps:

[0090] 1) Initialization: Each pixel in the last row of the overlapping area corresponds to the end point of a stitching line. The similarity values ​​of the pixels in the first row and columns of the overlapping area are calculated, and the corresponding cumulative similarity measurement values ​​of each position are initialized to the similarity values ​​of each point.

[0091] 2) Expand downward: The calculation of the current row is completed, and the cumulative similarity measurement of the stitching line is expanded downward until the last row. Specifically, the expansion method is to compare the cumulative similarity measurement values ​​of the three pixel points in the upper left, vertically above, and upper right of the previous row of the current point in each row, connect the point with the smallest value with the current point, record the propagation direction to the current point, and calculate the cumulative similarity measurement value of the current point. In this way, the cumulative similarity measurement value of all pixel points in the current row is calculated.

[0092] 3) Backtracking to select the best stitching line: After the expansion is completed, traverse all the cumulative similarity measurement values ​​​​of the last row and find the minimum value. The corresponding pixel point is the end point of the best stitching line. Then, backtrack according to the propagation direction recorded by each pixel until the first row of the overlapping area, and the best stitching line can be determined.

[0093] Step 3: Use the improved fade-in and fade-out method based on dynamic optimal stitching line to fuse the images

[0094] Step 3.1: Divide the overlapping area. Based on the best stitching line searched in the overlapping area, the overlapping area is divided into three parts. The first part is the area from the left boundary of the overlapping area to the best stitching line, the second part is the best stitching line, and the third part is from the best stitching line to the right boundary of the overlapping area.

[0095] Step 3.2: Fuse the three parts of the overlapping area separately. The two areas on the left and right of the best stitching line are still fused according to the existing fade-in and fade-out method, and then the searched best stitching line is added to complete the overall image fusion, which can eventually eliminate the problems of ghosting, blur, stitching seams and uneven transition in the overlapping area of ​​the stitched image.

[0096] The idea of ​​the fade-in and fade-out fusion algorithm is to take the weighted sum of the pixel values ​​of the overlapping area of ​​the two images as the final pixel value. This method is convenient and fast, but during the fusion process, the image details will be blurred, and when a moving object moves in the overlapping area, ghosting will appear. The improved fade-in and fade-out algorithm overcomes the above shortcomings. The algorithm is based on the dynamic best stitching line. First, the overlapping area is divided into three parts. The first part is the area from the left boundary of the overlapping area to the best stitching line, the second part is the best stitching line, and the third part is from the best stitching line to the right boundary of the overlapping area. Then, the two sides of the best stitching line are still fused according to the existing fade-in and fade-out method, and the best stitching line is added to complete the overall image fusion, which can eventually eliminate ghosting, blur, stitching seams and lighting differences. The formula of the algorithm is:

[0097]

[0098] Among them, I1 represents the pixel point from the reference image to the left boundary of the overlapping area, I1∩I seam Indicates the left boundary of the overlapping area to the optimal stitching line I seam The pixels between seam ∩I2 represents the pixel point from the best stitching line to the right edge of the overlapping area, I2 represents the pixel point from the right edge of the overlapping area to the image to be stitched, d1+d seam1 =1,0<d1<1,d seam2 +d2=1,0<d2<1, the specific calculation formulas of each coefficient are as follows:

[0099]

[0100] Among them, x i Indicates the horizontal coordinate of the current pixel in the overlapping area, x l Indicates the horizontal coordinate of the left boundary point of the overlapping area, x r Indicates the horizontal coordinate of the right boundary point of the overlapping area, x seam Represents the horizontal coordinate of the point on the optimal suture line.

[0101] Embodiment 1: Real-time stitching of five video streams with a resolution of 3840*2160 captured by multi-eye fisheye cameras.

[0102] Since the fisheye lens has severe radial distortion, each frame of the video image needs to be corrected before being registered, projected, and fused. In the present invention, in order to avoid the impact caused by the moving objects passing through the seam line and different lighting differences, a video fusion algorithm based on a dynamic optimal seam line and an improved fade-in and fade-out method is proposed. First, the background removal algorithm based on a mixed Gaussian (MOG) is used to remove the background of each frame of the video image to obtain a foreground image. Then, according to the corresponding objective function, the optimal seam line algorithm based on dynamic programming is used to obtain the optimal seam line without moving objects passing through. Finally, the improved fade-in and fade-out method is used to eliminate the stitching seams caused by the fusion of the optimal seam line. Figure 1 The specific implementation steps of the real-time splicing system shown are:

[0103] 1) Preprocessing of stitching, including camera calibration, distortion correction, image registration, camera parameter estimation and spherical projection transformation, obtain the distortion coordinate transformation matrix, the camera precise parameters and the spherical projection mapping transformation matrix and save them. The specific process of the preprocessing module is shown in Figure 2 The video image used in preprocessing is only the first frame image in each video stream, such as Figure 3 (a), (b) and (c) of

[0104] 2) According to the parameters saved in step 1), each frame image in each video stream is subjected to distortion correction and spherical projection mapping, and the obtained results are as follows: Figure 4 shown.

[0105] 3) Perform MOG background removal on the overlapping area image after Gaussian blur and downsampling to obtain the segmented foreground image, such as Figure 5 as shown in (a) and (b).

[0106] 4) Determine whether there are any pixels on the best stitching line searched out in the previous frame image that fall in the foreground area of ​​the current frame. If so, update the best stitching line, that is, recalculate the best stitching line of the current frame image. Otherwise, continue to use the best stitching line of the previous frame image.

[0107] 5) If the best seam line needs to be updated, the best seam line algorithm based on dynamic programming can be used for search and update.

[0108] 6) The best stitching line is searched in the small-scale image and linear interpolation is performed to obtain the best stitching line at the original scale of the image.

[0109] Figure 6 For images with moving objects passing through the fixed optimal stitching line, Figure 7 For Figure 6 The image obtained after the best seam line is dynamically updated shows that the best seam line avoids moving objects very well.

[0110] 7) Based on the found best stitching line, the stitched images are fused using an improved fade-in and fade-out fusion algorithm.

[0111] The idea of ​​the fade-in and fade-out fusion algorithm is to use the weighted sum of the pixel values ​​of the overlapping area of ​​the two images as the pixel value of the final stitched image. This method is convenient and fast, but during the fusion process, the image details will be blurred, and when a moving object moves in the overlapping area, ghosting will appear. The improved fade-in and fade-out algorithm overcomes the above shortcomings. The algorithm is based on the dynamic optimal stitching line. First, the overlapping area is divided into three parts. The first part is the area from the left boundary of the overlapping area to the optimal stitching line, the second part is the optimal stitching line, and the third part is from the optimal stitching line to the right boundary of the overlapping area. Then, the two sides of the optimal stitching line are still fused according to the existing fade-in and fade-out method, and the optimal stitching line is added to complete the overall image fusion, which can eventually eliminate ghosting, blur, stitching seams and lighting differences. The formula of the algorithm is:

[0112]

[0113] Among them, I1 represents the pixel point from the reference image to the left boundary of the overlapping area, I1∩I seam Indicates the left boundary of the overlapping area to the optimal stitching line I seam The pixels between seam ∩I2 represents the pixel point from the best stitching line to the right edge of the overlapping area, I2 represents the pixel point from the right edge of the overlapping area to the image to be stitched, d1+d seam1 =1,0<d1<1,d seam2 +d2=1,0<d2<1, the specific calculation formulas of each coefficient are as follows:

[0114]

[0115] Among them, x i Indicates the horizontal coordinate of the current pixel in the overlapping area, x l Indicates the horizontal coordinate of the left boundary point of the overlapping area, x r Indicates the horizontal coordinate of the right boundary point of the overlapping area, x seam Represents the horizontal coordinate of the point on the optimal suture line.

[0116] The binocular stitching result obtained with the suture line fixed is shown in the figure below. Figure 8 The binocular stitching result obtained by dynamically updating the optimal stitching line and combining the improved fade-in and fade-out method is shown in the figure below. Fig. 9 It can be clearly seen that the algorithm in the present invention has a better splicing effect. Fig.10 To use the algorithm in the present invention Figure 2 The panorama is obtained by stitching five images.

Claims

1. A video fusion algorithm based on dynamic optimal stitching line and improved fade-in and fade-out method, Features: First, a multi-camera system with fixed positions is used to collect video information at the same time to obtain multiple video streams. The video images in the multiple video streams are synchronized, decoded, distorted, aligned, projected, searched for seams, and fused to integrate the multiple video streams into a real-time panoramic video stream with a large field of view and high resolution. The specific steps are as follows: Step 1: Use a multi-camera system to shoot a video containing a high-speed moving object, and calibrate the multi-camera system to obtain the distortion coordinate mapping matrix of the camera lens; Step 2: Correct the first frame of each video stream collected by the multi-camera system according to the distortion coordinate mapping matrix and perform image registration, and then combine the bundle adjustment method to obtain the precise intrinsic parameter matrix and rotation matrix of each eyepiece lens; Step 3: Select the imaging plane of one of the lenses of the multi-eye camera system as the reference plane, and perform forward spherical mapping and then reverse mapping to the reference plane on the corrected first frame of video image according to the intrinsic parameter matrix and rotation matrix of each eyepiece lens to obtain the projection mapping matrix; Since each video image is taken by the camera at different angles, they are not on the same projection plane. If the overlapping parts are directly stitched together, the visual consistency of the actual scene will be destroyed. Therefore, it is necessary to perform projection transformation on the image before stitching, that is, to distort the image. Suppose the two-dimensional coordinates of a pixel in the image are (x, y), and its corresponding world coordinates are (X, Y, Z), then the relationship between the two is: Among them, R is the rotation matrix, K is the intrinsic parameter matrix of the camera; Step 4: Perform distortion correction and spherical projection mapping on each subsequent frame image in the video stream according to the distortion coordinate mapping matrix and the projection mapping matrix; According to the distribution of each lens in the multi-camera system, the projection method used is spherical projection; suppose the two-dimensional coordinates of the pixel point (x, y) after spherical projection mapping are (u, v), then the forward mapping formula of spherical projection is: Among them, s represents the scale, which is proportional to the focal length of the camera; the reverse mapping formula of spherical projection is: Step 5: Calculate the overlapping area between the images taken by adjacent camera lenses in the multi-camera system using the homography transformation matrix between adjacent images; Step 6: Gaussian blur and downsample the overlapping area images, that is, reduce the amount of calculation by reducing the image scale; Step 7: Perform background removal based on the mixed Gaussian model MOG on the overlapping area image after Gaussian blur and downsampling, so as to obtain the segmented foreground image; Step 8: Calculate the best stitching line of the overlapping area image of the first frame using the best stitching line algorithm. For each subsequent frame, determine whether it is necessary to re-search the best stitching line based on the foreground image. Determine whether it is necessary to re-search by determining whether there are pixels on the stitching line falling in the foreground image area of ​​the current frame. If so, recalculate the best stitching line of the current frame and update it. Otherwise, continue to use the best stitching line of the previous frame. According to the distortion coordinate mapping matrix and the projection mapping matrix, each frame of the video image in the video stream is subjected to distortion correction and projection mapping respectively; the overlapping area of ​​all video images is obtained, and each overlapping area image is repeatedly subjected to 5*5 Gaussian blur and downsampling; since there is a lot of redundancy in the data of a high-resolution image, it takes a lot of time to search for the best stitching line in the high-resolution image; first search for the best stitching line in a small-scale image that reduces the redundancy of the image data, and then obtain the best stitching line in the original scale of the image through linear interpolation; Suppose an original image is I, the image obtained after Gaussian blur and downsampling is G, and the 5*5 Gaussian template is w(m,n), then we have: First, MOG is used to remove the background of each overlapping area image after downsampling, and then the image is corroded and then expanded to finally obtain the foreground area of ​​the current image; Determine whether there are any pixels on the best stitching line searched out in the previous frame image that fall in the foreground area of ​​the current frame. If so, update the best stitching line, that is, recalculate the best stitching line of the current frame image, otherwise continue to use the best stitching line of the previous frame image; if the best stitching line needs to be updated, use the best stitching line algorithm based on dynamic programming to search and update; search for the best stitching line in the small-scale image and perform linear interpolation to obtain the best stitching line at the scale of the original image; The optimal seam line algorithm based on dynamic programming is specifically to find an optimal seam line in the overlapping area of ​​two adjacent images, where the pixels on the seam line are most similar to each other. In order to search for the optimal seam line, a similarity function e(i, j) is defined to measure the similarity of grayscale, gradient and texture intensity between pixels. The smaller e(i, j), the more similar the two images are at the pixel position: where ξ I (i,j) and Indicates the same pixel position I in the overlapping area 1 (i, j) and I 2 The intensity difference and gradient difference of (i,j), w 1 and w 2 is the weight of intensity difference and gradient difference, and w 1 +w 2 =1; w 1 Take it as 0.3, w 2 Take it as 0.7; Let E(x, y) represent the cumulative similarity measure along the stitching line from the stitching line starting point to the pixel point (x, y). Since the overlap area is restricted to only one pixel point in each row as the point on the final optimal stitching line, the optimization problem has an optimal substructure. The dynamic programming algorithm is used to calculate the optimal stitching line of the overlap area. The iterative equation is as follows: Step 9: If the best seam line is updated, the best seam line algorithm based on dynamic programming is used to search and update the image at a small scale; Step 10: Search for the best seam line in the small-scale image and perform linear interpolation to obtain the best seam line at the original scale; Step 11: Based on the best stitching line found, an improved fade-in and fade-out fusion algorithm is used to fuse multiple images acquired by the multi-camera system at the same time, and finally a whole fused panoramic image is obtained; Based on the searched optimal stitching line, the specific method of using the improved fade-in and fade-out algorithm to fuse the stitched images is as follows: 1) Divide the overlapping area: Based on the best stitching line searched in the overlapping area, the overlapping area is divided into three parts. The first part is the area from the left boundary of the overlapping area to the best stitching line, the second part is the best stitching line, and the third part is from the best stitching line to the right boundary of the overlapping area. 2) The three parts of the overlapping area are fused separately: the two areas on the left and right of the best stitching line are still fused according to the existing fade-in and fade-out method, and then the searched best stitching line is added to complete the overall image fusion, which can eventually eliminate the ghosting, blur, stitching seams and uneven transition problems in the overlapping area of ​​the stitched image; The fade-in and fade-out fusion algorithm is based on the dynamic optimal stitching line. The overlapping area is divided into three parts. The first part is the area from the left boundary of the overlapping area to the optimal stitching line, the second part is the optimal stitching line, and the third part is from the optimal stitching line to the right boundary of the overlapping area. The two sides of the optimal stitching line are still fused according to the existing fade-in and fade-out method, and the overall image fusion is completed with the optimal stitching line, which can eventually eliminate ghosting, blur, stitching seams and lighting differences. The formula of the algorithm is: Among them, I 1 Represents the pixel point from the reference image to the left boundary of the overlapping area, I 1 ∩I seam Indicates the left boundary of the overlapping area to the optimal stitching line I seam The pixels between seam ∩I 2 Represents the pixel points between the best stitching line and the right boundary of the overlapping area, I 2 Indicates the pixel from the right edge of the overlapping area to the image to be stitched, d 1 +d seam1 =1,0 <d 1 <1,d seam2 +d 2 =1,0 <d 2 <1, the specific calculation formulas of each coefficient are as follows: Among them, x i Indicates the horizontal coordinate of the current pixel in the overlapping area, x l Indicates the horizontal coordinate of the left boundary point of the overlapping area, x r Indicates the horizontal coordinate of the right boundary point of the overlapping area, x seam Represents the horizontal coordinate of the point on the optimal suture line.

2. The video fusion algorithm based on dynamic optimal stitching line and improved fade-in and fade-out method according to claim 1, Features: The multi-eye camera system is the Insta360 Pro2 multi-eye camera system, which is spherical in shape. Six fisheye lenses with the same optical properties are evenly distributed on the same horizontal plane around the body. The field of view is much wider than that of the human eye. The videos captured between adjacent lenses have a certain overlapping area, which accounts for about one-third of the video screen.

3. The video fusion algorithm based on dynamic optimal stitching line and improved fade-in and fade-out method according to claim 1, Features: The SURF algorithm is used to extract feature points from the first frame of the corrected video image and generate corresponding feature descriptors; The nearest neighbor algorithm is used to roughly match the feature points of adjacent corrected images, and then the RANSAC algorithm is used to purify the matched feature point pairs, and finally the homography transformation matrix between the two adjacent images is obtained; First, the intrinsic parameter matrix and rotation matrix of each eyepiece lens are estimated according to the obtained homography transformation matrix, and then the precise intrinsic parameter matrix and rotation matrix of the lens are obtained and saved by using the bundle adjustment method; Finally, according to the intrinsic parameter matrix and rotation matrix of the lens, the first frame of the corrected video image is first forward spherically mapped and then reversely mapped to the plane where the reference image is located to obtain the projection mapping matrix and save it.

4. The video fusion algorithm based on dynamic optimal stitching line and improved fade-in and fade-out method according to claim 1, Features The specific method for dynamically updating the optimal stitching line of the image obtained by Gaussian blurring and downsampling the overlapping area of ​​each frame of image in step 9 is: 1) Initialization: Each pixel in the last row of the overlapping area corresponds to the end point of a stitching line. The similarity values ​​of the pixels in each column of the first row of the overlapping area are calculated, and the accumulated similarity values ​​of the corresponding positions are initialized to the similarity values ​​of each point; 2) Expand downward: After the calculation of the current row is completed, expand downward to calculate the cumulative similarity measure of the stitching line until the last row; specifically, the expansion method is to compare the cumulative similarity measure values ​​of the three pixel points in the upper left, vertically above, and upper right of the previous row of the current point in each row, connect the point with the smallest value with the current point, record the propagation direction to the current point and calculate the cumulative similarity measure value of the current point; in this way, the cumulative similarity measure values ​​of all pixel points in the current row are calculated; 3) Backtracking to select the best stitching line: After the expansion is completed, traverse all the cumulative similarity measurement values ​​​​of the last row and find the minimum value. The corresponding pixel point is the end point of the best stitching line. Then, backtrack according to the propagation direction recorded by each pixel until the first row of the overlapping area, and the best stitching line can be determined.