Binocular video stitching method based on suture line search

Abstract

The invention discloses a binocular video stitching method based on suture line search, and the method comprises the following steps: S1, video collection and preprocessing: obtaining a video stream through employing a binocular camera, and carrying out the preprocessing of each frame of image; s2, a calibration stage: carrying out feature point extraction and grid construction on each video frame by adopting an improved grid optimization algorithm, and generating calibration data; s3, generating a lookup table: generating the lookup table according to the calibration data, and mapping each pixel to a new position; s4, suture line searching: performing suture line searching by using a suture line searching algorithm based on dynamic programming; s5, motion detection and suture line updating: calculating a gradient difference of a suture line area, judging whether a moving object passes through a seam or not, and if the moving object passes through the seam, updating the suture line; and S6, image fusion: carrying out average fusion on the seam region. According to the method, the improved grid optimization and dynamic programming algorithm is adopted, high-precision binocular video stitching is realized, and motion artifacts and ghosting phenomena are effectively eliminated.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a binocular video stitching method based on seam line search. Background Art

[0002] With the rapid development of computer vision and image processing technology, video stitching technology has been widely used in many fields, such as virtual reality, panoramic video production, monitoring systems and intelligent driving. Video stitching technology aims to seamlessly stitch together video streams from different perspectives to form a continuous and complete video picture, thereby expanding the field of view and improving the visual effect and information content of the video. However, existing video stitching technology still faces many challenges when processing dynamic scenes, especially scenes containing moving objects.

[0003] Traditional static image stitching methods mainly include three steps: registration, seam line search and fusion. Video stitching technology can be regarded as an extension of image stitching. Each video frame can be stitched independently or by a fixed alignment video stitching frame. However, in practical applications, video stitching requires not only high alignment accuracy, but also real-time processing of inter-frame changes during the stitching process, especially when there are moving objects passing through the seams, it is necessary to effectively eliminate motion artifacts and ghosting.

[0004] Existing image stitching technologies mainly use global projection warping and graph cut optimization methods for alignment and stitching. The global projection warping method aligns the image as a whole by performing a global transformation, but it is easy to cause distortion and misalignment in local areas for binocular videos with large parallax. The graph cut optimization method constructs an energy function in the overlapping area of ​​the image to find the optimal stitching line, but the computational complexity is high and it is difficult to meet the requirements of real-time video stitching.

[0005] In response to these problems, the prior art has proposed some improvement methods. For example, the grid optimization-based method can improve the accuracy and robustness of alignment by dividing the image into multiple grid units and aligning them using feature points and grid vertices. However, these methods still have ghosting and artifacts when processing dynamic scenes, especially when moving objects pass through seams, making it difficult to ensure the consistency of the stitching effect.

[0006] In practical applications, another difficulty in video stitching is motion detection and updating of stitching lines. Motion detection is a key step in determining whether there are moving objects in the video frames. Motion artifacts and ghosting phenomena that occur during the stitching process must be effectively detected and processed. Traditional motion detection methods are mainly based on frame difference and background subtraction methods, but these methods are easily affected by noise and lighting changes in complex scenes, and the detection accuracy is not high.

[0007] Therefore, how to provide a binocular video stitching method based on seam line search is a problem that technicians in this field urgently need to solve. Summary of the invention

[0008] One purpose of the present invention is to propose a binocular video stitching method based on seam line search. The present invention adopts an improved grid optimization algorithm and a dynamic programming seam line search algorithm to achieve high-precision, real-time binocular video stitching. By updating seam lines and motion detection in real time, motion artifacts and ghosting phenomena are effectively eliminated, dynamic scenes are adapted, and the consistency and integrity of the stitching results are ensured. The method has the advantages of high precision, high efficiency and broad application prospects.

[0009] A binocular video stitching method based on seam line search according to an embodiment of the present invention comprises the following steps:

[0010] S1, video acquisition and preprocessing: use a binocular camera to obtain the video stream, and preprocess each frame of the image to obtain the left viewpoint video sequence and the right viewpoint video sequence;

[0011] S2, calibration stage: using an improved grid optimization algorithm to extract feature points and construct a grid for each video frame, aligning the grid feature points, and generating calibration data;

[0012] S3, lookup table generation: Generate a lookup table based on the calibration data, map each pixel to a new position, and align the left view image with the right view image;

[0013] S4, seam line search: In the video stitching process, the seam line search algorithm based on dynamic programming is used to search for seams;

[0014] S5, motion detection and seam line update: calculate the gradient difference of the seam line area to determine whether there is a moving object passing through the seam. If there is a moving object passing through the seam, the seam line is updated, otherwise the seam line is not updated;

[0015] S6, image fusion: average fusion is performed on the seam area, and the seams are eliminated by weighted averaging the pixel values ​​of the overlapping area.

[0016] Furthermore, the calibration stage uses a frame difference method to extract the area containing the moving object and the background area outside the moving object, performs feature extraction on the background area, and performs a difference method on the image data of two consecutive frames:

[0017] D(x,y)=|I n+1 (x,y)-I n (x,y)|;

[0018]

[0019] Where D(x,y) represents the difference of image data, I n+1 (x, y) represents the n+1th frame image, I n (x, y) represents the nth frame image, R(x, y) represents the mask of the motion image, and T represents motion detection prediction, and its value is set to 30.

[0020] Furthermore, the S4 specifically includes:

[0021] S41, constructing an energy map, adding the image color difference intensity value and the image structure difference intensity value to obtain the energy value of each pixel;

[0022] S42, calculate the cumulative energy graph, the value of the first row in the cumulative energy graph and the energy Figure 1 Starting from the second row, traverse each row. The accumulated energy value corresponding to each pixel is the sum of the energy value of the current position and the minimum energy value corresponding to the three pixel positions in the previous row.

[0023] S43. Find the seam line corresponding to the minimum cumulative energy value, find the minimum energy value and the corresponding column index in the last row, traverse each row from the second to last row, find the recorded column index and compare it with the cumulative energy values ​​of the two adjacent pixels on the left and right, record the column index corresponding to the minimum energy value, and after completing the traversal, the stored column index is the optimal seam line position corresponding to the minimum energy map.

[0024] Furthermore, the energy function construction criterion of the optimal suture line is:

[0025] E(x,y)=E color 2 (x,y)+E geometry (x,y);

[0026] Among them, E color Indicates the image color difference intensity value, E geometry Indicates the intensity value of image structure difference;

[0027] The image color difference intensity value E color The solution formula is:

[0028] E color =|I1(x,y)-I2(x,y)|;

[0029] Wherein, I1(x, y) represents the image pixel value of the left view in the overlapping area, and I2(x, y) represents the image pixel value of the right view in the overlapping area;

[0030] The Sobel operator is introduced to calculate the gradient information of a pixel in the horizontal and vertical directions. The convolution template is as follows:

[0031]

[0032]

[0033] The image structure difference intensity value E geometry The solution formula is:

[0034] E geometry (x,y)=[S x (I1(x,y)-I2(x,y))] 2 +

[0035] [S y (I1(x,y)-I2(x,y))] 2 ;

[0036] Among them, S x and S y Represent the 3×3 Sobel operator template in the x and y directions respectively.

[0037] Furthermore, the motion detection specifically includes:

[0038] Determine pixel P i Whether it has changed:

[0039]

[0040] in, Represents the pixel P in the current seam i The original gradient of represents the pixel P in the seam at time t i The gradient of , δ represents a constant;

[0041] If C t If the total number of pixels in is greater than N, there is a new moving object in the overlapping area and the optimal seam should be updated, where N represents a set constant.

[0042] Furthermore, the S6 specifically includes:

[0043] S61, processing the overlapping area of ​​the left viewpoint image and the right viewpoint image to obtain pixel values ​​of the seam area;

[0044] S62, calculating the weighted average of pixels of the left viewpoint image and the right viewpoint image in the overlapping area:

[0045] I fused (x,y)=w1·I1(x,y)+w2·I2(x,y);

[0046] Among them, I fused(x, y) represents the pixel value of the fused image, I1(x, y) represents the pixel value of the left view image, I2(x, y) represents the pixel value of the right view image, w1 represents the weighting coefficient of the left view, w2 represents the weighting coefficient of the right view, and w1+w2=1 is satisfied;

[0047] S63: Calculate the pixel gradient of the seam area to obtain gradient information and

[0048] S64, adjusting the weighting coefficient according to the gradient information so that the weighting coefficient is inversely proportional to the gradient value:

[0049]

[0050]

[0051] S65. Use a Gaussian smoothing filter on the seam area to reduce the severity of pixel value changes, and gradually eliminate the seams through multiple iterative smoothing processes.

[0052] The beneficial effects of the present invention are:

[0053] (1) High-precision image alignment: An improved grid optimization algorithm is used to extract feature points and construct a grid for each video frame. The feature points in the left and right viewpoint images are aligned through the optimized grid to generate calibration data, thereby achieving high-precision image alignment. This method can not only effectively reduce distortion and misalignment in local areas, but also improve the accuracy of overall alignment.

[0054] (2) Strong real-time performance: The present invention generates a lookup table to map each pixel to a new position, aligning the left view image with the right view image, thereby simplifying the subsequent processing steps and improving the real-time performance of video stitching. At the same time, the dynamic programming seam search algorithm significantly reduces the computational complexity compared to the traditional graph cut optimization method, further improving the stitching efficiency.

[0055] (3) Eliminate motion artifacts and ghosting: During the video stitching process, the gradient difference of the seam area is calculated to determine whether there is a moving object passing through the seam. If there is a moving object passing through the seam, the seam is updated, otherwise it is not updated. This strategy based on motion detection and seam line update effectively solves the ghosting problem existing in traditional methods and significantly reduces the artifacts caused by moving objects.

[0056] (4) Adaptive image fusion: By performing average fusion and weighted average processing on the seam area, the weighting coefficient is adaptively adjusted using the image gradient information so that the weighting coefficient is inversely proportional to the gradient value, thereby achieving a smooth transition in the overlapping area and eliminating the stitching seam. A Gaussian smoothing filter is further applied to smooth the seam area to make the stitching result smoother and more natural.

[0057] (5) Adaptation to dynamic scenes: The method of the present invention is particularly suitable for dynamic scenes containing moving objects. By updating the stitching line and motion detection in real time, the breakage caused by the moving objects passing through the seams is effectively avoided, thereby ensuring the visual consistency and integrity of the stitching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0059] Figure 1 A flowchart of a binocular video stitching method based on seam line search proposed by the present invention;

[0060] Figure 2 A schematic diagram of a binocular video stitching framework of a binocular video stitching method based on seam line search proposed by the present invention;

[0061] Figure 3 This is an effect diagram of an outdoor garden stitching result of a binocular video stitching method based on stitching line search proposed by the present invention;

[0062] Figure 4 This is an effect diagram of outdoor high-rise building stitching results of a binocular video stitching method based on stitching line search proposed by the present invention;

[0063] Figure 5 This is an effect diagram of an outdoor road stitching result of a binocular video stitching method based on stitching line search proposed by the present invention;

[0064] Figure 6 This is an effect diagram of an outdoor construction site stitching result of a binocular video stitching method based on stitching line search proposed by the present invention;

[0065] Figure 7 This is an indoor company stitching result effect diagram of a binocular video stitching method based on stitching line search proposed by the present invention;

[0066] Figure 8 A comparison diagram of stitching effects of a moving object not passing through a seam in a binocular video stitching method based on seam line search proposed by the present invention;

[0067] Fig. 9This is a comparison chart of the stitching effects of a moving object passing through a seam in a binocular video stitching method based on seam line search proposed by the present invention. DETAILED DESCRIPTION

[0068] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0069] refer to Figure 1 and Figure 2 , a binocular video stitching method based on seam line search, comprising the following steps:

[0070] S1, video acquisition and preprocessing: use a binocular camera to obtain the video stream, and preprocess each frame of the image to obtain the left viewpoint video sequence and the right viewpoint video sequence;

[0071] S2, calibration stage: using an improved grid optimization algorithm to extract feature points and construct a grid for each video frame, aligning the grid feature points, and generating calibration data;

[0072] S3, lookup table generation: Generate a lookup table based on the calibration data, map each pixel to a new position, and align the left view image with the right view image;

[0073] S4, seam line search: In the video stitching process, the seam line search algorithm based on dynamic programming is used to search for seams;

[0074] S5, motion detection and seam line update: calculate the gradient difference of the seam line area to determine whether there is a moving object passing through the seam. If there is a moving object passing through the seam, the seam line is updated, otherwise the seam line is not updated;

[0075] S6, image fusion: average fusion is performed on the seam area, and the seams are eliminated by weighted averaging the pixel values ​​of the overlapping area.

[0076] In this embodiment, the calibration stage uses the frame difference method to extract the area containing the moving object and the background area outside the moving object, performs feature extraction on the background area, and performs the difference method on the image data of two consecutive frames:

[0077] D(x,y)=|I n+1 (x,y)-I n (x,y)|;

[0078]

[0079] Where D(x,y) represents the difference of image data, I n+1 (x, y) represents the n+1th frame image, In (x, y) represents the nth frame image, R(x, y) represents the mask of the motion image, and T represents motion detection prediction, and its value is set to 30.

[0080] In this implementation manner, the S4 specifically includes:

[0081] S41, constructing an energy map, adding the image color difference intensity value and the image structure difference intensity value to obtain the energy value of each pixel;

[0082] S42, calculate the cumulative energy graph, the value of the first row in the cumulative energy graph and the energy Figure 1 Starting from the second row, traverse each row. The accumulated energy value corresponding to each pixel is the sum of the energy value of the current position and the minimum energy value corresponding to the three pixel positions in the previous row.

[0083] S43. Find the seam line corresponding to the minimum cumulative energy value, find the minimum energy value and the corresponding column index in the last row, traverse each row from the second to last row, find the recorded column index and compare it with the cumulative energy values ​​of the two adjacent pixels on the left and right, record the column index corresponding to the minimum energy value, and after completing the traversal, the stored column index is the optimal seam line position corresponding to the minimum energy map.

[0084] In this embodiment, the energy function construction criterion of the optimal suture line is:

[0085] E(x,y)=E color 2 (x,y)+E geometry (x,y);

[0086] Among them, E color Indicates the image color difference intensity value, E geometry Indicates the intensity value of image structure difference;

[0087] The image color difference intensity value E color The solution formula is:

[0088] E color =|I1(x,y)-I2(x,y)|;

[0089] Wherein, I1(x, y) represents the image pixel value of the left view in the overlapping area, and I2(x, y) represents the image pixel value of the right view in the overlapping area;

[0090] The Sobel operator is introduced to calculate the gradient information of a pixel in the horizontal and vertical directions. The convolution template is as follows:

[0091]

[0092]

[0093] The image structure difference intensity value E geometry The solution formula is:

[0094] E geometry (x,y)=[S x (I1(x,y)-I2(x,y))] 2 +

[0095] [S y (I1(x,y)-I2(x,y))] 2 ;

[0096] Among them, S x and S y Represent the 3×3 Sobel operator template in the x and y directions respectively.

[0097] In this implementation manner, the motion detection specifically includes:

[0098] Determine pixel P i Whether it has changed:

[0099]

[0100] in, Represents the pixel P in the current seam i The original gradient of represents the pixel P in the seam at time t i The gradient of , δ represents a constant;

[0101] If C t If the total number of pixels in is greater than N, there is a new moving object in the overlapping area and the optimal seam should be updated, where N represents a set constant.

[0102] In this implementation manner, S6 specifically includes:

[0103] S61, processing the overlapping area of ​​the left viewpoint image and the right viewpoint image to obtain pixel values ​​of the seam area;

[0104] S62, calculating the weighted average of pixels of the left viewpoint image and the right viewpoint image in the overlapping area:

[0105] I fused (x,y)=w1·I1(x,y)+w2·I2(x,y);

[0106] Among them, I fused(x, y) represents the pixel value of the fused image, I1(x, y) represents the pixel value of the left view image, I2(x, y) represents the pixel value of the right view image, w1 represents the weighting coefficient of the left view, w2 represents the weighting coefficient of the right view, and w1+w2=1 is satisfied;

[0107] S63: Calculate the pixel gradient of the seam area to obtain gradient information and

[0108] S64, adjusting the weighting coefficient according to the gradient information so that the weighting coefficient is inversely proportional to the gradient value:

[0109]

[0110]

[0111] S65. Use a Gaussian smoothing filter on the seam area to reduce the severity of pixel value changes, and gradually eliminate the seams through multiple iterative smoothing processes.

[0112] Embodiment 1:

[0113] In order to verify the effect of the method of the present invention, we conducted simulation experiments on five scenes with parallax captured by the binocular platform. The experimental results are as follows: Figure 3-Figure 7 As shown in Figure 2, although the stitching line search algorithm based on dynamic programming solves the local optimal solution, it can still achieve good stitching results in most scenarios.

[0114] Compared with image stitching, video stitching has higher requirements on time complexity. In order to evaluate the speed of the method of the present invention, Figure 3-Figure 7 The running time of the five scenes is compared with the seam search algorithm based on graph cut optimization, as shown in Table 1. The video stitching framework proposed in the present invention constructs a lookup table in the calibration stage, and warps and aligns other frames through the lookup table, which greatly improves the stitching efficiency. Therefore, the time complexity of the warping stage is not compared, and the speed difference of the seam search part is mainly compared.

[0115] Table 1 Comparison of time complexity of different seam search algorithms

[0116]

[0117] As shown in Table 1, the running time of the seam line search algorithm based on dynamic programming of the present invention is reduced by 79.9%, 70.6%, 86.7%, 81.6% and 92.3% respectively compared with the algorithm based on graph cut optimization. At the same time, the seam line search algorithm based on dynamic programming is conducive to hardware implementation, and real-time video stitching in hardware is also a direction in the future.

[0118] In addition, to prove the effectiveness of the seam update strategy, this embodiment conducts experiments on images of whether the moving object passes through the seam, such as Figure 8 and Fig. 9 As shown in the figure, the results show that the seam updating strategy can avoid the phenomenon of cutting caused by moving objects passing through the seams, and the algorithm is simple and effective.

[0119] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A binocular video stitching method based on seam line search, characterized in that: The steps include: S1, video acquisition and preprocessing: use a binocular camera to obtain the video stream, and preprocess each frame of the image to obtain the left viewpoint video sequence and the right viewpoint video sequence; S2, calibration stage: using an improved grid optimization algorithm to extract feature points and construct a grid for each video frame, aligning the grid feature points, and generating calibration data; S3, lookup table generation: Generate a lookup table based on the calibration data, map each pixel to a new position, and align the left view image with the right view image; S4, seam line search: In the video stitching process, the seam line search algorithm based on dynamic programming is used to search for seams; S5, motion detection and seam line update: calculate the gradient difference of the seam line area to determine whether there is a moving object passing through the seam. If there is a moving object passing through the seam, the seam line is updated, otherwise the seam line is not updated; S6. Image fusion: Perform average fusion on the seam area and eliminate the seams by taking weighted average of the pixel values ​​in the overlapping area.

2. A binocular video stitching method based on seam line search according to claim 1, characterized in that: The calibration stage uses the frame difference method to extract the area containing the moving object and the background area outside the moving object, performs feature extraction on the background area, and performs the difference method on the image data of two consecutive frames: D(x,y)=|I n+1 (x,y)-I n (x,y)|; Where D(x,y) represents the difference of image data, I n+1 (x, y) represents the n+1th frame image, I n (x, y) represents the nth frame image, R(x, y) represents the mask of the motion image, and T represents motion detection prediction, and its value is set to 30.

3. The binocular video stitching method based on seam line search according to claim 1, characterized in that: The S4 specifically includes: S41, constructing an energy map, adding the image color difference intensity value and the image structure difference intensity value to obtain the energy value of each pixel; S42, calculating a cumulative energy map, wherein the value of the first row in the cumulative energy map is consistent with the energy map, and traversing each row starting from the second row, the cumulative energy value corresponding to each pixel is the sum of the energy value at the current position and the minimum energy value corresponding to the three pixel positions in the previous row; S43. Find the seam line corresponding to the minimum cumulative energy value, find the minimum energy value and the corresponding column index in the last row, traverse each row from the second to last row, find the recorded column index and compare it with the cumulative energy values ​​of the two adjacent pixels on the left and right, record the column index corresponding to the minimum energy value, and after completing the traversal, the stored column index is the optimal seam line position corresponding to the minimum energy map.

4. The binocular video stitching method based on seam line search according to claim 3, characterized in that: The energy function construction criterion of the optimal suture line is: E(x,y)=E color 2 (x,y)+E geometry (x,y); Among them, E color Indicates the image color difference intensity value, E geometry Indicates the intensity value of image structure difference; The image color difference intensity value E color The solution formula is: E color =|I1(x,y)-I2(x,y)|; Wherein, I1(x, y) represents the image pixel value of the left view in the overlapping area, and I2(x, y) represents the image pixel value of the right view in the overlapping area; The Sobel operator is introduced to calculate the gradient information of a pixel in the horizontal and vertical directions. The convolution template is as follows: The image structure difference intensity value E geometry The solution formula is: E geometry (x,y)=[S x ·(I1(x,y)-I2(x,y))] 2 +[S y ·(I1(x,y)-I2(x,y))] 2 ; Among them, S x and S y Represent the 3×3 Sobel operator template in the x and y directions respectively.

5. The binocular video stitching method based on seam line search according to claim 1, characterized in that: The motion detection specifically includes: Determine pixel P i Whether it has changed: in, Indicates the pixel P in the current seam i The original gradient of represents the pixel P in the seam at time t i The gradient of , δ represents a constant; If C t If the total number of pixels in is greater than N, there is a new moving object in the overlapping area and the optimal seam should be updated, where N represents a set constant.

6. The binocular video stitching method based on seam line search according to claim 1, characterized in that: The S6 specifically includes: S61, processing the overlapping area of ​​the left viewpoint image and the right viewpoint image to obtain pixel values ​​of the seam area; S62, calculating the weighted average of pixels of the left viewpoint image and the right viewpoint image in the overlapping area: I fused (x,y)=w1·I1(x,y)+w2·I2(x,y); Among them, I fused (x, y) represents the pixel value of the fused image, I1(x, y) represents the pixel value of the left view image, I2(x, y) represents the pixel value of the right view image, w1 represents the weighting coefficient of the left view, w2 represents the weighting coefficient of the right view, and w1+w2=1 is satisfied; S63: Calculate the pixel gradient of the seam area to obtain gradient information and S64, adjusting the weighting coefficient according to the gradient information so that the weighting coefficient is inversely proportional to the gradient value: S65. Use a Gaussian smoothing filter on the seam area to reduce the severity of pixel value changes, and gradually eliminate the seams through multiple iterative smoothing processes.

Images