Regionalized symmetric weighted multi-camera video stitching method

Abstract

The invention relates to the technical field of computer vision, and discloses a multi-camera video stitching method based on regionalized symmetric weighting. The method comprises the following steps: realizing multi-camera frame-level synchronous acquisition (time sequence jitter is less than or equal to 10ms) through an event-driven multi-thread mechanism; obtaining distortion parameters (k1, k2 and k3) by adopting a Zhang Zhengyou calibration method, and correcting the image in real time according to a nonlinear model; dynamically constructing a fusion region: firstly expanding pixels of an overlapping region We belonging to [100, 200] to compensate a registration error, then generating an optimal suture line through a graph cutting method, forming a suture line region through morphological expansion, and finally dividing the optimal suture line region into a suture line region and a transition region; during partition fusion, an image source with the minimum pixel difference is directly selected in a suture line area to eliminate ghosts, and a dynamic boundary weight strategy is adopted in a transition area to achieve seamless transition. Finally, two fusion problems of ghosting elimination and splicing seam elimination are realized based on the two, image transition becomes smoother and more natural, and the method is suitable for panoramic security and protection monitoring and the like in industrial complex scenes such as wind power and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a multi-camera video real-time splicing method, which is especially suitable for eliminating ghosting, color difference and motion blur problems in panoramic monitoring systems. BACKGROUND

[0002] In today's society, with the rapid development of various industries, people's demand for monitoring in some specific occasions is increasing. According to the traditional way, only relying on manual monitoring can no longer meet people's needs, and video monitoring has gradually become the first choice. However, most of the video monitoring cameras are fixed angle, and can only obtain limited field of view. Because of the limited focal length, the video captured by the wide-angle fisheye camera is severely distorted, which will have a great negative impact on monitoring. In this case, video splicing technology has a place to play and can fully play its role.

[0003] In the existing multi-camera video splicing technology, the traditional method has significant defects: the weighted fusion method is sensitive to registration accuracy and is easy to produce "ghosting" (double image) of moving objects in the overlapping area; the best seam line method can eliminate ghosting, but will introduce visible seams; the multi-resolution fusion method has high computational complexity, is difficult to process in real time, and cannot solve the problem of motion blur. In addition, the lack of synchronization between multiple cameras leads to time sequence misplacement, further exacerbating the distortion of the picture. The root cause of the problem is that the existing technology has three limitations: (1) the overlapping area is not processed differently according to the structural features of the image (such as edge area and smooth area); (2) the cumulative effect of registration error at the fusion boundary is ignored; (3) there is no dynamic fusion mechanism for moving objects, making it difficult to eliminate ghosting, color difference and blur. SUMMARY

[0004] The purpose of the present application is to solve the four defects of ghosting, seams, motion blur and time sequence misplacement in the prior art. The present application provides a multi-camera video splicing method, which realizes the generation of panoramic video without ghosting and seamless transition through a regional fusion mechanism and a dynamic boundary weight strategy.

[0005] To achieve the above purpose, the core technical scheme of the present application is as follows:

[0006] Step 1: camera distortion correction. Based on Zhang Zhengyou's calibration method, the radial distortion parameters (k1, k2, k3) of the camera are obtained, and the image is corrected in real time through a nonlinear distortion model:

[0007]

[0008] is the ideal coordinate value of the image point under the linear model, (x, y) is the actual image coordinate value, r 2It is the square of the radius from the imaging point to the center point.

[0009] Step 2: Synchronous acquisition from multiple cameras. An event-driven multi-threaded mechanism is adopted, allocating threads of the same priority to each camera, and triggering frame-level synchronization through signals, with timing jitter ≤10ms.

[0010] Step 3: Dynamic construction of video frame fusion region. ① Overlap area expansion: After calculating the initial overlap width W, expand W to both sides. e (W e ∈[100, 200]) pixels, forming an extended buffer band; ② Suture line generation: the path with the smallest pixel difference in the overlapping area is calculated using the graph cut method as the initial suture line. An expansion operation is performed on the suture line, with an expansion kernel size of 5×5 rectangular structuring elements, generating the suture line region [X... sl ,X sr Other regional divisions are shown in the table below.

[0011]

[0012]

[0013] Step 4: Regionalized symmetric weighted fusion, outputting stitched video frames. For seam line region fusion, directly select the image source with the smallest pixel difference for output, obtaining the initial image source C(x,y).

[0014]

[0015] The threshold is the average pixel difference value in the suture region. Then, symmetrical weighted fusion is performed to obtain the fused image of the suture region.

[0016]

[0017] The two images to be merged are L(x,y) on the left and R(x,y) on the right. The fusion of the transition region adopts symmetric dynamic weighted fusion.

[0018]

[0019] Further, the final fused image is obtained and output.

[0020]

[0021] Step 5: After fusing the multi-camera output video frame by frame according to step 4, output the fused image. Repeat step 4 to obtain the fused video stream.

[0022] Beneficial effects

[0023] This patent provides a regionalized symmetrical weighted multi-view camera video stitching method, which has the following advantages:

[0024] 1. Regionalized fusion mechanism: The fusion area is divided into a suture line area (hard selection eliminates ghosting) and a transition area (symmetrical weighting achieves smoothness, with no obvious suture line), solving the problem of ghosting and seam coexistence.

[0025] 2. Dynamic boundary compensation: Compensates for registration errors by expanding the width of the overlap area.

[0026] 3. Enhanced Synchronization: The event-driven multithreading mechanism controls timing jitter within 10ms, preventing misalignment of moving objects. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of this patent;

[0028] Figure 2 This is a schematic diagram of the regionalization of the method of this patent;

[0029] Figure 3 This is a schematic diagram illustrating the principle of symmetric weighted fusion of the method described in this patent.

[0030] Figure 4 This patented method was tested by capturing images using a multi-view camera.

[0031] Figure 5 This is the image after fusion of the experimental results using the method described in this patent; Detailed Implementation

[0032] The specific embodiments of this patent will be further described below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, this patent provides a regionalized symmetric weighted multi-camera video stitching method, the algorithm flow of which is as follows: 1) First, the image to be fused is projected onto a cylindrical surface; 2) Then, the overlapping area is calculated and expanded; 3) Next, the stitch line is estimated using the optimal stitch line method, and the stitch line region is obtained using the dilation algorithm based on the generated mask; 4) Finally, the stitch line region fusion image is obtained by fusing with the optimized symmetric weighted algorithm, and the transition region is fused using this image.

[0034] Video image fusion aims to stitch together overlapping and registered video frames into a seamless, naturally transitioning panoramic video image. Many external and internal factors exist during video image fusion, such as registration errors, the presence of "ghosting" seams during fusion, moving objects crossing overlapping areas, differences in brightness and exposure, color differences, and shadows at edges and corners. These factors all affect the final quality of the fused video. Therefore, this paper proposes a regionalized weighted fusion image stitching algorithm. Regionalization involves extracting and comparing feature points from the images to be stitched to obtain the overlapping areas. Weighting coefficients are then applied to the pixel values ​​within these overlapping areas to address issues like "ghosting" seams and color differences, resulting in a seamless and more natural-looking stitched image.

[0035] As can be seen from the image stitching process, estimating the transformation model involves calculating the homography matrix to perform image transformation. After calculating the homography matrix of the transformation relationship between images, the images to be fused are projected onto the coordinate system of the reference image. The size of the overlapping region can be calculated based on the vertex coordinates between the images, and then the overlapping region is cropped to obtain highly consistent images.

[0036] Since the homography matrix is ​​calculated using the camera's intrinsic and extrinsic parameters, errors inevitably occur during the calculation process using methods such as the checkerboard method, leading to inaccuracies in the calculated homography matrix and consequently, inaccurate calculations of the overlapping region. To reduce these errors, the overlapping region needs to be expanded. Furthermore, seam lines typically appear within the overlapping region. After estimating the seam lines, a separate area (the seam line region) can be designated for separately fusing the seam, thus eliminating obvious seam issues. The remaining portion within the overlapping region is defined as the transition region.

[0037] The overlapping region is calculated based on the vertex coordinates of the image, and then expanded by adding W to both sides after calculating the width of the overlapping region. e Width threshold, W e Typically, 100 to 200 pixel values ​​are used, and the specific values ​​can be determined based on the experimental results of the stitching scene.

[0038] The calculation of the suture line area can be obtained by using the dilation algorithm. The specific steps are as follows: (1) Use the mask method to find the suture line. The boundary of the mask after obtaining the suture line is the suture line. (2) Expand the mask by the dilation algorithm. (3) Then perform an AND operation between the mask of the mapping transformation diagram and the expanded mask. Therefore, the expanded area is limited to both sides of the suture line, and other boundaries are not affected.

[0039] The expanded overlapping area and the obtained suture line area are collectively referred to as the fusion region. The constructed fusion region is as follows:Figure 2 As shown in the figure. The green area in the figure is the suture line area, and its boundary is X. sl X sr The light blue areas on both sides represent the transition zone within the overlapping area; the orange areas on both sides represent the expansion area. The width of the merged region, composed of these three parts, becomes W + 2 * W. e The transition and expansion regions in the image are areas outside the seam line, which are blended into the entire image. Therefore, these two regions are combined to form a new transition region, with the left and right boundary coordinates set to X. l and X r .

[0040] The symmetric weighted average fusion algorithm primarily adds weight coefficients to pixel values ​​within overlapping image regions. It not only fuses pixel information from both ends of the overlapping region but also considers the positional relationships of pixels within the region. These coordinate relationships are linked to the weight calculation, ensuring that each pixel value has a different weight. For example... Figure 3 The diagram shown illustrates the principle of symmetric weighted fusion.

[0041] To verify the algorithm's practicality, four images of a certain area were acquired, each with a resolution of 1080*1920. The original acquired images are shown below. Figure 4 As shown.

[0042] First, the images are preprocessed according to the distortion correction parameters of each camera. Then, an algorithmic fusion process is performed to obtain a fused image. Using OpenCV vision library functions, five fusion algorithms—the method described in this patent, the symmetric weighted algorithm, the multi-resolution algorithm, the optimal seam line + symmetric weighted algorithm, and the optimal seam line + multi-resolution algorithm—are compared and analyzed. The fusion results are as follows: Figure 5 As shown.

[0043] The quality of image fusion can be evaluated using more scientific quantitative indicators. While the former method is simple and intuitive, it is inevitably affected by subjective factors of individuals; the latter objective analysis method is more convincing. Since the research objective of this paper is to stitch and fuse panoramic video images, there is no ideal reference image for comparison, so indicators that do not require a reference image can be used. The above fusion effect image is compared and analyzed from the following four aspects: (1) Image information entropy, which represents the richness of information contained in the image. The larger the value, the more useful information there is; (2) Image average gradient, which represents the clarity of the image. The larger the value, the clearer the image; (3) Image grayscale mean, which represents the brightness of the image. The larger the value, the brighter and more uniform the image; (4) Image standard deviation, which represents the dispersion of image pixels. The larger the value, the better the quality.

[0044] The effect of the above algorithms combined is objectively and quantitatively evaluated, and the calculated index results are shown in the table below.

[0045]

[0046] As can be seen from the data in the table above, the method described in this patent shows improvements over other algorithms in image information entropy, average gradient value, and mean grayscale value. This indicates that the image fused by this patent method contains more useful information and is clearer. Because the image undergoes exposure processing, the brightness of darker areas is compensated, resulting in an overall brighter image, which is also evidenced by the increased mean grayscale value, comparable to the self-resolved brightness differences achieved by multi-resolution methods. While the overall quality of the fused image is better, the improved symmetric weighting algorithm introduces errors in pixel differentiation, resulting in a slightly lower standard deviation compared to the optimal stitching line + symmetric weighting algorithm. Considering all objective evaluation indicators, the overall effect of the fused image is basically consistent with the subjective evaluation results, thus this patent method represents a significant improvement in image fusion.

[0047] This patent provides a regionalized symmetric weighted multi-camera video stitching method, proposing a regionalized symmetric weighted fusion algorithm to solve problems such as registration errors, ghosting at the stitching seams, and brightness / exposure differences in the image fusion process. Through experimental verification, image fusion was performed on four sets of real-world images, and the effects of five fusion algorithms were compared. This patented method improves the weighted fusion algorithm according to different regions to form a new fusion algorithm, using different boundary calculation weights for the stitching seams and remaining transition areas for separate fusion. This allows for the fusion of the stitching lines individually, eliminating obvious seam problems; the optimal stitching line is found using the graph cut method to eliminate ghosting. Finally, based on these two algorithms, the two major fusion challenges of eliminating ghosting and stitching seams are solved, resulting in smoother and more natural transitions between images. This method is suitable for panoramic security monitoring in complex industrial scenarios such as wind power.

Claims

1. A method for stitching multi-view camera videos using regionalized symmetric weighting, characterized in that... include: (1) Synchronous acquisition: By using an event-driven multi-threading mechanism, each camera is assigned a thread with the same priority to achieve frame-level synchronous acquisition of multiple video streams; (2) Distortion correction: The radial distortion parameters (k1, k2, k3) of the camera are obtained based on Zhang Zhengyou's calibration method, and each frame of the image is corrected in real time according to a nonlinear model. (3) Registration and transformation: Extract feature points of the corrected image, estimate the homography matrix using the RANSAC algorithm, and project adjacent images to the same coordinate system; (4) Construction of the fused region: Calculate the initial overlapping region width W, and expand it to both sides by pixels W. e Form an extended region, with an extended width W. e ∈[100, 200]; The optimal suture line is generated based on the graph cut method, and the suture line region is generated by morphological dilation operation, with its boundary set to X. sl X sr The fusion area is divided into suture line areas (X). sl ≤x<X sr ), left transition region (X) l ≤x<X sl ), right transition region (X) sr ≤x<X r ); (5) Improved Symmetric Weighted Fusion by Regionalization: Outside the fusion region, L(x,y) or R(x,y) is directly output; in the seam line region, the path with the smallest pixel difference is selected to output L(x,y) or R(x,y) to obtain the fused image C(x,y); the transition region is fused using a symmetric weighted algorithm. The overall fusion formula is:

2. The method according to claim 1, characterized in that: The expanded width W obtained in step (4) e The preferred resolution is 150 pixels, used to compensate for homography matrix calculation errors.

3. The method according to claim 1, characterized in that: The generation of the suture region in step (5) specifically includes: calculating the path with the smallest pixel difference in the overlapping area using the graph cut method as the initial suture; performing an expansion operation on the suture, with the expansion kernel size being a 5×5 rectangular structuring element.

4. The method according to claim 1, characterized in that: The weighting coefficients in step (5) satisfy the following constraints: W1+W2=1, W3+W4=1, W5+W6=1, 0<W1,W2,W3,W4,W5,W6<1. x is the x-coordinate of the current pixel. sl and X sr It is the x-coordinate of the left and right boundaries of the suture area. The weight value changes linearly with the distance of the pixel from the boundary, and is symmetrical on both the left and right sides.

5. The method according to claim 1, characterized in that: The event-driven multithreading mechanism in step (1) is as follows: create an independent thread for each camera and set the same priority; trigger the synchronous acquisition command through semaphore, with timing jitter ≤10ms.

Images