A panoramic image stitching method and device based on optimal mapping matrix

Abstract

The present invention belongs to the field of video image stitching and fusion, and is specifically a panoramic image stitching method and device based on an optimal mapping matrix. The method comprises: collecting n video streams obtained by simultaneously shooting an airport scene area with n camera devices; each camera device shoots a part of the scene, and the scene shot by each camera device has an overlapping area with the scene shot by the adjacent camera device; n is a positive integer greater than 1; according to the preset distortion coefficients of each camera device, the image frames in the n video streams are subjected to distortion correction; the image frames at the same time after correction are processed using feature descriptors to obtain the optimal mapping matrix between the image frames; the image frames at the time are stitched according to the mapping matrix to obtain a panoramic image of the airport.

Description

Technical Field

[0001] The invention belongs to the field of video image stitching and fusion, and specifically relates to a panoramic image stitching method and device based on an optimal mapping matrix. Technical Background

[0002] In the application of airport video surveillance, ordinary video surveillance only uses independent cameras to monitor a limited field of view. If you want to monitor a large area, you will design several cameras to monitor every certain area, and then gather the monitoring images obtained by all the lenses together. There is a problem here that multiple cameras will monitor the same area, resulting in repeated monitoring images, and additional manpower is needed to view the monitoring images, which not only wastes hardware resources, but also consumes a lot of manpower, time and energy.

[0003] The existing panoramic image stitching algorithms are analyzed: 1) The stitching algorithm based on region correlation is easily affected by brightness and contrast, resulting in the inability to complete stitching and poor robustness; 2) The stitching algorithm based on feature is more robust, but the processing of stitching seams and color differences is not ideal. Summary of the invention

[0004] Purpose of the invention: In view of the shortcomings of the current stitching algorithm, there is an urgent need to study a stitching algorithm with good stitching effect, strong robustness, smooth transition of stitching seams, and no color difference to complete the stitching method of multiple cameras.

[0005] Technical solution:

[0006] A panoramic image stitching method based on an optimal mapping matrix, comprising:

[0007] Collect n video streams obtained by simultaneously shooting the airport scene area by n camera devices; each camera device shoots a part of the scene, and the scene shot by each camera device has an overlapping area with the scene shot by the adjacent camera device; n is a positive integer greater than 1;

[0008] According to the preset distortion coefficients of each camera device, the image frames in the n video streams are subjected to distortion correction;

[0009] The feature descriptor is used to process the corrected image frames at the same time to obtain the optimal mapping matrix between the image frames;

[0010] The image frames at that moment are stitched together according to the mapping matrix to obtain a panoramic image of the airport.

[0011] The adjacent left and right images in the corrected image frames at the same time are processed by feature descriptors to obtain the optimal mapping matrix between the image frames, including:

[0012] The feature points of the overlapping area of ​​the left and right images are extracted based on the sub-pixel technology of bilinear difference. The feature points in the left image are called left feature points, which constitute the left feature point set. The feature points in the right image are called right feature points, which constitute the right feature point set.

[0013] Take m pairs of descriptors from the left and right feature point sets, where a pair of descriptors includes a left feature point and a corresponding right feature point; m is a positive integer;

[0014] Randomly divide m pairs of descriptors into Group, calculate the mapping matrix for each group

[0015] Substitute the first set of mapping matrices into the left feature point set and calculate the new right feature point set;

[0016] Determine whether the distance between the corresponding feature points in the right feature point set and the new right feature point set is greater than one pixel;

[0017] If the proportion of the distance less than one pixel relative to the calculated distance is greater than or equal to the preset proportion, the feature points of the left and right feature point sets corresponding to the distance greater than or equal to one pixel are deleted; the mapping matrix is ​​calculated using the remaining left feature points in the left feature point set and the remaining right feature points in the right feature point set, and the mapping matrix is ​​used as the optimal mapping matrix between the left and right images;

[0018] If the proportion of the distance less than one pixel relative to the distance calculated above is less than or equal to the preset proportion, the two sets of mapping matrices are brought into the left feature point set until the optimal mapping matrix is ​​calculated.

[0019] The method further comprises:

[0020] When the last group also fails to obtain the optimal mapping matrix between the left and right images, m pairs of descriptors are taken from the left and right feature point sets again.

[0021] After the image frames at the time are spliced ​​according to the mapping matrix to obtain a panoramic image of the airport, the method further includes:

[0022] The corrected image frame at the same moment is divided into two areas, upper and lower, according to the horizon;

[0023] These two areas are used together as sampling points to participate in the chromatic aberration correction of the panoramic image.

[0024] The method further comprises:

[0025] According to the different imaging of each camera, the H, S, and V channels of the panoramic image are smoothed separately.

[0026] The specific operations of smoothing are as follows:

[0027] First, calculate the global histogram statistics of the H, S, and V channels of the left and right images, and find the mean of each channel and The difference between the average values ​​of the three channels is calculated, and weighted complement is performed near the stitching seams according to the difference of each channel to make the image smooth.

[0028] Specific weighting strategies include:

[0029] if Then the H channel needs to be weighted, and the weighting coefficient can be adjusted according to the actual situation, otherwise no processing is required; if Then the S channel needs to be weighted, and the weighting coefficient can be adjusted according to the actual situation, otherwise no processing is required; if Then the V channel needs to be weighted, and the weighting coefficient can be adjusted according to the actual situation, otherwise no processing is required.

[0030] A panoramic image stitching device based on an optimal mapping matrix, comprising:

[0031] The acquisition module is used to acquire n video streams obtained by simultaneously shooting the airport scene area by n camera devices; each camera device shoots a part of the scene, and the scene shot by each camera device has an overlapping area with the scene shot by the adjacent camera device; n is a positive integer greater than 1;

[0032] A correction module, used to perform distortion correction on image frames in n video streams according to preset distortion coefficients of various camera devices;

[0033] A processing module, used for processing the corrected image frames at the same time by using feature descriptors to obtain an optimal mapping matrix between the image frames;

[0034] The stitching module is used to stitch the image frames at that moment according to the mapping matrix to obtain a panoramic image of the airport.

[0035] A computer-readable storage medium stores a computer program, which implements the above method when executed by a processor.

[0036] Beneficial effects:

[0037] (1) Perform bilinear interpolation on the feature points of image matching to improve the accuracy of calculating the mapping matrix, making the image matching more accurate and greatly improving the ghosting phenomenon of the target passing through the seam after image splicing;

[0038] (2) Using multiple sets of mapping matrices and corresponding point pairs, by calculating the influence of each mapping matrix on other point pairs, removing the point pairs with greater influence and retaining the point pairs with higher precision, and using these high-precision point pairs to calculate a high-precision mapping matrix. The optimal mapping matrix improves the accuracy of image registration and reduces the complexity of subsequent image fusion;

[0039] (3) In the image fusion stage, instead of directly using the pixel weights of the left and right images as the pixel values ​​at the seam, we first analyze the global HSV three-channel mean of the left and right images, and then determine which values ​​need to be weighted based on the difference between the left and right images. This not only reduces the amount of calculation, but also achieves no color difference after image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flowchart of a panoramic image stitching method applicable to a large area.

[0041] Figure 2 This is the effect diagram of feature point extraction for two images.

[0042] Figure 3 Calculate the exposure gain factor g for the block.

[0043] Figure 4 Calculate the exposure gain factor g for each small block.

[0044] Figure 5 This is the image fusion effect of two images stitched together.

[0045] Figure 6 A panoramic image after stitching two images. DETAILED DESCRIPTION

[0046] Stitching technology is an important branch of computer graphics and computer vision. Its development is based on the stitching technology of static images, that is, by collecting video data streams captured by multiple cameras at the same time, using relevant stitching algorithms to obtain a complete real-time stitching video stream, and then output and forward it. The obtained fused image can greatly improve people's dynamic perception, identification and monitoring capabilities of objects and scenes within a larger viewing range and depth.

[0047] At present, panoramic monitoring systems are widely used in various fields. Compared with general monitoring systems, panoramic monitoring systems have a larger observation field of view and can observe more objects at the same time. Image stitching technology combines the images of some areas of the collected video stream to form a large synthetic image with a wider field of view and higher resolution. Existing image stitching algorithms have good performance in some small scenes, but they often have certain defects in the actual application process of large wide-area scenes, which may cause the stitched images to have shape distortion, ghosting, and unstable network transmission.

[0048] Example 1

[0049] The present invention is implemented by the following technical solutions: Figure 1 As shown:

[0050] (1) Image acquisition is the guarantee for obtaining source video images. The system obtains enough image sequences through the camera device to provide resources for the subsequent processes. At the same time, it is necessary to ensure that the camera devices are of the same model to ensure the consistency of their own parameters. At the same time, there must be overlapping areas between the perspectives of adjacent devices to ensure that adjacent images contain some of the same scenes. After the location is determined, multiple videos are collected through RTSP and then decoded into GPU memory using OpenCV and Cuda. After image preprocessing, including denoising, frame synchronization and other functions, multiple cameras can be synchronized to obtain real-time video streams.

[0051] (2) The captured video image sequence may differ due to external factors of the device or scene, so this step requires lens distortion correction. The distortion mathematical model is as follows:

[0052]

[0053] where δ x (x, y), δ y (x, y) is the distortion coefficient, r is the radius from the imaging point to the center point, and k1, k2, k3 are radial distortion coefficients.

[0054] The present invention estimates the parameters of the camera device by using Zhang Zhengyou's chessboard calibration method. The main process is:

[0055] 1) By constantly changing the position of the device or calibration plate, the calibration images required at different angles can be obtained;

[0056] 2) Detect the corner points of the calibrated image to obtain the corresponding spatial position information;

[0057] 3) The internal parameters and distortion coefficients of the camera device are calculated using the position information obtained in 2).

[0058] (3) Image registration is the core step of video stitching. In essence, image registration is to accurately find the corresponding positions of the overlapping parts of the images, and then determine the transformation relationship between the two images. The quality of registration directly affects the accuracy, speed and output effect of the stitching system. It can effectively handle translation, scaling, rotation, affine and other situations.

[0059] The present invention adopts a feature-based image registration algorithm, which consists of feature point extraction, feature point registration, image transformation and image fusion.

[0060] 1) Feature points include corner points of the image and points of interest that show some singularity relative to their field. This time, SIFT feature point detection is used, and feature points are screened through the RANSAC algorithm. After finding 4 or more feature points, image fusion and splicing can be performed. Figure 2 As shown in the figure. Through geometric projection, images can be well stitched together, but if different images have different exposure levels, then the overlapping parts of the stitched image will also have obvious edges, which makes the image look very unnatural. Therefore, we also need to perform exposure compensation on each image to make all images have the same exposure level.

[0061] Commonly used exposure compensation methods include gain compensation and block compensation. Gain compensation is to assign a gain coefficient to each image to make the image intensity of the overlapping parts equal or similar. It can be achieved using the error function:

[0062]

[0063] In the formula, g i and g j is the gain coefficient of image i and image j, σ n and σ g denote the standard deviation of error and gain, σ N =10 (if the intensity range is 0 to 255), σ g =0.1. ij Represents the average intensity of the overlapping parts of the images.

[0064]

[0065] Where R, G and B represent the intensity values ​​of the red, green and blue components of the color image, respectively, R(i, j) represents the overlapping part of image i and image j, and N ij Represents the number of pixels of the overlapping part R(i, j).

[0066] The traditional method of splicing image exposure correction is to select each image as a sampling source for calculation. If N images are used for splicing, N sampling sources will be used for calculation to obtain n gain coefficients. This method can solve the color difference problem well when the global color difference of the image is not large. Figure 3 shown.

[0067] However, when there is a large difference between the images of the upper and lower scenes in all the stitched images, the effect will be very poor. For example, in the airport scene, the color difference between the sky and the ground is very large. When using this method to correct the color difference, the color difference of the ground scene will be corrected, while the sky scene will still have a large color difference.

[0068] Through the regional chromatic aberration correction method, each image is divided into upper and lower regions, so that these two regions are used as sampling points to participate in the calculation of chromatic aberration correction. N is changed to N*2 to obtain a more accurate g value. At the same time, this also requires higher computing performance, and more g values ​​need to be calculated in more block areas. When the amount of calculation increases, using GPU to optimize the processing speed is the best choice. Figure 4 shown.

[0069] 2) After the feature points are extracted, image matching and fusion are performed. The model is selected according to the scene size. The spherical model is selected for large scenes. Matching is performed according to the corresponding feature point set detected to obtain the corresponding transformation matrix. The multi-channel images are converted into a coordinate system based on the model parameters. Before image fusion, the relationship between the overlapping areas of the two adjacent images is used to establish a histogram mapping table between the two adjacent images. The two images are mapped and transformed as a whole through the mapping table, and finally the overall brightness and color correction consistency is achieved.

[0070] 3) The image mask is calculated by the SeamFinder class in OpenCV. The subsequent frames use the mask and the calculated camera parameters to map the image so that the positions of the overlapping areas of the two adjacent images match during the gluing process. The image fusion effect is as follows: Figure 5 shown.

[0071] 4) GPU parallel processing is used to re-encode, render and output the fused image, so as to complete the real-time video stream after splicing multiple videos. Figure 6 shown.

[0072] Example 2

[0073] The present invention is implemented by the following technical solutions:

[0074] 1) Image acquisition is the guarantee for obtaining source video images. The system obtains enough image sequences through the camera equipment to provide resources for the subsequent processes. At the same time, the camera equipment models must be the same to ensure the consistency of their own parameters. At the same time, there must be overlapping areas between the perspectives of adjacent devices to ensure that adjacent images contain some of the same scenes. After the location is determined, multiple videos are collected through RTSP and then decoded to GPU memory using OpenCV and Cuda. After image preprocessing, including denoising, frame synchronization and other functions, multiple cameras can be synchronized to obtain real-time video streams.

[0075] 2) The captured video image sequence may differ due to external factors of the device or scene, so this step requires lens distortion correction. The distortion mathematical model is as follows:

[0076]

[0077] where δ x (x, y), δ y (x, y) is the distortion coefficient, r is the radius from the imaging point to the center point, and k1, k2, k3 are radial distortion coefficients.

[0078] 3) Assuming the left image is image_left and the right image_right, calibrate the left and right cameras to obtain the corresponding intrinsic parameter matrix k left , k right ;

[0079] 4) Extract the feature points of the left and right images based on the sub-pixel technology of bilinear difference, and use the matrix k left , k right Correct the feature points of the left and right images to obtain the corrected feature points as the point set to be matched in the image;

[0080] 5) Take m pairs of descriptors from the feature points of the left and right images for image matching to form p1, p2, ..., p m with p′1, p′2, ..., p′ m For the corresponding matching point pairs, m≥12 is set according to the empirical value;

[0081] 6) Randomly divide m pairs of descriptors into Group, calculate the corresponding mapping matrix Assume that the first set of points p1, p2, p3, p4 and p′1, p′2, p′3, p′4 corresponding to the left and right images, the corresponding mapping matrix k1;

[0082] 7) Use the mapping matrix k1 to find the left image point set p5, p6, ..., p m The corresponding point set in the right image is p″5, p″6, ..., p″ m, calculate p′5, p′6, ..., p′ in the right figure m and p″5, p″6, ..., p″ calculated by mapping matrix m The Euclidean distances between d5, d6, ..., d m ;

[0083] 8) Set the feature point matching threshold m_threshold to 1 pixel. If d5, d6, ..., d m If more than 80% of the points meet the condition less than m_threshold, delete the other points that do not meet the condition, and then use all the remaining matching point pairs to calculate the optimal mapping matrix k. Otherwise, change a set of point sets and mapping matrices and repeat the calculation process in step 7);

[0084] 9) If there is no point set pair that satisfies the condition in step 8), repeat steps 5)-8) until the optimal mapping matrix k that satisfies the condition is calculated;

[0085] 10) The left and right images are spliced ​​together according to the optimal mapping matrix, and local information fusion technology is used on the left and right sides of the splicing point to achieve a smooth transition of the image.

[0086] 11) Image fusion technology, according to the different imaging of each camera, only the H, S, and V channels of the image need to be processed separately to meet the smoothing conditions. The specific operations are as follows:

[0087] First, calculate the global histogram statistics of the H, S, and V channels of the left and right images, and find the mean of each channel and Calculate the difference of the average values ​​of the three channels, and perform weighted supplementation near the stitching seam according to the difference of each channel to make the image smooth. The specific weighting strategy is as follows:

[0088] if Then the H channel needs to be weighted, and the weighting coefficient can be adjusted according to the actual situation, otherwise no processing is required; if Then the S channel needs to be weighted, and the weighting coefficient can be adjusted according to the actual situation, otherwise no processing is required; if Then the V channel needs to be weighted, and the weighting coefficient can be adjusted according to the actual situation, otherwise no processing is required.

Claims

1. A panoramic image stitching method based on an optimal mapping matrix, characterized in that: include: Collect n video streams obtained by simultaneously shooting the airport scene area with n camera devices; Each camera device captures a portion of the scene, and the scene captured by each camera device has an overlapping area with the scene captured by an adjacent camera device; n is a positive integer greater than 1; According to the preset distortion coefficients of each camera device, the image frames in the n video streams are subjected to distortion correction; The feature descriptor is used to process the corrected image frames at the same time to obtain the optimal mapping matrix between the image frames; The image frames at that moment are stitched together according to the optimal mapping matrix to obtain a panoramic image of the airport; Among them, the corrected image frames at the same time have adjacent left images and right images. The image frames at the same time after correction are processed by using feature descriptors to obtain the optimal mapping matrix between the image frames, including: The feature points of the overlapping area of ​​the left and right images are extracted based on the sub-pixel technology of bilinear difference. The feature points in the left image are called left feature points, which constitute the left feature point set. The feature points in the right image are called right feature points, which constitute the right feature point set. Take m pairs of descriptors from the left and right feature point sets, where a pair of descriptors includes a left feature point and a corresponding right feature point; m is a positive integer; Randomly divide m pairs of descriptors into Group, calculate the mapping matrix for each group , , ..., ; Substitute the first set of mapping matrices into the left feature point set and calculate the new right feature point set; Determine whether the distance between the corresponding feature points in the right feature point set and the new right feature point set is greater than one pixel; If the proportion of the distance less than one pixel relative to the calculated distance is greater than or equal to the preset proportion, the feature points of the left and right feature point sets corresponding to the distance greater than or equal to one pixel are deleted; the mapping matrix is ​​calculated using the remaining left feature points in the left feature point set and the remaining right feature points in the right feature point set, and the mapping matrix is ​​used as the optimal mapping matrix between the left and right images; If the proportion of the distance less than one pixel relative to the distance calculated above is less than the preset proportion, the two sets of mapping matrices are brought into the left feature point set to calculate the optimal mapping matrix.

2. The method according to claim 1, characterized in that After the image frames at the time are spliced ​​according to the mapping matrix to obtain a panoramic image of the airport, the method further includes: The corrected image frame at the same moment is divided into two areas, upper and lower, according to the horizon; These two areas are used together as sampling points to participate in the chromatic aberration correction of the panoramic image.

3. The method according to claim 1, characterized in that The method further comprises: According to the different imaging of each camera, the H, S, and V channels of the panoramic image are smoothed separately.

4. The method according to claim 3, characterized in that The specific operations of smoothing are as follows: First, calculate the global histogram statistics of the H, S, and V channels of the left and right images, and find the mean of each channel , , and , , , find the difference of the average values ​​of the three channels, and perform weighted complementation near the splicing seams according to the difference of each channel to make the image smooth.

5. The method according to claim 4, characterized in that Specific weighting strategies include: if , then the H channel needs to be weighted, and the weighting coefficient is adjusted according to the actual situation, otherwise no processing is required; if , then the S channel needs to be weighted, and the weighting coefficient is adjusted according to the actual situation, otherwise no processing is required; if , then the V channel needs to be weighted, and the weighting coefficient is adjusted according to the actual situation, otherwise no processing is required.

6. A panoramic image stitching device based on an optimal mapping matrix, characterized in that: include: A collection module, used for collecting n video streams obtained by simultaneously shooting the airport scene area by n camera devices; Each camera device captures a portion of the scene, and the scene captured by each camera device has an overlapping area with the scene captured by an adjacent camera device; n is a positive integer greater than 1; A correction module, used to perform distortion correction on image frames in n video streams according to preset distortion coefficients of various camera devices; A processing module, used for processing the corrected image frames at the same time by using feature descriptors to obtain an optimal mapping matrix between the image frames; A stitching module, used to stitch the image frames at that moment according to the optimal mapping matrix to obtain a panoramic image of the airport; The processing module is specifically used for: extracting feature points in the overlapping area of ​​the left image and the right image based on the sub-pixel technology of bilinear difference. The feature points in the left image are called left feature points, which constitute the left feature point set. The feature points in the right image are called right feature points, which constitute the right feature point set. Take m pairs of descriptors from the left and right feature point sets, where a pair of descriptors includes a left feature point and a corresponding right feature point; m is a positive integer; wherein the image frames at the same time after correction have adjacent left and right images; Randomly divide m pairs of descriptors into Group, calculate the mapping matrix for each group , , ..., ; Substitute the first set of mapping matrices into the left feature point set and calculate the new right feature point set; Determine whether the distance between the corresponding feature points in the right feature point set and the new right feature point set is greater than one pixel; If the proportion of the distance less than one pixel relative to the calculated distance is greater than or equal to the preset proportion, the feature points of the left and right feature point sets corresponding to the distance greater than or equal to one pixel are deleted; the mapping matrix is ​​calculated using the remaining left feature points in the left feature point set and the remaining right feature points in the right feature point set, and the mapping matrix is ​​used as the optimal mapping matrix between the left and right images; If the proportion of the distance less than one pixel relative to the distance calculated above is less than the preset proportion, the two sets of mapping matrices are brought into the left feature point set to calculate the optimal mapping matrix.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images