Method for realizing multi-image fusion without distortion by single-chip calibration multi-super wide-angle lens

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a method for realizing multi-image fusion without distortion by single-chip calibration of multiple super-wide-angle lenses, comprising the following steps: obtaining images collected by multiple super-wide-angle lenses of the same specification; calibrating the multiple super-wide-angle lenses based on a standard checkerboard; performing distortion correction on the images collected by the multiple super-wide-angle lenses; splicing the corrected multiple images to obtain a panoramic image; establishing a distortion correction and image splicing parameter data table; processing real-time collected images based on the data table to obtain a panoramic image without distortion; realizing single-chip-based output of a panoramic image without distortion by multiple super-wide-angle lenses, greatly reducing equipment cost and power consumption; improving the precision of distortion correction by using a standard checkerboard for calibration; effectively solving the distortion problem of super-wide-angle lenses by combining a radial distortion model and a spherical projection model, greatly improving the visual experience of users.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a method for realizing multi-image fusion without distortion by single-chip calibration of multiple super-wide-angle lenses. BACKGROUND

[0002] A panoramic image is an image that presents a wide scene, which can be viewed in 360 degrees without dead angles. Such an image is usually composed of multiple wide-angle lenses and is widely used in conferences, security, vehicle monitoring, tourism, construction, urban planning and other fields, allowing people to have a more comprehensive understanding of events or appreciate the surrounding real scene. At the same time, it can also be used in virtual reality, augmented reality and other technologies to provide a more immersive experience.

[0003] Existing panoramic cameras generally use fisheye lenses or wide-angle lenses. These lenses all have a major defect, i.e., optical distortion. Without algorithm distortion correction, the image will have distortion problems, which brings a poor visual experience to users. Currently, panoramic cameras on the market usually need to use multiple chips or high-performance processors to process distortion correction and image stitching, which leads to an increase in device cost and power consumption SUMMARY

[0004] The purpose of the present application is to provide a method for realizing multi-image fusion without distortion by single-chip calibration of multiple super-wide-angle lenses, so as to solve the distortion problem in existing panoramic cameras and reduce device cost and power consumption.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] The method for realizing multi-image fusion without distortion by single-chip calibration of multiple super-wide-angle lenses comprises the following steps:

[0007] Obtaining images collected by multiple super-wide-angle lenses of the same specification;

[0008] Calibrating the multiple super-wide-angle lenses based on a standard checkerboard;

[0009] Correcting distortion of the images collected by the multiple super-wide-angle lenses;

[0010] Stitching the corrected multiple images to obtain a panoramic image;

[0011] Establishing a distortion correction and image stitching parameter data table;

[0012] Processing real-time collected images based on the data table to obtain a panoramic image without distortion.

[0013] Specifically, the standard checkerboard comprises:

[0014] a first checkerboard with an aspect ratio of 2:1; and

[0015] The second chessboard has a length-width ratio of 1:2.

[0016] Specifically, the calibration of the plurality of ultra-wide-angle lenses based on the standard chessboard includes:

[0017] Separate calibration of each ultra-wide-angle lens using the second chessboard;

[0018] Joint calibration of every two adjacent ultra-wide-angle lenses using the first chessboard.

[0019] Specifically, the distortion correction of the images collected by the plurality of ultra-wide-angle lenses includes:

[0020] Distortion correction of the images collected by each ultra-wide-angle lens based on a radial distortion model;

[0021] Global adjustment of the images collected by adjacent lenses based on a spherical projection model.

[0022] Specifically, the radial distortion model is:

[0023]

[0024] where (x s ,y s ) is the point coordinate before distortion correction, (x d ,y d ) is the point coordinate after distortion correction, (x cd ,y cd ) is the distortion center point coordinate, and (k1, k2, k3,...) is the distortion parameter.

[0025] Specifically, the spherical projection model is:

[0026]

[0027] where (x d ,y d ) is the point coordinate before spherical projection, (x sp ,y sp ) is the point coordinate after spherical projection, and r is the spherical projection radius.

[0028] Specifically, the stitching of the corrected plurality of images includes:

[0029] Separation of the corrected images into a plurality of frequency band images using filter convolution in the YUV color space;

[0030] Linear blending of every two adjacent projection images in each frequency band image using a fade-in and fade-out or pyramid method.

[0031] Specifically, the stitching of the plurality of corrected images further comprises:

[0032] According to the visible angle after the stitching, a less-than-360-degree stitching or a 360-degree stitching mode is selected.

[0033] Specifically, when the less-than-360-degree stitching is selected, the number of fusion regions is one less than the number of images.

[0034] When the 360-degree stitching is selected, the number of fusion regions is equal to the number of images.

[0035] Specifically, the establishing of the distortion correction and image stitching parameter data table comprises:

[0036] The distortion correction parameters, the mapping and deformation parameters, and the image stitching parameters are stored in the data table.

[0037] The data table is input into an image processing chip for real-time image processing.

[0038] The beneficial effects of the present application include:

[0039] 1. The single-chip-based multi-super-wide-angle lens non-distortion panoramic image output is realized, and the device cost and power consumption are greatly reduced.

[0040] 2. The accuracy of distortion correction is improved by using a standard checkerboard for calibration.

[0041] 3. The radial distortion model and the spherical projection model are combined to effectively solve the distortion problem of the super-wide-angle lens.

[0042] 4. Through the establishment of the parameter data table, real-time and efficient image processing is realized, and the processing speed of the system is improved.

[0043] 5. The final output panoramic image is non-distortion, which greatly improves the user's visual experience. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a less-than-360-degree stitching schematic diagram of the present application.

[0045] Figure 2 It is a 360-degree stitching of the present application.

[0046] Figure 3 It is a schematic diagram of the pre-correction picture of the present application.

[0047] Figure 4 It is a schematic diagram of the post-correction picture of the present application. DETAILED DESCRIPTION

[0048] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0049] Embodiment 1:

[0050] This embodiment provides a method for single-chip calibration of multiple ultra-wide-angle lenses to achieve multi-image fusion without distortion, and the specific steps are as follows:

[0051] Step 1: Obtain images collected by multiple ultra-wide-angle lenses of the same specification.

[0052] This embodiment uses four ultra-wide-angle lenses of the same specification, each with a field of view of 220 degrees. These lenses are mounted on a circular base and distributed at 90-degree intervals to achieve 360-degree panoramic coverage.

[0053] Step 2: Calibrate multiple ultra-wide-angle lenses based on standard checkerboards.

[0054] This step uses two different specifications of standard checkerboards:

[0055] a) 2:1 checkerboard: aspect ratio of 2:1, used for joint calibration of adjacent lenses.

[0056] b) 1:2 checkerboard: aspect ratio of 1:2, used for single lens calibration.

[0057] The specific calibration process is as follows:

[0058] 2.1 Single lens calibration:

[0059] Use the 1:2 checkerboard to calibrate each lens individually. Ensure that the checkerboard covers the lens view and that the lens is perpendicular to the checkerboard plane. Collect multiple checkerboard images at different angles for each lens to calculate distortion parameters later.

[0060] 2.2 Joint calibration of adjacent lenses:

[0061] Use the 2:1 checkerboard to calibrate each two adjacent lenses jointly. Place the checkerboard in the overlapping area of the two lenses to ensure that both lenses can capture part of the checkerboard. This step helps with subsequent image stitching and global adjustment.

[0062] Step 3: Distortion correction of images collected by multiple ultra-wide-angle lenses

[0063] 3.1 Radial distortion correction:

[0064] Perform radial distortion correction on images collected by each lens individually. Use the following radial distortion model:

[0065]

[0066] where (x, y) is the pixel coordinate in the original image, (x', y') is the pixel coordinate in the corrected image, and k is the radial distortion coefficient.s , y) is the point coordinate before distortion correction, (x s , y) is the point coordinate after distortion correction, (x d , y) is the point coordinate after distortion correction, (x d , y) is the point coordinate after distortion correction, (x cd , y) is the point coordinate after distortion correction, (x cd , y) is the point coordinate after distortion correction, (x

[0067] The point in the space is projected to the image plane, and then the plane image point is extended to the point of the distortion image. Since the relationship between the distortion point and the space point is the relationship between the point and the straight line, the derived mapping relationship between the space point and the distortion point is also the same as the fundamental matrix. Therefore, the distortion center can be obtained by using the method of obtaining the pole by using the F matrix;

[0068] The determinant of the matrix formed by the three-point coordinates corresponding to the three-point collineation in the non-distortion image is 0. The distortion parameters are obtained. Assuming that the distortion center coordinates are known, and the origin of the image coordinate system is placed at the distortion center, for the 3 points pi=(xi yi)T collinear on the image, ri=||(xi yi)|| is the length of the 3 points from the distortion center, there is a relationship:

[0069]

[0070] According to the above determinant, for a plurality of three-point collinear determinants, the distortion parameters are obtained by solving the least squares problem;

[0071] The solving process of the distortion parameters is as follows:

[0072] a) Using the checkerboard image collected in the single lens calibration, the checkerboard corner points are extracted.

[0073] b) Based on the principle that the determinant of the matrix formed by the three-point coordinates corresponding to the three-point collineation in the non-distortion image is 0, an equation group is established.

[0074] c) The least squares method is used to solve the distortion parameters.

[0075] 3.2 Spherical projection correction:

[0076] For the configured calibration target, the angle bisector of every two adjacent lenses basically faces the calibration target, and the positioning points on the calibration target are in the overlapping area of the visual range of the two lenses. The global adjustment image collection work is completed. For each group of images, find the corner points of the calibration target in the image, and match the corner points of the two images according to the positional relationship with the positioning points to obtain the matching point pair. Assuming that the matching point pair of the i-th group is (pli, p2i),

[0077] The images collected by the adjacent lenses are globally adjusted, and the following spherical projection model is used:

[0078]

[0079] where (x d ,y d ) is the pre-spherical projection point coordinate, (x sp ,y sp ) is the post-spherical projection point coordinate, and r is the spherical projection radius. Before each lens passes through the spherical projection, a projective transformation is performed to make the pre-projection planar image tangent to the sphere with a hypothetical radius r, and the tangent point coordinate (xcs,ycs) is the spherical projection center coordinate. Assuming that the homography matrix of the projective transformation is H, the total coordinate mapping from the original image to the target image is F(). The spherical projection center coordinate (xcs,ycs), the spherical projection radius r, and the homography matrix H in the above are the parameters required for global adjustment and optimization. Combined with the attitude angle of each lens in the imaging group (based on one of the lenses), the Levenberg-Marquardt algorithm can be used here to optimize the required parameters, with the sum of the absolute values of the coordinate differences of all group matching point pairs as the cost indicator:

[0080] (xes,yes,r,H) = arg min Z abs(F(pli)-F(p2i))

[0081] The projection images mapped by the above mapping are separated into multiple frequency band images using a filter convolution in the YUV color space. Each two adjacent projection images are linearly fused in each frequency band image using a fade-in and fade-out or pyramid method to obtain a spherical stitched panoramic image.

[0082] The global adjustment process is as follows:

[0083] a) Using the checkerboard images collected during the joint calibration of adjacent lenses, corresponding points in the two images are extracted.

[0084] b) Using the Levenberg-Marquardt algorithm to optimize the spherical projection parameters, including the spherical projection center coordinate (xcs,ycs), the spherical projection radius r, and the homography matrix H.

[0085] c) Applying the optimized parameters to image correction.

[0086] Step 4: Stitching the corrected multiple images to obtain a panoramic image

[0087] 4.1 Image frequency band separation: converting the corrected image to the YUV color space and using a Gaussian filter to perform multi-scale decomposition on the Y channel to obtain multiple frequency band images.

[0088] 4.2 Image fusion: Linearly fuse each two adjacent projection images in each frequency band. The fusion method can be selected as fade-in fade-out or Multiresolution Spline method.

[0089] 4.3 Mosaic method selection:

[0090] According to the actual application requirements, one of the following two mosaic methods is selected:

[0091] a) Less than 360° mosaic: The first and last are not connected, and the number of fusion regions is one less than the number of images.

[0092] b) Equal to 360° mosaic: The first and last are connected, and the number of fusion regions is equal to the number of images.

[0093] Shooting chessboard: Select 2:1 chessboard, shoot chessboard with two lenses as a group, 0-1, 1-2, 2-3, 3-0, a total of four groups, and ensure that the camera lens is perpendicular to the chessboard plane. Multiple ultra-wide-angle lenses are used for acquisition and shooting.

[0094] Splice multiple corrected images to generate a panoramic image. In the splicing process, the effects of factors such as depth of field and lighting need to be considered to ensure that the image splicing effect is natural and beautiful.

[0095] Multi-image stitching includes the calculation of registration (Registration), which maps images to the same coordinate system according to a geometric motion model.

[0096] Taking the splicing of five images as an example, according to the visible angle after splicing, it is divided into "less than 360° mosaic" and "equal to 360° mosaic", the difference lies in the number of subsequent fusion regions. The method is set by the firmware.

[0097] "Less than 360° mosaic": The first and last are not connected, and the number of fusion regions is (5-1) = 4, as shown in Figure 1 ;

[0098] "= 360° mosaic": The first and last are connected, and the number of fusion regions is 5, as shown in Figure 2 ;

[0099] Step 5: Establish a distortion correction and image stitching parameter data table

[0100] Store all the parameters obtained in the above steps in a structured data table, including:

[0101] Radial distortion parameters of each lens;

[0102] Spherical projection parameters;

[0103] Image stitching parameters (such as fusion region position, weight, etc.).

[0104] Step 6: Process the real-time collected images based on the data table to obtain a distortion-free panoramic image.

[0105] 6.1 Load the parameter data table into the memory of the image processing chip.

[0106] 6.2 Collect images of multiple lenses in real time.

[0107] 6.3 Correct the distortion of the image of each lens using the loaded parameters.

[0108] 6.4 Stitch the corrected images into a panoramic image according to the stitching parameters.

[0109] 6.5 Output the distortion-free panoramic image.

[0110] Through the above steps, the invention realizes multi-super-wide-angle lens distortion-free panoramic image output based on a single chip. This method not only solves the distortion problem in traditional panoramic cameras, but also greatly reduces the cost and power consumption of the device.

[0111] Embodiment 2:

[0112] This embodiment provides an optimization method for dynamic scenes, which is improved based on Embodiment 1:

[0113] 1. Dynamic distortion parameter adjustment.

[0114] In order to cope with the possible small deformation of the lens caused by light changes and temperature changes in different environments, this embodiment introduces a dynamic distortion parameter adjustment mechanism:

[0115] a) Reserve certain computing resources in the image processing chip for real-time evaluation of image distortion.

[0116] b) Set a distortion evaluation period (such as every 10 minutes), and measure the distortion of the output image within this period.

[0117] c) If the detected distortion exceeds the preset threshold, trigger a fast calibration process:

[0118] Use natural feature points (such as straight lines, corner points, etc.) in the scene instead of chessboard to perform fast calibration.

[0119] Update the related parameters in the distortion correction parameter data table.

[0120] 2. Adaptive stitching area adjustment.

[0121] In order to cope with the possible stitching marks under different lighting conditions, this embodiment introduces an adaptive stitching area adjustment mechanism:

[0122] a) In the stitching process, the color histogram difference of adjacent images in the overlapping area is calculated in real time.

[0123] b) According to the histogram difference, the width of the stitching area and the fusion weight are dynamically adjusted.

[0124] c) In the case of large light difference, a local color balance algorithm can be enabled to reduce the stitching marks.

[0125] 3. Motion compensation

[0126] For high-speed motion scenes (such as vehicle-mounted applications), this embodiment introduces a motion compensation mechanism:

[0127] a) Use a hardware-accelerated optical flow algorithm to calculate the motion vector between adjacent frames.

[0128] b) In the stitching process, the image is fine-tuned according to the motion vector to reduce the stitching misalignment caused by motion.

[0129] 4. Intelligent exposure control

[0130] To cope with different light conditions that different lenses may face, this embodiment introduces an intelligent exposure control mechanism:

[0131] a) Perform exposure analysis on each lens image to obtain the best exposure parameters.

[0132] b) In the stitching process, local exposure adjustment is performed according to the best exposure parameters of each region.

[0133] c) Use multi-exposure fusion technology to achieve high dynamic range (HDR) imaging while stitching.

[0134] Through the above optimization, this embodiment further improves the quality of panoramic images and the adaptability of the system.

[0135] Embodiment 3:

[0136] This embodiment provides a distortion correction and image stitching method based on deep learning, which makes the following improvements based on Embodiment 1:

[0137] 1. Distortion correction based on deep learning

[0138] a) Data preparation: Collect a large number of distorted images and corresponding ideal non-distorted image pairs under different scenes and different light conditions. Use data augmentation techniques to expand the data set, including rotation, scaling, brightness adjustment, etc.

[0139] b) Network design: Design a convolutional neural network with U-Net structure to learn the mapping relationship from distortion to non-distortion. The input is the original distorted image, and the output is the corrected non-distorted image.

[0140] c) Training Process: Supervised learning is performed using paired distorted images and ideal undistorted images. The loss function combines L1 loss and perceptual loss to ensure the sharpness and details of the corrected images.

[0141] d) Deployment: The trained model is quantized and deployed into the image processing chip. The model inference results replace the traditional distortion correction algorithms.

[0142] 2. Deep Learning-based Image Stitching

[0143] a) Data Preparation:

[0144] A large number of adjacent image pairs and their ideal stitching results are collected.

[0145] The dataset should include various lighting conditions, motion scenes, and other complex situations.

[0146] b) Network Design:

[0147] A dual-branch Siamese network structure is designed to learn the optimal stitching method for adjacent images.

[0148] The input is two adjacent images, and the output is the stitched seamless image.

[0149] c) Training Process:

[0150] Supervised learning is performed using paired adjacent images and ideal stitching results.

[0151] The loss function includes reconstruction loss, perceptual loss, and adversarial loss to ensure the naturalness and coherence of the stitching results.

[0152] d) Deployment:

[0153] The trained model is quantized and deployed into the image processing chip.

[0154] The model inference results replace the traditional image stitching algorithms.

[0155] 3. Online Learning and Adaptation

[0156] To enable the system to continuously adapt to new environments and scenarios, this embodiment introduces an online learning mechanism:

[0157] a) Reserve certain computational resources in the image processing chip for online fine-tuning of the model.

[0158] b) Design a lightweight quality assessment network to evaluate the effectiveness of distortion correction and image stitching.

[0159] c) When poor performance is detected, trigger the online learning process:

[0160] Collecting a recent batch of image data.

[0161] Using federated learning technology, the model is updated in collaboration with the cloud model while protecting privacy.

[0162] Applying the updated model parameters to the local system.

[0163] By introducing deep learning technology, the embodiment realizes a more intelligent and adaptive distortion correction and image stitching method, which can better cope with complex and variable actual application scenarios.

[0164] The method for realizing multi-image fusion without distortion based on single-chip calibration of multiple ultra-wide-angle lenses provided by the application realizes high-quality panoramic image output through innovative calibration technology, distortion correction algorithm and image stitching method. The method not only solves the distortion problem in traditional panoramic cameras, but also greatly reduces the equipment cost and power consumption. By introducing dynamic parameter adjustment, adaptive stitching and deep learning technology, the adaptability and intelligence of the system are further improved. The method of the application can be widely applied in the fields of security monitoring, vehicle-mounted camera, virtual reality, etc., and provides users with clearer and more natural panoramic visual experience.

[0165] It should be noted that the above description is only a preferred embodiment of the application, and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the principles of the application shall be included in the protection scope of the application.

Claims

1. A method for single-chip calibration of a multi-hyper wide-angle lens to achieve multi-image fusion without distortion, characterized in that The method comprises the following steps: obtaining a plurality of images collected by a plurality of same-specification ultra-wide-angle lenses; calibrating the plurality of ultra-wide-angle lenses based on a standard chessboard; correcting distortion of the images collected by the plurality of ultra-wide-angle lenses; stitching the corrected images to obtain a panoramic image; establishing a parameter data table for distortion correction and image stitching; processing real-time collected images based on the data table to obtain a panoramic image without distortion; the standard chessboard comprises: a first chessboard with a length-width ratio of 2:1; and a second chessboard with a length-width ratio of 1:2; the calibration of the plurality of ultra-wide-angle lenses based on the standard chessboard comprises: separately calibrating each ultra-wide-angle lens using the second chessboard; and jointly calibrating each two adjacent ultra-wide-angle lenses using the first chessboard.

2. The method of claim 1, wherein the distortion correction of the images collected by the plurality of ultra-wide-angle lenses comprises: correcting distortion of the images collected by each ultra-wide-angle lens based on a radial distortion model; and globally adjusting the images collected by adjacent lenses based on a spherical projection model.

3. The method of claim 2, wherein the radial distortion model is: where (x s ,y s ) is a pre-distortion corrected point coordinate, (x d ,y d ) is a post-distortion corrected point coordinate, (x cd ,y cd ) is a distortion center point coordinate, and (k1, k2, k3,...) is a distortion parameter.

4. The method of claim 2, wherein the spherical projection model is: where (x d ,y d ) is the pre-spherical projection point coordinate, (x sp ,y sp ) is the post-spherical projection point coordinate, and r is the spherical projection radius.

5. The method of claim 1, wherein the stitching of the corrected images comprises: separating a plurality of frequency band images from the corrected images in a YUV color space using a filter convolution; linearly fusing each two adjacent projection images in each frequency band image in a fade-in and fade-out or pyramid manner.

6. The method of claim 1, wherein the stitching of the corrected images further comprises: selecting a less-than-360° stitching or equal-to-360° stitching mode according to a visible angle after stitching.

7. The method of claim 6, wherein : when the less-than-360° stitching is selected, the number of fusion regions is one less than the number of images; when the equal-to-360° stitching is selected, the number of fusion regions is equal to the number of images.

8. The method of claim 1, wherein the establishment of the parameter data table for distortion correction and image stitching comprises: storing distortion correction parameters, mapping and deformation parameters, and image stitching parameters in the data table; and inputting the data table into an image processing chip for real-time image processing.

Images