Multi-lens large-view-field high-resolution imaging method, device, equipment and medium

Abstract

The invention discloses a multi-lens large-view-field high-resolution imaging method and device, equipment and a medium. The method comprises the following steps: synchronously acquiring images acquired by a central lens and all fisheye lenses; performing distortion correction processing on the images acquired by the fisheye lenses based on the camera calibration parameters to obtain corrected fisheye images; performing motion feature matching on the corrected fisheye image and the center lens image; registering the corrected fisheye image to the visual angle of the central lens image based on a motion feature matching result; carrying out fusion processing on the registered image; and post-processing the fused image to obtain a final image. According to the method, distortion correction processing is carried out on the fisheye lens, feature matching and registration are carried out on the fisheye lens and the center lens, complex scenes such as weak textures and violent or dynamic illumination change can be effectively dealt with, and robustness and imaging quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of lens technology, and in particular to a multi-lens large-field-of-view high-resolution imaging method, device, equipment and medium. Background Art

[0002] The so-called large field of view is the ability to provide a relatively wide field of view. The existing imaging system uses a combination of wide-angle lenses and fish-eye lenses to achieve large field of view imaging requirements. Although the wide-angle lens can obtain a larger field of view, the edge resolution is significantly reduced. The fish-eye lens can obtain a larger field of view, but the image distortion is serious. Therefore, it is impossible to meet the needs of high resolution while meeting the needs of a large field of view.

[0003] A Chinese patent with publication number CN109191415B proposes a universal image fusion method, which emphasizes the calculation of the homography matrix based on the reference field of view and the current field of view for image registration. The patent only proposes to fuse the two image acquisition systems to improve the imaging quality, but does not take into account the fisheye lens distortion problem. When using the patented method to process fisheye images, feature extraction and matching are easily affected by factors such as distortion and illumination changes, resulting in reduced stitching accuracy and robustness.

[0004] The Chinese patent with publication number CN112258581B proposes a method for on-site calibration of a panoramic camera with multiple fisheye lenses. This method can solve the problem caused by fisheye lens distortion, but this method is prone to mismatching during feature matching, resulting in poor imaging quality, especially in scenes with repeated or weak textures, and cannot effectively handle moving objects in dynamic scenes. The Chinese patent with publication number CN104835118A proposes a method for collecting panoramic images through two fisheye cameras. This method is mainly aimed at only two opposing fisheye lenses, and cannot effectively handle moving objects in dynamic scenes.

[0005] In summary, the existing multi-lens imaging system including fisheye lens still has room for improvement in imaging, especially for scenes with weak texture, drastic lighting changes and dynamic scenes, the existing methods are difficult to meet the actual needs of imaging. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a multi-lens large field of view high-resolution imaging method, device, equipment and medium, in order to improve the quality of multi-lens imaging.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a multi-lens large field of view high-resolution imaging method, wherein the multi-lens includes a central lens and four fisheye lenses, the four fisheye lenses are geometrically distributed around the central lens, and the field of view angles of the central lens and the four fisheye lenses overlap each other, the central lens is used to capture an image of a central area, and the optical resolution of the central lens is higher than that of the fisheye lens, and the fisheye lens is used to capture a large field of view image of a peripheral area; the method includes:

[0009] Synchronously acquire images collected by the central lens and each fisheye lens;

[0010] Based on the camera calibration parameters, the images captured by each fisheye lens are subjected to distortion correction processing to obtain a corrected fisheye image;

[0011] Perform motion feature matching on the corrected fisheye image and the center lens image;

[0012] The corrected fisheye image is registered to the viewing angle of the central lens image based on the motion feature matching result;

[0013] Perform fusion processing on the registered images;

[0014] The fused image is post-processed to obtain the final image.

[0015] In a second aspect, the present invention further provides a multi-lens large field of view high-resolution imaging device, the multi-lens comprising a central lens and four fisheye lenses, the four fisheye lenses are geometrically distributed around the central lens, and the field of view angles of the central lens and the four fisheye lenses overlap each other, the central lens is used to capture an image of a central area, and the optical resolution of the central lens is higher than that of the fisheye lens, and the fisheye lens is used to capture a large field of view image of a peripheral area; the device comprises:

[0016] An image acquisition unit, used for synchronously acquiring images collected by the central lens and each fisheye lens;

[0017] A distortion correction unit, used to perform distortion correction processing on the images collected by each fisheye lens based on the camera calibration parameters to obtain a corrected fisheye image;

[0018] A motion feature matching unit, used for performing motion feature matching on the corrected fisheye image and the center lens image;

[0019] An image registration unit, for registering the corrected fisheye image to the viewing angle of the central lens image based on the motion feature matching result;

[0020] A fusion processing unit, used for performing fusion processing on the registered images;

[0021] The post-processing unit is used to perform post-processing on the fused image to obtain a final image.

[0022] In a third aspect, the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a multi-lens, large-field-of-view, high-resolution imaging method as described above is implemented.

[0023] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes a multi-lens, large-field-of-view, high-resolution imaging method as described above.

[0024] Compared with the prior art, the beneficial effects of the present invention are: a multi-lens large field of view high-resolution imaging method, the multi-lens includes a central lens and four fisheye lenses, the four fisheye lenses are geometrically distributed around the central lens, and the field of view angles of the central lens and the four fisheye lenses overlap each other, the central lens is used to capture the image of the central area, and the optical resolution of the central lens is higher than that of the fisheye lens, and the fisheye lens is used to capture the large field of view image of the peripheral area; the method includes: synchronously acquiring the images collected by the central lens and each fisheye lens; performing distortion correction processing on the images collected by each fisheye lens based on the camera calibration parameters to obtain a corrected fisheye image; performing motion feature matching on the corrected fisheye image and the central lens image; aligning the corrected fisheye image to the viewing angle of the central lens image based on the motion feature matching result; performing fusion processing on the aligned image; and post-processing the fused image to obtain the final image. The present invention can effectively cope with complex scenes such as weak texture, drastic or dynamic lighting changes by performing distortion correction processing on the fisheye lens and feature matching and registration on the central lens, thereby improving robustness and imaging quality.

[0025] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0027] Figure 1 A flowchart of a multi-lens large field of view high-resolution imaging method provided by a specific embodiment of the present invention; Figure 2A schematic block diagram of a multi-lens, large-field-of-view, high-resolution imaging device provided in a specific embodiment of the present invention; Figure 3 A schematic block diagram of a computer device provided for a specific embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The present invention is applicable to large scenes, wide ranges and visual information collection scenarios that require clear details in the central area. For example, in the field of security monitoring, it is used for real-time monitoring of large areas in large public places such as airport waiting halls, railway station squares, and atriums of large shopping malls. It is necessary to capture clear individual characteristics in crowded places (the central lens plays a role), and to take into account the dynamics of large fields of view such as surrounding passages, entrances and exits (the fisheye lens is responsible). For another example, in the field of intelligent transportation, it can be installed at complex intersections. The central lens focuses on traffic lights and vehicle license plates and driving status details of the main lanes, while the fisheye lens covers the traffic flow and pedestrian conditions of the entire intersection and surrounding branches. For another example, in the acquisition of virtual reality / augmented reality content, it can provide broad and detailed image materials for creating immersive scenes, meeting the needs of panoramic immersion and close-up of key elements. It can also be used for panoramic mapping of local areas in geographic surveying and mapping, such as recording the topography and geomorphology around urban landmarks, taking into account the details of the building itself and the situation of the surrounding environment.

[0030] The multi-lens, large-field-of-view, high-resolution imaging method of the present invention can be integrated into an imaging device, which can be a professional-grade industrial camera integration device designed as an independent device with a protective casing and a stable mounting structure, suitable for long-term stable operation in harsh outdoor environments; it can also be a core imaging module embedded in smart terminal devices (such as high-end smart phones, professional-grade tablets for on-site investigation and recording), vehicle-mounted intelligent vision systems, drone aerial photography gimbal camera components, etc.

[0031] A multi-lens large-field-of-view high-resolution imaging method, the multi-lens comprising a central lens and four fisheye lenses, the four fisheye lenses being geometrically distributed around the central lens, and the field angles of the central lens and the four fisheye lenses overlapping each other, the central lens being used to capture images of a central area, and the optical resolution of the central lens being higher than that of the fisheye lenses, and the fisheye lenses being used to capture images of a peripheral area with a large field of view.

[0032] Specifically, the central lens uses a fixed-focus lens with high optical resolution, such as the Canon EF 50mm f / 1.2LUSM, whose optical resolution can clearly capture the tiny details in the central area. For example, when shooting the scene of cultural relic restoration, it can accurately present the texture and color changes on the surface of the cultural relic, as well as the fine operation parts of the restoration tools. The four fisheye lenses use models suitable for capturing a large field of view, such as the Nikon AF DX Fisheye-Nikkor10.5mm f / 2.8G ED. They are evenly distributed in a circular geometry around the central lens, and the angle between the optical axes of adjacent fisheye lenses is 90 degrees, ensuring that there is no visual blind spot in both the horizontal and vertical directions, and covering the surrounding area in all directions.

[0033] A multi-lens large-field-of-view high-resolution imaging method comprises the following steps: S10-S60.

[0034] S10, synchronously acquiring images captured by the central lens and each fisheye lens.

[0035] In this embodiment, synchronous acquisition of images captured by the central lens and each fisheye lens is the basis for subsequent high-quality imaging. All lenses (referring to the central lens and each fisheye lens) are triggered to be exposed synchronously, which can be achieved through hardware triggering or software synchronization, ensuring that all images are captured at the same time, ensuring time synchronization, and then reading image data from all lenses.

[0036] S20, performing distortion correction processing on the images collected by each fisheye lens based on the camera calibration parameters to obtain a corrected fisheye image.

[0037] In one embodiment, step S20 specifically includes the following steps: S201 - S203 .

[0038] S201: Obtain camera calibration parameters of each fisheye lens.

[0039] In this embodiment, a calibration field is constructed, and a checkerboard calibration plate is selected. The side length of the checkerboard is controlled to an accuracy of ±0.05mm to ensure the accuracy of the calibration benchmark. The calibration field is equipped with a uniform, stable, and constant color temperature (such as 5500K) lighting source to avoid uneven light from causing shadows or reflections to interfere with camera acquisition and affect the calibration accuracy. The four fisheye lenses are sequentially mounted on a pan-tilt fixture with high-precision displacement adjustment function. The pan-tilt accuracy can reach ±0.1° rotation and ±0.1mm translation, which is convenient for adjusting the relative position of the lens and the calibration plate to ensure that clear and complete calibration images are obtained in different postures.

[0040] For each fisheye lens, set multiple different shooting positions and angle combinations. For example, at 0.5m, 1m, and 2m from the calibration plate, shoot at horizontal angles of ±30°, ±60°, and vertical angles of ±20°, ±40°, respectively. At least 10 images are collected at each position and angle to ensure that all possible imaging states of the lens are covered. A total of about 60 calibration images are obtained for each lens.

[0041] The Zhang Zhengyou calibration algorithm is used in combination with the characteristics of the fisheye lens for optimization. For the calculation of internal parameters, the correspondence between the pixel coordinates of the corner points of the chessboard in the image and the world coordinates is used to minimize the reprojection error through multiple iterations, and accurately solve the focal length (for example, the focal length of a fisheye lens is finally determined to be 8mm, with an error range of ±0.02mm), the principal point coordinates (the principal point X coordinate is accurate to ±0.5 pixels, and the Y coordinate accuracy is the same) and the radial distortion coefficient (generally the second-order and third-order radial distortion coefficients are accurate to ±0.001) and other parameters. In terms of external parameters, based on the obtained internal parameters, the imaging coordinate changes of the same world coordinate point under different lens postures are used to calculate the rotation and translation matrices using a method based on singular value decomposition (SVD). The accuracy of each element of the rotation matrix is ​​controlled at ±0.005, and the accuracy of the translation matrix is ​​±0.1mm, ensuring that the spatial position and attitude parameters of the lens relative to the calibration plate are accurate.

[0042] S202: Select a distortion correction algorithm to correct the collected fisheye image.

[0043] In this embodiment, for radial distortion correction, a second-order polynomial model is used: d =x(1+k1r 2 +k2r 4 ), y d =y(1+k1r 2 +k2r 4 ), where (x, y) is the original image pixel coordinate, (x d ,y d ) is the corrected coordinate, k1 and k2 are radial distortion coefficients. Taking a fisheye lens image as an example, k1=0.25, k2=-0.08, the original image is traversed pixel by pixel, and the correction coordinates are calculated according to the above formula. For non-integer coordinates, bilinear interpolation is used to obtain the corrected pixel value. For example, the coordinates of a pixel point at the edge of the original image are (100.3, 200.7), and the calculated corrected coordinates are (98, 202.5). The bilinear interpolation is performed using the pixel values ​​of its four neighborhoods to obtain the precise pixel value after correction to ensure the continuity of the image geometry. For tangential distortion, the formula is used:

[0044] x d =x+[2p1xy+p2(r 2 +2x2 )],y d =y+[p1(r 2 +2y 2 )+2p2xy], where p1 and p2 are tangential distortion coefficients. Combined with the radial distortion correction step, the tangential distortion effect is corrected synchronously to ensure that the image details, especially the texture at the edge, are truly restored. For example, when shooting a circular object, it will be elliptical due to tangential distortion before correction, and the circular outline will be accurately restored after correction.

[0045] The fisheye image is divided into multiple concentric annular areas, and different correction strategies are adopted according to the degree of distortion in different areas. For example, a low-order polynomial model is used in the central area (radius 0-50 pixels), because its distortion is relatively small, and a simple model can be used for efficient correction and avoid image blur caused by excessive calculation; the middle annular area (radius 51-150 pixels) combines high-order polynomials and linear stretching transformations to adjust the uniformity of pixel distribution while correcting distortion; the edge area (radius 151 pixels to the image boundary) introduces a correction method based on a mapping lookup table (LUT), which calculates a large number of typical coordinate distortion mapping relationships based on calibration parameters in advance and stores them in the LUT. During real-time correction, the correction coordinates are quickly indexed to obtain the correction coordinates, thereby improving the speed and accuracy of edge correction. For example, when shooting a panoramic view of a city, after the edge building outline is corrected by the segmented model, the lines are straight, the details are clear, and there is no obvious jagged or stretched deformation.

[0046] S203: Generate a corrected fisheye image.

[0047] During the correction process, the image quality is monitored and optimized in real time. The image gradient information is used to evaluate the clarity of the corrected image. If it is found that the local area is blurred due to the correction (such as the gradient amplitude is lower than the set threshold), the interpolation algorithm parameters are automatically adjusted or the local smoothing filter strength is increased to restore the image details while ensuring the accuracy of the geometric shape. For example, in the natural scenery image taken by the fisheye lens, the edge of the leaves is initially blurred after correction. By adaptively adjusting the filter parameters, the edge of the leaves is re-sharpened to make the texture clear and natural.

[0048] Maintain the color consistency of the image. Since distortion correction may affect the color distribution, the color histogram of the corrected image is compared and matched with the color histogram of the original calibration plate image based on color histogram matching technology, and the red, green, and blue channel gains are adjusted to ensure that the image color is not distorted. For example, when shooting flowers, the color vividness of the flowers after correction is consistent with the real scene.

[0049] For steps S201-S203, the distortion in the fisheye lens image is effectively eliminated through accurate calibration parameters and adaptation algorithms, the fisheye image distortion is accurately corrected, the straight object imaging is restored to be straight, and the circular object contour is not deformed.

[0050] S30, performing motion feature matching on the corrected fisheye image and the center lens image.

[0051] In one embodiment, step S30 specifically includes the following steps: S301 - S304 .

[0052] S301 : Perform feature point detection on the calibrated fisheye image and the center lens image using a feature detector.

[0053] In this embodiment, a Harris corner point detector is selected for feature point detection for the center lens image, and a FAST detector is selected for feature point detection for the eye image.

[0054] S302. For each detected feature point, calculate feature descriptors of the pixels around it, wherein when calculating the feature descriptor of the central lens image, the resolution-related information between pixels is enhanced, and when calculating the feature descriptor of the fisheye image, the field of view-related information is enhanced.

[0055] In this embodiment, the central lens image feature descriptor strengthens the resolution information in the following manner: taking the feature point as the center, selecting a 3×3 neighborhood window, subdividing the window into 9 sub-regions, and statistically analyzing the gradient amplitude histograms in 8 directions (0°-360° is equally divided into 8 intervals) in each sub-region, and combining them to form a 9×8=72-dimensional feature vector. When calculating the gradient amplitude and direction, a high-precision Sobel operator is used, and the operator kernel is fine-tuned according to the central lens pixel sampling characteristics to make it more suitable for high-resolution image detail extraction. For example, when shooting a microscopic circuit image of a circuit board, for feature points generated by fine structures such as circuit intersections and solder joint edges, this high-resolution sensitive HOG descriptor can accurately encode the direction and intensity differences of subtle grayscale changes between pixels, ensuring that the detailed features of different circuits can be distinguished. Even for similar circuit layouts, the Euclidean distance difference of the feature descriptor can reach more than 10, effectively improving the subsequent matching recognition.

[0056] The fisheye image feature descriptor highlights the field of view characteristics by using annular partition statistics combined with radial gradient features to construct a descriptor. The circular area with the feature point as the center and a radius of R (dynamically determined according to the field of view range of the fisheye image and the position of the feature point, generally R is 10-50 pixels) is divided into three concentric annular bands. The gray gradient integral value along the radial direction is calculated for each annular band, and the statistics such as the gray mean and variance of the pixels in the annular band are counted to form a 3×4=12-dimensional basic feature vector, which is then expanded by combining the distance from the feature point to the image center and the relative angle information of the annular band, and finally a 20-dimensional descriptor is generated. In panoramic natural scenery shooting, such as the large field of view features such as the lake shore curve and the mountain outline captured by the fisheye lens, this descriptor can effectively summarize the feature distribution trend within the field of view and the relative position relationship with the image center. Even under different lighting conditions or partial occlusion of the scene, the success rate of similar contour feature matching can still be maintained at more than 80%, enhancing the robustness of fisheye image feature matching.

[0057] S303: Compare the feature descriptor of the fisheye image with the feature descriptor of the center lens image to find all feature point pairs that meet a similarity threshold.

[0058] In this embodiment, the normalized cross correlation (NCC) method is used to calculate the similarity of feature descriptors. For the feature descriptor pairs of the center lens and the fisheye lens, the NCC value is calculated, and the initial similarity threshold is set to 0.7. This threshold is obtained by statistically analyzing the matching experiments of a large number of combined images of different scenes (including indoor, outdoor, static, dynamic and other types), and minimizes mismatches while ensuring a certain number of matches. For example, in the shooting of indoor home scenes, when matching furniture arrangements and decorative details, when the NCC value is greater than 0.7, it can be preliminarily determined as a potential matching feature point pair. After this initial screening, a set of possible matching point pairs can be quickly filtered out from the massive feature point combinations, accounting for about 20%-30% of the total number of combinations, greatly reducing the amount of subsequent precise matching calculations, while retaining most of the real matching point pairs. In the process of calculating NCC, in order to improve the calculation efficiency, the integral image technology is used to quickly obtain the local area pixel sum and square sum to reduce repeated calculations. For a 1000×1000 pixel image, the feature point matching calculation time is shortened by about 40% compared with the traditional pixel-by-pixel calculation method, meeting the needs of imaging systems with high real-time requirements.

[0059] S304: Eliminate incorrectly matched feature point pairs.

[0060] In one embodiment, step S304 specifically includes the following steps: S3041-S3048.

[0061] S3041. Randomly select a part of feature points from all feature point pairs as a feature point pair subset.

[0062] In this embodiment, considering the computational efficiency and model representativeness, a proportional random sampling rule is set. For the full set containing N feature point pairs, the extraction ratio is between 70% and 85% depending on the complexity of the scene and the amount of data. For example, when shooting a relatively simple indoor meeting scene with a relatively uniform feature distribution, N is 500 feature point pairs, and 75%, or 375 feature point pairs, are extracted; for complex and changeable outdoor sports event scenes, the number of feature point pairs may be as high as 2,000. At this time, 80%, or 1,600 point pairs, are extracted as a subset to ensure that subsequent model estimation can cover sufficiently rich image feature information while avoiding inefficiency caused by excessive calculation.

[0063] In the sampling process, the idea of ​​stratified sampling is introduced, and stratification is performed according to the position of feature points in the image (divided into central area, middle ring belt, edge area) and the density of feature points (sparse area, dense area). For example, in the panoramic view of the city taken by drone, more point pairs are extracted from the dense layer of feature points in the downtown area with dense buildings, while fewer points are extracted from the sparse layer such as relatively empty parks. However, it is ensured that there are enough samples in each layer to participate in the initial construction of the model, so that the constructed transformation model takes into account the characteristics of each part of the image from the beginning, improving the universality of the model.

[0064] S3042, constructing a transformation model of the relative position parameters and distortion parameters of the central lens and the four fisheye lenses, and solving the model parameters using the data of the feature point pair subset.

[0065] In this embodiment, a model structure based on perspective transformation is selected to fully consider the complex relative position and imaging distortion relationship between the central lens and the four fisheye lenses. The model parameters include rotation angle (three dimensions, rotation angles θx, θy, θz around the X, Y, and Z axes), translation (Tx, Ty, Tz), scaling factor (Sx, Sy, Sz), and radial distortion coefficients related to lens distortion (k1, k2 for fisheye lens correction compensation) and tangential distortion coefficients (p1, p2), a total of 12 parameters. For example, in a vehicle-mounted multi-lens surround view system, the relative position of the lens will change slightly due to bumps during vehicle driving. These parameters can dynamically capture the real-time geometric changes between lenses to ensure accurate image matching.

[0066] When solving the model parameters using feature point subset data, the objective function is constructed based on the principle of least squares, and the coordinate correspondence between the feature point subset in the source image (fisheye image) and the target image (center lens image) is substituted. The optimal solution of the parameters is solved through an iterative optimization algorithm (such as the Levenberg-Marquardt algorithm). Taking the industrial inspection scene as an example, the imaging matching of multiple feature points on the product surface is carried out. After multiple iterations (generally 10-15 times), the convergence accuracy of each parameter reaches: rotation angle error ±0.1°, translation error ±0.5mm, scaling factor error ±0.02, distortion coefficient error ±0.001. This lays a solid foundation for the subsequent accurate reprojection error calculation.

[0067] S3043. Calculate the reprojection errors of all matched feature point pairs according to the transformation model.

[0068] S3044. According to the reprojection error, the matched feature points are divided into inner points and outer points. The inner points refer to the points whose reprojection errors are less than the set threshold, and the outer points refer to the points whose reprojection errors are greater than the set threshold.

[0069] For steps S3043-3044, in this embodiment, the reprojection error calculation process is as follows: according to the solved transformation model, the feature point coordinates in the fisheye image are projected to the center lens image coordinate system through model transformation. The Euclidean distance between the projected coordinates and the actual coordinates of the corresponding matching feature points in the center lens image is calculated as the reprojection error. In the intelligent warehouse monitoring scenario, for feature points such as shelves and cargo stacks, the reprojection error is calculated one by one, and the calculation formula is:

[0070]

[0071] Where (x proj ,y proj ) is the projection coordinate, (x actual ,x actual ) are the actual coordinates.

[0072] When setting the reprojection error threshold, the lens resolution, scene scale, and imaging accuracy requirements are comprehensively considered. For high resolution (such as more than 50 million pixels), small scenes (such as laboratory precision instrument shooting) and high-precision matching requirements, the threshold is set to 1-2 pixels; for low resolution, large scenes (such as high-altitude panoramic monitoring of cities), the threshold is relaxed to 3-5 pixels. For example, in the security monitoring of museum cultural relics, in order to accurately capture the displacement details of cultural relics, the threshold is set to 1.5 pixels, which effectively distinguishes between reliable matches and mismatched points, and ensures the integrity of cultural relics in subsequent image fusion.

[0073] Traverse all matching feature point pairs, mark those with reprojection errors less than the set threshold as inliers and include them in subsequent model optimization; those with reprojection errors greater than the threshold are judged as outliers (mismatches) and temporarily stored for subsequent analysis (for statistical mismatch reasons, etc.). For example, in outdoor construction monitoring, after reprojection error screening, about 60%-70% of the initial matching point pairs are confirmed as inliers, and outliers are mainly caused by feature extraction deviations caused by dust at the construction site, equipment occlusion, and light shadows. Subsequent targeted elimination of these outliers can greatly improve the reliability of image registration.

[0074] S3045. Re-estimate the transformation model using interior points.

[0075] S3046: Update the transformation model according to the iteration requirement to obtain multiple transformation models.

[0076] For steps S3045-S3046, in this embodiment, the transformation model parameters are re-estimated using the newly selected set of inliers, and the least squares method combined with the optimization algorithm is used again to solve. At this time, since the inliers more accurately reflect the true correspondence of the image, the model parameter update converges to a more accurate direction. Taking the robot visual navigation scene as an example, the lens imaging is affected by the dynamic environment during the movement of the robot. Each re-estimation of the model can adjust the parameters in real time. For example, the perspective deformation of the fisheye lens is aggravated due to approaching obstacles. The radial distortion coefficient and translation parameters are corrected in time through the inliers feedback to ensure the accuracy of visual positioning. After iteration, the model parameters are reduced by about 30%-40% relative to the previous round of errors.

[0077] Set an upper limit on the number of iterations, generally 3-5 times, to prevent excessive iterations from falling into local optimality or affecting real-time performance due to waste of computing resources. At the same time, introduce dynamic termination conditions to monitor the rate of change of model parameters between two adjacent iterations. When the rate of change of each parameter is less than the set minimum value (such as the rate of change of the rotation angle is less than 0.01° / time, and the rate of change of the translation amount is less than 0.1mm / time), terminate the iteration in advance. In the real-time mapping scenario of UAVs, facing the rapid flight and image collection of complex terrain, this efficient iteration strategy can not only ensure the accuracy of the model, but also complete the model update within seconds to adapt to the requirements of real-time data processing.

[0078] S3047. Select a transformation model with the largest number of internal points and the highest matching degree with the layout and imaging characteristics of the central lens and four fisheye lenses from multiple transformation models as the final transformation model, wherein the matching degree of the layout and imaging characteristics is measured by calculating the difference between the model and the geometric optical model of the central lens and the fisheye lens.

[0079] S3048: The feature point pairs that do not conform to the final transformation model are regarded as mismatches and are removed.

[0080] For steps S3047-S3048, in this embodiment, the degree of matching is measured by the difference between the calculation model and the geometric optical model of the center lens and the fisheye lens. Based on the principle of optical imaging, the perspective transformation model and the ideal lens imaging ray tracing model are compared in terms of light propagation path, imaging viewing angle, projection relationship, etc. For example, by calculating the mean deviation of the incident angle of light predicted by the calculation model and the incident angle of the actual lens optical model, as well as the sum of squares of the difference between the imaging size scaling ratio and the theoretical value, the model matching is comprehensively quantified, and the lower the difference, the higher the matching. In the 3D modeling multi-lens acquisition scene, this indicator can accurately screen out the transformation model that best fits the physical laws of real lens imaging, ensuring the geometric consistency of the model in the reconstruction of complex 3D scenes.

[0081] The feature point pairs that do not conform to the final selected optimal transformation model are completely eliminated. In the application of panoramic video stitching, after this strict screening, the "ghosting" and "tearing" phenomena caused by mismatching at the stitching of adjacent frame images are basically eliminated, the picture is smooth and natural, and the stitching accuracy reaches the sub-pixel level, which greatly improves the visual quality of panoramic videos and provides a stable and high-quality image foundation for immersive virtual reality experience, long-term security monitoring and backtracking, etc.

[0082] For steps S3041-S3048, through a sophisticated mismatch elimination process, the selected feature point pairs can accurately reflect the true geometric correspondence between the center lens and the fisheye lens image, so that the subsequent image registration error is controlled within a very small range (sub-pixel level), and seamless connection is achieved in the fields of panoramic stitching and multimodal image fusion, such as the continuous and smooth trajectory of the target in the 360° panoramic picture of security monitoring, without image jitter or dislocation caused by inaccurate registration. In addition, whether the light changes drastically (such as the transition of light and shadow at sunrise and sunset), the scene is highly disturbed by dynamic interference (such as busy traffic intersections) or the state of the lens itself fluctuates (such as the vibration of drone flight), this solution can adaptively adjust the transformation model, eliminate mismatches caused by environmental factors, and ensure imaging stability. In the intelligent traffic violation evidence collection multi-lens system, the characteristics of the illegal vehicle can still be accurately locked under complex road conditions, and the misjudgment rate is less than 2%, which improves the accuracy of law enforcement evidence.

[0083] S40: aligning the corrected fisheye image to the viewing angle of the central lens image based on the motion feature matching result.

[0084] In one embodiment, step S40 specifically includes the following steps: S401 - S403 .

[0085] S401. Based on the matched feature point pairs, relative motion estimation is performed on the corrected fisheye image and the center lens image. During the motion estimation process, the stability of the installation structure of the center lens and the four fisheye lenses and the dynamic changes of the overlapping area of ​​the field of view are combined, and a motion estimation model based on the optical flow method is used for estimation.

[0086] In this embodiment, the stability of the mounting structure of the center lens and the four fisheye lenses has an important influence on the relative motion estimation. For example, in a vehicle-mounted multi-lens imaging system, vibrations will occur during the driving of the vehicle, but the lens is fixed to the vehicle body through a sturdy anti-shake mounting bracket, and its overall relative position remains stable to a certain extent, but there may be slight low-frequency shaking. In the motion estimation model, the constraint information of this mounting structure is added, and reasonable displacement and rotation range limit parameters are set. For example, based on the mechanical properties of the mounting bracket and past experimental data, the translation displacement fluctuation range in the horizontal and vertical directions is set within ±0.5mm, and the rotation angle variation range around each coordinate axis is within ±0.2°. This is used as prior knowledge to filter out optical flow calculation results that do not meet actual physical constraints, thereby improving the accuracy and reliability of motion estimation.

[0087] Since the lenses have a field of view and overlap with each other, in different shooting scenes, the movement of objects in the overlapping area or the change of the lens's own posture will cause the overlapping area to change dynamically. Take the multi-lens system that monitors indoor activities as an example. When a person walks in the overlapping area of ​​the center lens and the fisheye lens, the feature distribution in the overlapping area will change at different times. In the optical flow model, dynamic weights are set for the overlapping area, and higher weights are given to feature point pairs in the overlapping area (such as setting the weight to 0.8 and the weight of feature point pairs in the non-overlapping area to 0.2), so that motion estimation focuses more on feature changes in the overlapping area, because the feature correspondence in these areas is more critical for subsequent image registration and can more accurately reflect the relative motion state between images.

[0088] Assume that in a multi-lens shooting scene at a sports event, the central lens focuses on the movements of the athletes in the center of the stadium, and the fisheye lens covers the spectators around the stadium and the overall situation of the venue. Among the matched feature point pairs, there is a set of billboard corner points corresponding to the side of the stadium. In two adjacent frames of images, the feature-based optical flow method is combined with the above lens characteristic optimization to calculate the horizontal displacement of the feature point pair as 5 pixels (considering the image resolution and the actual venue distance conversion relationship, the corresponding horizontal displacement of the actual venue is about 0.5m), and the vertical displacement is 3 pixels (the actual venue vertical displacement is about 0.3m). At the same time, based on the comprehensive calculation of multiple feature point pairs, it is obtained that the overall small rotation angle around a certain coordinate axis is about 0.1°. These displacement and rotation information constitute the preliminary results of relative motion estimation, which provides key data for the subsequent construction of the geometric transformation matrix.

[0089] S402: construct a geometric transformation matrix based on the motion estimation result and the optical imaging center coordinate relationship and field angle relationship between the central lens and the four fisheye lenses.

[0090] In this embodiment, the central lens and the four fisheye lenses each have different optical imaging center coordinates, and the relative position relationship of these coordinates in three-dimensional space determines the geometric transformation relationship between the images. Through a precise lens calibration process in advance, the coordinate value of the optical imaging center of each lens in a unified coordinate system can be obtained (for example, the imaging center coordinate of the central lens is (0,0,0), and the imaging center coordinate of a fisheye lens is (10mm,5mm,3mm). The coordinate unit here is determined according to the actual calibration accuracy, generally accurate to ±0.1mm). When constructing the geometric transformation matrix, these coordinate differences are incorporated into the calculation of the translation transformation. For example, when calculating the translation matrix elements of the fisheye image to the central lens image perspective transformation, the distance parameters that need to be translated in the X, Y, and Z directions are calculated based on the coordinate difference to ensure that the image can be correctly aligned in space.

[0091] Different lenses have different field of view angles, and the differences in their coverage and angle ranges also need to be reflected in the geometric transformation. For example, the field of view of the center lens is 60° (horizontally), and the field of view of a fisheye lens is 180° (horizontally). When aligning the fisheye image to the center lens image angle, it is necessary to determine the scaling factor and rotation angle and other parameters based on the proportional relationship of the field of view angle and the corresponding relationship of the imaging area. Through the relationship between trigonometric functions and the principle of imaging geometry, the ratio of the fisheye image to be scaled in the horizontal and vertical directions relative to the center lens image (assuming that the horizontal scaling ratio is 0.5 and the vertical scaling ratio is 0.6) and the angle that may need to be rotated (such as a clockwise rotation of 10°) are calculated. These scaling, rotation and other parameters are integrated with the previous translation parameters into a complete affine transformation matrix to construct a geometric transformation matrix that conforms to the actual imaging characteristics of the lens.

[0092] Simplify the two-dimensional plane case (actually it is a three-dimensional space transformation, the principle is similar), assuming that after the above calculation, the translation parameters are (Tx = 10, Ty = 5) (in pixels), the scaling factor is (Sx = 0.8, Sy = 0.7), and the rotation angle is 15° (clockwise), then the constructed affine transformation matrix (expressed in homogeneous coordinate form) is as follows:

[0093] This matrix is ​​the geometric transformation matrix required to align the fisheye image to the central lens image viewing angle, and is used for subsequent geometric transformation operations.

[0094] S403 , performing geometric transformation and image compensation on the corrected fisheye image according to the geometric transformation matrix, so that the corrected fisheye image is aligned with the central lens image in terms of viewing angle.

[0095] In one embodiment, step S403 specifically includes the following steps: S4031 - S4033 .

[0096] S4031, performing coordinate transformation on each pixel point on the corrected fisheye image according to the transformation matrix.

[0097] In this embodiment, coordinate transformation is performed on each pixel of the corrected fisheye image according to the constructed geometric transformation matrix. For example, the pixel with coordinates (x, y) in the fisheye image is multiplied by the above affine transformation matrix (calculated in homogeneous coordinates) to obtain the transformed coordinates (x', y'), and the calculation formula is as follows (simplified in two dimensions):

[0098]

[0099] S4032. Calculate the pixel value of the fisheye image after coordinate transformation using an interpolation method.

[0100] After the coordinate transformation, the newly obtained pixel coordinates are often not integer coordinates, while the pixel values ​​of the image are defined at integer coordinate positions. Therefore, bilinear interpolation can be used to obtain accurate pixel values ​​after the transformation. For example, if the transformed coordinates are calculated to be (10.3, 20.7), the pixel value at the position (10.3, 20.7) is calculated using the pixel values ​​of the four integer coordinate pixels around the pixel ((10, 20), (10, 21), (11, 20), (11, 21)) according to the weights of bilinear interpolation to ensure the continuity and rationality of the pixel values ​​after the geometric transformation of the image, and avoid image jaggedness, blur, and other phenomena.

[0101] S4033. Map the pixel values ​​of the transformed fisheye image to the viewing angle of the central lens image according to the interpolation result.

[0102] After the previous coordinate transformation and interpolation calculation, the fisheye image has been adjusted accordingly in terms of geometry and pixel value, but it still needs to be accurately "placed" in the perspective of the central lens image to achieve complete alignment of the two in perspective. This is the purpose of pixel value mapping.

[0103] In the implementation, the coordinate system of the center lens image is used as the reference, and the pixel values ​​of the fisheye image after transformation and interpolation are filled into the corresponding area according to their corresponding coordinate positions. For example, the coordinate range of the center lens image is from (0,0) to (W center ,H center ), where W center Indicates the number of pixels in the width direction, H center It represents the number of pixels in the height direction. After the transformed pixel coordinates of the fisheye image are appropriately scaled and offset, their pixel values ​​are filled into the coordinate positions corresponding to the central lens image.

[0104] If part of the fisheye image exceeds the coordinate range of the center lens image during the mapping process, it can be processed according to the specific application scenario. For example, in applications such as panoramic stitching, the excess part can be cropped or the boundary processing technology of image fusion can be used to make it transition naturally with the center lens image; in some scenarios where all information needs to be retained, the display range of the center lens image can be expanded or scrolling can be used to display the complete fisheye image after transformation.

[0105] Taking the multi-lens imaging application of drone aerial photography of city panoramas as an example, the central lens focuses on the city's landmark building area, and the fisheye lens captures a large field of view of surrounding streets, buildings, etc. After the above-mentioned coordinate transformation, interpolation and mapping operations, the image information such as distant streets and building outlines captured by the fisheye lens can be accurately integrated into the perspective of the central lens image. For example, a curved street originally at the edge of the fisheye image, after transformation and mapping, naturally appears at the edge of the central lens image in a geometric shape and pixel expression that conforms to the perspective of the central lens, and seamlessly connects with the landmark building image captured by the central lens, forming a complete, continuous and unified perspective city panoramic image, which provides a good foundation for subsequent image fusion, analysis and other operations.

[0106] For steps S4031-S4033, through detailed coordinate transformation, interpolation calculation and perspective mapping steps, it is possible to ensure that the corrected fisheye image and the center lens image are aligned with high precision in perspective. In application scenarios such as panoramic image stitching and multi-lens monitoring systems, the transition between the stitching of images is natural, and there will be no obvious misalignment, gaps or image distortion. This makes the final large-field-of-view high-resolution image more realistic and complete, and can provide better image data support for security monitoring, geographic surveying and mapping, virtual reality and other fields.

[0107] S50: performing fusion processing on the registered images.

[0108] In one embodiment, step S50 specifically includes the following steps: S501 - S508 .

[0109] S501, obtaining the difference in spectral response characteristics between the central lens and the four fisheye lenses.

[0110] The spectral response characteristics of the central lens and the four fisheye lenses can be measured in advance by a high-precision spectrum analyzer. Specifically, a high-precision spectrum analyzer such as the USB4000 model of Ocean Optics is used, whose wavelength measurement range covers the visible light and near-infrared bands (350-1000nm) and the spectral resolution can reach 1.5nm. The central lens and the four fisheye lenses are installed on the fixing fixture of the optical platform in sequence, ensuring that the optical axis of the lens is strictly parallel to the incident optical axis of the spectrum analyzer, and the error is controlled within ±0.1° to ensure the measurement accuracy.

[0111] The measurement environment is set as a darkroom to eliminate stray light interference, the internal temperature is kept constant at 25℃±1℃, and the humidity is maintained at 40%-60% to avoid environmental factors affecting the performance of lens optical components and spectral measurement results.

[0112] For each lens, a standard D65 light source (color temperature 6500K, color rendering index greater than 95) is used as illumination. The light source is uniformly introduced through optical fiber and vertically illuminates the Spectralon standard white board with a diffuse reflectance close to 100%. The lens collects the reflected light from the white board and focuses it on the optical fiber probe of the spectrum analyzer. When measuring each lens, the light intensity values ​​of multiple wavelength points (at least 500 data points evenly distributed in the measurement band) are collected, and the measurement is repeated 5 times to take the average value, which is recorded as the spectral response curve of the lens. For example, the central lens has a higher response in the 550nm green light band, with a light intensity value of 800 count units, while a certain fisheye lens has a relatively low response in this band, with only 600 count units. The differences in the spectral response characteristics of each lens are accurately quantified, providing a key data basis for subsequent color correction.

[0113] S502: Perform color deviation analysis on the images collected by each fisheye lens and the central lens under a standard color chart to construct a color correction matrix.

[0114] S503: Perform color correction on the registered fisheye image and the center lens image according to the color correction matrix.

[0115] For steps S502-S503, select an X-Rite ColorChecker 24-color standard color card and place it in the center of the lens field of view to ensure that the color card fills about 30% of the image area to obtain sufficient color information. Use the center lens and four fisheye lenses to capture the color card image respectively, and set the image resolution to no less than 2000×3000 pixels to ensure that the color card color block boundaries are clearly discernible and each color block sampling point is no less than 100 pixels.

[0116] The captured image is converted into a color space, and the RGB data is converted into a CIE Lab uniform color space, which is more in line with the visual perception characteristics of the human eye and is convenient for accurate quantification of color deviation. For example, in the center lens image, the red block of the standard color card (the theoretical value of CIE Lab coordinates is L*=53.24, a*=80.09, b*=67.20) has measured coordinates of L*=52.80, a*=78.50, b*=65.80, and the difference with the theoretical value is calculated. Similar operations are performed on each color block in each fisheye lens and center lens image to accumulate color deviation data.

[0117] Based on the collected color deviation data, the least squares method is used to construct a 3×3 color correction matrix M. Assume that the RGB value of the original image pixel is the vector [xyz], and the corrected RGB value is [x'y'z']. By solving the linear equation system M×[x yz]=[x'y'z'], the color of the corrected image is as close as possible to the theoretical color of the standard color card. For example, after calculation, it is obtained that an element of the correction matrix M(1,1)=1.02, M(1,2)=-0.03, M(1,3)=0.01, etc., and the matrix is ​​optimized by singular value decomposition to ensure its stability and accuracy, and reduce noise amplification or color distortion during the color correction process. The matrix is ​​used to perform color correction pixel by pixel on the registered fisheye image and the center lens image to restore the true color.

[0118] S504: Obtain the difference in illumination reflection between the central area and the peripheral area.

[0119] S505: Calculate the average brightness of different regions based on the brightness adjustment algorithm of region segmentation, and adjust the brightness according to a preset brightness balance rule.

[0120] For steps S504-S505, in the actual shooting scene, use a light sensor (such as the Hagner E4 all-sky irradiance sensor) to measure the light intensity and reflectivity of the central area and the surrounding area respectively. Place the sensor at the center of the central lens field of view and the typical position of the surrounding fisheye lens field of view (such as the edge, corner, etc.), collect light data at different time periods (such as early morning, noon, evening) and different weather (sunny, cloudy, cloudy), and the measurement time interval for each position is 1 hour. Continuously collect 24 hours of data to take the statistical mean to accurately obtain the light reflection difference characteristics. For example, at noon on a sunny day, the light intensity in the central area reaches 1000 lux and the reflectivity is 0.3, while the light intensity in the surrounding area is only 600 lux and the reflectivity is 0.25, which clearly shows the brightness imbalance.

[0121] The image is divided into a central area, an intermediate transition area, and a peripheral area by using a regional division method based on image grayscale histogram threshold segmentation combined with morphological processing. For example, a bimodal analysis is performed on the image grayscale histogram to determine the thresholds T1=120 and T2=200. Grayscale values ​​less than T1 are divided into peripheral areas, those greater than T2 are divided into central areas, and those between the two are transition areas. For different areas, the average brightness value is calculated (the integral image fast algorithm is used to calculate the sum of the grayscale of regional pixels and then average them). According to the preset brightness balance rules, such as keeping the brightness ratio between the central area and the peripheral area within the range of 1.2-1.5, the brightness of the darker peripheral area is increased through linear stretching, gamma correction and other algorithms, and the gain of the overly bright central area is appropriately reduced to make the overall image brightness uniform and natural.

[0122] S506: Based on the theoretical model of the field of view overlap between the central lens and the four fisheye lenses and the actual imaging calibration results, the lens field of view model is analyzed to calculate the overlapping area.

[0123] According to the optical design parameters (focal length, field of view angle, image plane size, etc.) of the central lens and the four fisheye lenses, a theoretical model of field of view overlap is constructed using the principles of geometric optics. In a three-dimensional space coordinate system, with the intersection of the lens optical axes as the origin, the field of view boundary equations of each lens are calculated through trigonometric function relationships to determine the theoretical overlap area range. For example, it is known that the field of view angle of the central lens is 60°, the field of view angle of the fisheye lens is 180°, and the focal lengths are 50mm and 8mm respectively. The overlap boundary angle at a certain distance from the center in the horizontal direction is calculated, and a mathematical model is established to describe the geometric shape of the overlap area.

[0124] S507: In the overlapping area, the images are fused using a weight allocation strategy, and the weight allocation is dynamically adjusted according to the image quality evaluation index and the importance of the field of view of the central lens and the four fisheye lenses.

[0125] Comprehensively consider image quality factors such as clarity, contrast, and noise level. Clarity uses a gradient-based variance calculation method to calculate the variance of the gradient amplitude for each small area of ​​the image (such as an 8×8 pixel block). The larger the variance, the higher the clarity; contrast is measured by calculating the dynamic range of the image grayscale histogram (the difference between the maximum and minimum values); the noise level is evaluated using the local standard deviation statistic, and a small standard deviation indicates low noise. For example, the clarity index value of the central area of ​​the center lens image reaches 80 (normalized to the range of 0-100), the contrast is 40, and the noise level is 5, while the corresponding indicators of the peripheral areas of a fisheye lens are 60, 30, and 8, respectively, quantifying the image quality differences in each area.

[0126] Determine the importance of the field of view according to the shooting target and application scenario. In the scene of security monitoring focusing on key areas, the importance of the field of view of the center lens is set to 0.7, and the sum of the importance of the field of view of the surrounding fisheye lenses is 0.3; in the panoramic display scene, appropriately increase the weight of the field of view of the fisheye lens, such as 0.4 for the center lens and 0.15 for the average weight of the fisheye lens. Combined with the image quality index, dynamically allocate the fusion weight. If the clarity of a fisheye lens image suddenly increases in the overlapping area (such as due to good local light), increase its weight accordingly so that the fused image can use the best imaging part of each lens in real time to generate a high-quality fusion result. The weighted average fusion algorithm is used to calculate the fused RGB value of the pixels in the overlapping area according to the weight to achieve a smooth transition of the image.

[0127] S508: Smoothing the edge of the fusion area.

[0128] The Sobel operator is combined with the morphological expansion and corrosion operation to detect the edge of the fusion area, and a transition zone with an edge width of about 3-5 pixels is obtained. For example, at the splicing of panoramic images, the boundary pixel position is determined by edge detection, and the two sides of the edge are expanded by 2 pixels to form a transition area. The focus is on smoothing this area to avoid abrupt fusion marks.

[0129] A smoothing method based on Gaussian weighted average is adopted. For each pixel point in the transition area, a 5×5 neighborhood window is set with it as the center. The weight of the neighborhood pixels is calculated according to the Gaussian distribution. The closer to the center pixel, the greater the weight. The neighborhood pixels are weighted averaged according to the weights to calculate the new pixel value and replace the original pixel value. The algorithm is iterated multiple times (usually 3-5 times) to make the brightness and color gradient of the fused edge natural, eliminate the splicing gaps and color mutations, and improve the overall visual quality of the image.

[0130] For steps S501-S508, by accurately measuring the lens spectral response and color card calibration, the color deviation of images collected by different lenses is greatly reduced, and the compensation of light reflection differences and brightness adjustment make the overall brightness of the image uniform, without obvious contrast between bright and dark areas, and the visual comfort is significantly improved. For example, when monitoring night scenes, the center is too bright and the surroundings are too dark, which causes the target to be lost, or when displaying indoor panoramic views, a natural light distribution effect is created, which makes it easier for the human eye to capture image details and improve the viewing experience. In addition, the overlapping area is determined, the weight is reasonably allocated, and the edges are smoothed to achieve perfect image splicing and fusion, and the gaps, misalignments, and color brightness jumps at the splicing are basically eliminated.

[0131] S60: Post-process the fused image to obtain a final image.

[0132] In one embodiment, step S60 specifically includes the following steps: S601 - S604 .

[0133] S601: Use an edge detection algorithm to identify gaps and discontinuous areas in the fused image.

[0134] In this embodiment, the Canny edge detection algorithm is selected as the basis, and is optimized for the characteristics of multi-lens fusion images. Since the fused image may have weak edge traces left over from previous processing such as registration and color correction, as well as complex situations where the edges of objects in the actual scene are mixed, the high and low thresholds are adaptively adjusted based on the traditional Canny algorithm. First, the local grayscale gradient histogram of the image is calculated, and the high threshold is dynamically set to the 80th percentile of the gradient amplitude and the low threshold is the 20th percentile based on the histogram distribution. This can capture strong edges and connect weak edges through the low threshold to form a complete outline. For example, in a panoramic city street view image, the edges at the junction of buildings and the sky and at the joints of different buildings can be accurately located through this optimization, and the edge positioning accuracy can reach the sub-pixel level, with the error controlled at ±0.3 pixels.

[0135] Combined with multi-scale analysis technology, the Gaussian pyramid is used to decompose the image at multiple resolutions, detect edges at different scales, and then fuse the results of each scale to enhance the detection ability of gaps of different sizes and edges of discontinuous areas. For example, for small defects such as gaps between leaves and discontinuous architectural decorative lines, they can be clearly located at a high-resolution scale; and large discontinuous areas such as gaps between large billboards and building wall splicing can be quickly outlined at a low-resolution coarse scale, comprehensively improving the comprehensiveness and accuracy of detection.

[0136] S602: Using a texture synthesis algorithm, generate a filling texture according to the texture features of the area surrounding the gap, and fill the gap and discontinuous area.

[0137] In this embodiment, the local binary pattern (LBP) is used in combination with the grayscale co-occurrence matrix (GLCM) to extract texture features, with the area around the gap as the focus. LBP is used to describe the local micro-texture structure of the image, while GLCM quantifies the spatial correlation of pixel grayscale to obtain feature information such as texture direction, roughness, and contrast. For example, in a multi-lens fusion landscape image containing grass and a stone road, the gap is located in the stone road area. By analyzing the surrounding normal stone texture, its grayscale periodic variation law (grayscale co-occurrence frequency in a specific direction in GLCM) and local texture pattern (LBP eigenvalue distribution) are extracted, and it is determined that the main direction of the texture is approximately horizontal, and the average roughness is 0.4 (normalized value) and other key features.

[0138] According to the extracted texture features, appropriate sample blocks are selected around the gap for filling. A search strategy based on feature similarity is adopted, and a threshold is set, such as the area with a texture feature Euclidean distance less than 0.5 (normalized distance) as a candidate sample block. Sample blocks adjacent to the gap position and with coherent texture are given priority to ensure a natural transition of the filling texture. For example, when filling the gap of a stone slab road, multiple 8×8 pixel sample blocks are selected from the adjacent complete stone slab area, whose texture features have a high degree of matching with the gap to avoid introducing abrupt textures.

[0139] Using the texture synthesis method based on image blocks, the selected sample blocks are filled into the gap area one by one according to the best matching principle. The PatchMatch algorithm is adopted to randomly initialize the filling position first, and then iteratively search for better matching blocks, and continuously update the filling results. In each iteration, the gradient difference and texture feature similarity between the filling block and the surrounding synthesized area boundary are calculated, and the filling direction and position are optimized to ensure that the texture is continuous and smooth after filling. For example, when filling the gaps in the grass, as the iteration proceeds, the texture of the filled grass leaves gradually merges with the surrounding area. After 5-8 iterations, the growth direction and density of the grass leaves in the gap are consistent with the surrounding area, and it is visually difficult to distinguish the filling traces. The fusion degree of the filled area and the original image exceeds 90%.

[0140] S603: Construct a white balance adjustment matrix by performing color temperature analysis on the white object images collected by the central lens and the four fisheye lenses.

[0141] During the initial calibration and regular calibration of the multi-lens system, images of standard white objects (such as Spectralon whiteboards, white areas of Kodak gray cards, etc.) are specifically taken to ensure that the white objects are evenly illuminated, the light intensity is controlled at 500-1000 lux, the color temperature is close to the D65 (6500K) light source conditions, and the image resolution is not less than 2000×2000 pixels to avoid inaccurate color sampling due to uneven light or low resolution.

[0142] The collected white object images are preprocessed with grayscale world hypothesis correction to make the average grayscale value of the image close to the ideal neutral gray (such as 128), remove the initial color deviation caused by factors such as exposure deviation, and facilitate subsequent accurate color temperature analysis. For example, the initial average grayscale of a white board image taken by a fisheye lens is 110, which reaches 127 after linear stretch correction, improving the basic color accuracy.

[0143] The color temperature estimation algorithm based on Planck's blackbody radiation law is used to analyze the relative radiation energy distribution of white object images in different bands (such as 400-700nm visible light band, sampling at intervals of 10nm), and the lens color temperature is estimated by matching with the standard color temperature spectrum curve. For example, the color temperature of a white object photographed by the center lens is calculated to be 5800K, while that of a fisheye lens is 6200K, which deviates from the standard D65 color temperature.

[0144] A 3×3 white balance adjustment matrix is ​​constructed based on the color temperature differences of each lens. Using the VonKries diagonal model and taking the standard D65 color temperature as the benchmark, the gain adjustment coefficients of the red, green, and blue channels of each lens are calculated and filled into the diagonal elements of the matrix. For example, for a fisheye lens with a high color temperature, the gain of the blue channel is appropriately reduced (such as the coefficient is set to 0.9), the gain of the red channel is slightly increased (such as the coefficient is set to 1.05), and the green channel remains relatively stable (the coefficient is about 1.0), ensuring that the color of images from different lenses is unified in the white balance dimension.

[0145] S604: Perform color balance adjustment on the filled image according to the white balance adjustment matrix.

[0146] For the filled fused image, perform white balance adjustment matrix multiplication in units of pixels. Assume that the RGB value of the image pixel is the vector [xyz], and the adjusted RGB value is [x'y'z']. The color value is changed pixel by pixel by [x'y'z'] = white balance adjustment matrix × [xy z]. For example, the original RGB value of a pixel in the image is [100120110]. After adjustment by the corresponding lens white balance matrix, it becomes [103120107]. The color temperature difference is accurately compensated, so that the white area presents a true neutral white, and other colors are also balanced and coordinated. The color error rate (compared with the standard color) is reduced by about 80%.

[0147] During the adjustment process, in order to avoid numerical overflow or precision loss due to matrix multiplication, the pixel values ​​are dynamically limited and normalized. Ensure that the adjusted RGB value is within the legal range of 0-255. If it exceeds the range, it will be truncated or linearly scaled to maintain the rich color levels of the image, prevent color block or color fault phenomenon, and ensure high-quality color presentation of the final image.

[0148] The present invention can effectively cope with complex scenes such as weak textures, drastic or dynamic lighting changes, etc. by performing distortion correction processing on the fisheye lens and performing feature matching and registration with the central lens, thereby improving robustness and imaging quality.

[0149] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0150] The embodiment of the present invention also provides a multi-lens large field of view high resolution imaging device, which is used to perform the steps in any embodiment of the multi-lens large field of view high resolution imaging method described above. Figure 2 , Figure 2A schematic block diagram of a multi-lens, large-field-of-view, high-resolution imaging device 100 provided in an embodiment of the present application is shown. The multi-lens, large-field-of-view, high-resolution imaging device 100 specifically includes:

[0151] The image acquisition unit 110 is used to synchronously acquire the images captured by the central lens and each fisheye lens. The distortion correction unit 120 is used to perform distortion correction processing on the images captured by each fisheye lens based on the camera calibration parameters to obtain a corrected fisheye image. The motion feature matching unit 130 is used to perform motion feature matching on the corrected fisheye image and the central lens image. The image registration unit 140 is used to register the corrected fisheye image to the viewing angle of the central lens image based on the motion feature matching result. The fusion processing unit 150 is used to perform fusion processing on the registered image. The post-processing unit 160 is used to perform post-processing on the fused image to obtain the final image.

[0152] In one embodiment, the distortion correction unit 120 is specifically used to: obtain camera calibration parameters of each fisheye lens; select a distortion correction algorithm to correct the collected fisheye image; and generate a corrected fisheye image.

[0153] In one embodiment, the motion feature matching unit 130 is specifically used to: perform feature point detection on the corrected fisheye image and the center lens image using a feature detector; for each detected feature point, calculate the feature descriptors of the pixels surrounding it, wherein the resolution-related information between pixels is enhanced when calculating the feature descriptor of the center lens image, and the field of view-related information is enhanced when calculating the feature descriptor of the fisheye image; compare the feature descriptors of the fisheye image with the feature descriptors of the center lens image to find all feature point pairs that meet the similarity threshold; and eliminate the feature point pairs that are incorrectly matched.

[0154] In one embodiment, the motion feature matching unit 130 is further specifically used to: randomly select a part of feature points from all feature point pairs as a feature point pair subset; construct a transformation model of the relative position parameters and distortion parameters of the central lens and the four fisheye lenses, and solve the model parameters using the data of the feature point pair subset; calculate the reprojection errors of all matched feature point pairs according to the transformation model; divide the matched feature point pairs into inner points and outer points according to the reprojection errors, wherein the inner points refer to points whose reprojection errors are less than a set threshold, and the outer points refer to points whose reprojection errors are greater than the set threshold; use the inner points to re-estimate the transformation model; update the transformation model according to the iteration requirements to obtain multiple transformation models; select the transformation model with the largest number of inner points and the highest matching degree with the layout and imaging characteristics of the central lens and the four fisheye lenses from the multiple transformation models as the final transformation model, wherein the matching degree of the layout and imaging characteristics is measured by calculating the difference between the model and the geometric optical model of the central lens and the fisheye lens; and regard the feature point pairs that do not conform to the final transformation model as mismatches and eliminate them.

[0155] In one embodiment, the image registration unit 140 is specifically used to: perform relative motion estimation on the corrected fisheye image and the center lens image based on the matched feature point pairs, and during the motion estimation process, combine the stability of the installation structure of the center lens and the four fisheye lenses and the dynamic changes in the field of view overlapping area, and use a motion estimation model based on the optical flow method to perform estimation; construct a geometric transformation matrix based on the motion estimation result and the optical imaging center coordinate relationship and field of view angle relationship between the center lens and the four fisheye lenses; perform geometric transformation and image compensation on the corrected fisheye image based on the geometric transformation matrix, so that the corrected fisheye image is aligned with the center lens image in terms of viewing angle.

[0156] In one embodiment, the image registration unit 140 is further specifically used to: perform coordinate transformation on each pixel point on the corrected fisheye image according to the transformation matrix; calculate the pixel value of the fisheye image after the coordinate transformation using the interpolation method; and map the pixel value of the transformed fisheye image to the viewing angle of the central lens image according to the interpolation result.

[0157] In one embodiment, the fusion processing unit 150 is specifically used to: obtain the difference in spectral response characteristics between the central lens and the four fisheye lenses; perform color deviation analysis on the images captured by each fisheye lens and the central lens under a standard color card to construct a color correction matrix; perform color correction on the registered fisheye image and the central lens image according to the color correction matrix; obtain the difference in light reflection between the central area and the peripheral area; calculate the average brightness of different areas based on a brightness adjustment algorithm based on area segmentation, and adjust the brightness according to a preset brightness balance rule; analyze the lens field of view model based on the field of view overlap theoretical model of the central lens and the four fisheye lenses and the actual imaging calibration results to calculate the overlapping area; in the overlapping area, use a weight distribution strategy to fuse the images, and the weight distribution is dynamically adjusted according to the image quality evaluation index and field of view importance of the central lens and the four fisheye lenses; and smooth the edges of the fusion area.

[0158] In one embodiment, the post-processing unit 160 is further specifically used to: use an edge detection algorithm to identify gaps and discontinuous areas in the image after fusion processing; use a texture synthesis algorithm to generate a filling texture based on the texture features of the area around the gap, and fill the gaps and discontinuous areas; construct a white balance adjustment matrix by performing color temperature analysis on the white object images captured by the central lens and four fisheye lenses; and adjust the color balance of the filled image according to the white balance adjustment matrix.

[0159] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned multi-lens large field of view and high-resolution imaging device 100 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0160] The multi-lens large field of view high resolution imaging device can be implemented in the form of a computer program. The computer program can be used in Figure 3 Runs on the computer device shown.

[0161] See also Figure 3 , Figure 3 700 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 700 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0162] like Figure 3 As shown, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-lens large field of view and high-resolution imaging method as described above are implemented.

[0163] The computer device 700 includes a processor 720 , a memory, and a network interface 750 connected via a system bus 710 , wherein the memory may include a non-volatile storage medium 730 and an internal memory 740 .

[0164] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. When the computer program 732 is executed, the processor 720 can execute a multi-lens large field of view high-resolution imaging method.

[0165] The processor 720 is used to provide computing and control capabilities and support the operation of the entire computer device 700.

[0166] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute a multi-lens large field of view high-resolution imaging method.

[0167] The network interface 750 is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 700 to which the present application scheme is applied. The specific computer device 700 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. The processor 720 is used to run the program code stored in the memory to implement a multi-lens large field of view high-resolution imaging method.

[0168] Those skilled in the art will understand that Figure 3 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Figure 3 The embodiments shown are consistent and will not be described again here.

[0169] It should be understood that in the embodiment of the present application, the processor 720 may be a central processing unit (CPU), and the processor 720 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0170] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the multi-lens large field of view high-resolution imaging method disclosed in the embodiment of the present invention is implemented.

[0171] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0172] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-lens large field of view high-resolution imaging method, characterized in that: The multi-lens includes a central lens and four fisheye lenses, the four fisheye lenses are geometrically distributed around the central lens, and the field of view angles of the central lens and the four fisheye lenses overlap each other, the central lens is used to capture an image of a central area, and the optical resolution of the central lens is higher than that of the fisheye lenses, and the fisheye lenses are used to capture a large field of view image of a peripheral area; the method includes: Synchronously acquire images collected by the central lens and each fisheye lens; Based on the camera calibration parameters, the images captured by each fisheye lens are subjected to distortion correction processing to obtain a corrected fisheye image; Perform motion feature matching on the corrected fisheye image and the center lens image; The corrected fisheye image is registered to the viewing angle of the central lens image based on the motion feature matching result; Perform fusion processing on the registered images; The fused image is post-processed to obtain the final image.

2. The multi-lens large field of view high-resolution imaging method according to claim 1, characterized in that: The performing feature matching on the corrected fisheye image and the center lens image comprises: Using a feature detector to perform feature point detection on the rectified fisheye image and the center lens image; For each detected feature point, the feature descriptor of the surrounding pixels is calculated, wherein the resolution-related information between pixels is enhanced when calculating the feature descriptor of the center lens image, and the field of view-related information is enhanced when calculating the feature descriptor of the fisheye image; Compare the feature descriptors of the fisheye image with those of the center lens image and find all feature point pairs that meet the similarity threshold; Eliminate the wrongly matched feature point pairs.

3. The multi-lens large field of view high-resolution imaging method according to claim 2, characterized in that: The step of eliminating mismatched feature point pairs includes: Randomly select a part of feature points from all feature point pairs as feature point pair subsets; Construct a transformation model of the relative position parameters and distortion parameters of the central lens and the four fisheye lenses, and solve the model parameters using the data of the feature point pair subset; According to the transformation model, the reprojection errors of all matching feature point pairs are calculated; According to the reprojection error, the matched feature points are divided into inliers and outliers, wherein the inliers refer to the points whose reprojection error is less than the set threshold, and the outliers refer to the points whose reprojection error is greater than the set threshold; Re-estimate the transformation model using inliers; The transformation model is updated according to iteration requirements to obtain multiple transformation models; Selecting the transformation model with the largest number of internal points and the highest matching degree with the layout and imaging characteristics of the central lens and four fisheye lenses from multiple transformation models as the final transformation model, wherein the matching degree of the layout and imaging characteristics is measured by calculating the difference between the model and the geometric optical models of the central lens and the fisheye lens; Feature point pairs that do not conform to the final transformation model are regarded as mismatches and are removed.

4. The multi-lens large field of view high-resolution imaging method according to claim 1, characterized in that: The method of registering the corrected fisheye image to the viewing angle of the central lens image based on the motion feature matching result includes: According to the matched feature point pairs, the relative motion of the corrected fisheye image and the center lens image is estimated. During the motion estimation process, the stability of the installation structure of the center lens and the four fisheye lenses and the dynamic changes of the overlapping area of ​​the field of view are combined, and the motion estimation model based on the optical flow method is used for estimation; According to the motion estimation results, a geometric transformation matrix is ​​constructed by combining the optical imaging center coordinate relationship and the field angle relationship between the central lens and the four fisheye lenses; The corrected fisheye image is geometrically transformed and image compensated according to the geometric transformation matrix so that the corrected fisheye image is aligned with the central lens image in terms of viewing angle.

5. The multi-lens large field of view high-resolution imaging method according to claim 4, characterized in that: The step of performing geometric transformation and image compensation on the corrected fisheye image according to the geometric transformation matrix so as to align the corrected fisheye image with the central lens image in terms of viewing angle comprises: Perform coordinate transformation on each pixel point on the corrected fisheye image according to the transformation matrix; Use interpolation method to calculate the pixel value of the fisheye image after coordinate transformation; According to the interpolation result, the pixel values ​​of the transformed fisheye image are mapped to the viewing angle of the central lens image.

6. The multi-lens large field of view high-resolution imaging method according to claim 1, characterized in that: The step of fusing the registered images comprises: Obtain the differences in spectral response characteristics between the central lens and the four fisheye lenses; Perform color deviation analysis on the images collected by each fisheye lens and the center lens under a standard color chart to construct a color correction matrix; Performing color correction on the registered fisheye image and the center lens image according to the color correction matrix; Obtain the difference in light reflection between the central area and the surrounding area; The brightness adjustment algorithm based on region segmentation calculates the average brightness of different regions and adjusts the brightness according to the preset brightness balance rules; Based on the theoretical model of the field of view overlap between the central lens and the four fisheye lenses and the actual imaging calibration results, the lens field of view model is analyzed to calculate the overlapping area; In the overlapping area, the images are fused using a weight allocation strategy, and the weight allocation is dynamically adjusted according to the image quality evaluation indicators and field of view importance of the central lens and the four fisheye lenses; Smooths the edges of the blended area.

7. The multi-lens large field of view high-resolution imaging method according to claim 1, characterized in that: The post-processing of the fused image to obtain a final image includes: Use edge detection algorithms to identify gaps and discontinuous areas in the fused image; Using texture synthesis algorithm, fill texture is generated according to the texture features of the area around the gap, and the gap and discontinuous area are filled; The white balance adjustment matrix is ​​constructed by analyzing the color temperature of white object images collected by the central lens and four fisheye lenses; Performs color balance adjustment on the filled image according to the white balance adjustment matrix.

8. A multi-lens, large-field-of-view, high-resolution imaging device, characterized in that: The multi-lens includes a central lens and four fisheye lenses, the four fisheye lenses are geometrically distributed around the central lens, and the field of view angles of the central lens and the four fisheye lenses overlap each other, the central lens is used to capture the image of the central area, and the optical resolution of the central lens is higher than that of the fisheye lens, and the fisheye lens is used to capture a large field of view image of the peripheral area; the device includes: An image acquisition unit, used for synchronously acquiring images collected by the central lens and each fisheye lens; A distortion correction unit, used to perform distortion correction processing on the images collected by each fisheye lens based on the camera calibration parameters to obtain a corrected fisheye image; A motion feature matching unit, used for performing motion feature matching on the corrected fisheye image and the center lens image; An image registration unit, for registering the corrected fisheye image to the viewing angle of the central lens image based on the motion feature matching result; A fusion processing unit, used for performing fusion processing on the registered images; The post-processing unit is used to perform post-processing on the fused image to obtain a final image.

9. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a multi-lens large field of view high-resolution imaging method as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes a multi-lens large field of view high-resolution imaging method as described in any one of claims 1 to 7.

Images