Panoramic image generation method, apparatus, device, and storage medium

Abstract

The application provides a panoramic image generation method and device, equipment and a storage medium. The method comprises single-lens calibration of a wide-angle camera according to a checkerboard calibration method to obtain a distortion correction matrix; time-space synchronous acquisition of the wide-angle camera and an array camera is controlled to obtain an initial panoramic image and a sub-image set; non-linear distortion correction of the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without distortion; sub-image feature information of each sub-image in the sub-image set is obtained, and the corresponding projection position, perspective transformation matrix and scale transformation matrix of each sub-image in the standardized panoramic image are obtained according to the sub-image feature information; a panoramic canvas coordinate system is established according to the standardized panoramic image, each sub-image is projected into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, the corresponding size of the sub-image is adjusted in the panoramic canvas coordinate system according to the scale transformation matrix, and a panoramic image is generated according to the adjusted panoramic canvas coordinate system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a panoramic image generation method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of computer vision technology, array camera technology has been widely used in many fields due to its ability to generate high-resolution, high signal-to-noise ratio, and high dynamic range images through the cooperative work of multiple cameras. Array cameras can effectively improve image quality and provide important data support for 3D reconstruction and panoramic imaging by capturing images from different angles through multiple cameras. However, array cameras need to solve a key problem in practical applications: how to accurately calibrate and stitch the images captured by multiple cameras to generate high-quality panoramic images.

[0003] In the prior art, the calibration of array cameras usually requires individual calibration of each camera, which involves a complex chessboard calibration process. The calibration of each camera requires obtaining its intrinsic and extrinsic parameters, including focal length, principal point, distortion parameters, and the relative position and attitude relationship between cameras, through a chessboard image. However, this individual calibration method has the following significant problems: first, the calibration workload is huge, especially when the array camera contains multiple cameras, the calibration process is time-consuming and tedious; second, due to the small differences in the calibration parameters of each camera, the matching error between sub-images is large, and the generated panoramic image has obvious stitching seams, geometric distortion, or color inconsistency, which seriously affects the quality and visual effect of the final image.

[0004] Therefore, there is an urgent need for a panoramic image generation method to solve at least one of the above problems. SUMMARY

[0005] The present application provides a panoramic image generation method, device, equipment and storage medium, aiming to solve the problem that in the prior art, the calibration of array cameras usually requires individual calibration of each camera, which involves a complex chessboard calibration process and has a huge calibration workload, and the matching error between sub-images is large, which affects the quality and visual effect of the final image.

[0006] In a first aspect, the present application provides a panoramic image generation method, comprising:

[0007] According to the chessboard calibration method, single-lens calibration is performed on the wide-angle camera to obtain the distortion correction matrix corresponding to the wide-angle camera;

[0008] The wide-angle camera and the array camera are controlled to perform time-space synchronous acquisition, to obtain an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; each sub-image in the sub-image set has an overlapping area with the initial panoramic image;

[0009] The initial panoramic image is subjected to non-linear distortion correction according to the distortion correction matrix, to generate a standardized panoramic image without geometric distortion.

[0010] Sub-image feature information of each sub-image in the sub-image set is obtained, and a projection position, a perspective transformation matrix and a scale transformation matrix of each sub-image in the standardized panoramic image are obtained according to the sub-image feature information.

[0011] A panoramic canvas coordinate system is established according to the standardized panoramic image, each sub-image is projected into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and the projection position, and the corresponding size of each sub-image in the panoramic canvas coordinate system is adjusted according to the scale transformation matrix, and a panoramic image is generated according to the adjusted panoramic canvas coordinate system.

[0012] In some embodiments, the non-linear distortion correction of the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion comprises: obtaining corresponding radial distortion coefficients and tangential distortion coefficients in the distortion correction matrix; obtaining a mapping relationship between original pixel points and corrected pixel points; performing inverse mapping calculation on all pixel points of the initial panoramic image according to the mapping relationship, to generate a standardized panoramic image eliminating barrel distortion according to bilinear interpolation.

[0013] In some embodiments, the projection position, the perspective transformation matrix and the scale transformation matrix of each sub-image in the standardized panoramic image are obtained according to the sub-image feature information, which comprises: obtaining a local feature point set of each sub-image and a global feature point set of the standardized panoramic image according to a preset image feature detector; performing neighbor feature matching on each sub-image to obtain a corresponding relationship between the local feature point set and the global feature point set; filtering a matching point pair satisfying homography constraint from the local feature point set according to the corresponding relationship, and obtaining a homography matrix corresponding to the matching point pair according to least square method; decomposing the homography matrix to obtain the perspective transformation matrix containing rotation and translation components and the scale transformation matrix containing a scaling factor; and determining a projection center coordinate of the sub-image in the standardized panoramic image according to a decomposition result corresponding to an eigenvalue of the homography matrix.

[0014] In some embodiments, the establishing a panorama canvas coordinate system according to the normalized panorama image comprises: taking a geometric center of the normalized panorama image as a coordinate origin, establishing a compound coordinate system of a polar coordinate system and a Cartesian coordinate system; determining a coverage range of the panorama canvas coordinate system according to a field of view angle parameter of the wide-angle camera; establishing a mapping relationship from a pixel coordinate to a panorama coordinate; setting a canvas boundary constraint condition in the Cartesian coordinate system, and establishing the panorama canvas coordinate system according to the compound coordinate system, the coverage range and the mapping relationship.

[0015] In some embodiments, the projecting each of the sub-images into the panorama canvas coordinate system according to the corresponding perspective transformation matrix and the projection position comprises: performing affine transformation on a pixel point set of each of the sub-images according to a translation vector corresponding to the perspective transformation matrix and the projection position; establishing a dynamic weight fusion region in the panorama canvas coordinate system; calculating a projection confidence weight of each of the sub-images in an overlapping projection region corresponding to the overlapping region; performing consistency optimization on a pixel gradient field of the overlapping projection region according to a Poisson fusion algorithm, eliminating a joint effect between the sub-images, and completing the projection of each of the sub-images into the panorama canvas coordinate system.

[0016] In some embodiments, the adjusting a corresponding size of each of the sub-images in the panorama canvas coordinate system according to the scale transformation matrix comprises: analyzing an anisotropic scaling factor corresponding to the scale transformation matrix; calculating a sub-image resolution adjustment ratio according to the anisotropic scaling factor; performing multiple interpolation resampling on the sub-image to adjust an original resolution corresponding to the sub-image to a preset resolution; determining a length-width ratio constraint condition corresponding to the sub-image according to a tolerance threshold determined according to a mounting position tolerance of the array camera, so as to keep the length-width ratio of the sub-image in the scale transformation process in conformity with the length-width ratio constraint condition.

[0017] In some embodiments, the single-lens calibration of the wide-angle camera according to the checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera comprises: arranging a plurality of checkerboard calibration boards in different poses in a field of view range of the wide-angle camera; acquiring a calibration board image corresponding to each of the calibration boards collected by the wide-angle camera, and using Zhang's calibration method to solve a camera intrinsic parameter matrix and a distortion coefficient vector; and constructing the distortion correction matrix according to the camera intrinsic parameter matrix and the distortion coefficient vector.

[0018] In a second aspect, the present application provides a panorama image generation device, the device comprising:

[0019] a lens calibration unit configured to calibrate a wide-angle camera according to a checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera;

[0020] The synchronization acquisition unit is configured to control the wide-angle camera and the array camera to perform time-space synchronization acquisition, and acquire an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; each sub-image in the sub-image set has an overlapping area with the initial panoramic image.

[0021] The distortion correction unit is configured to perform non-linear distortion correction on the initial panoramic image according to the distortion correction matrix, and generate a standardized panoramic image without geometric distortion.

[0022] The feature acquisition unit is configured to acquire sub-image feature information of each sub-image in the sub-image set, and acquire a projection position, a perspective transformation matrix and a scale transformation matrix of each sub-image in the standardized panoramic image according to the sub-image feature information.

[0023] The canvas establishment unit is configured to establish a panoramic canvas coordinate system according to the standardized panoramic image, project each sub-image into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image according to the adjusted panoramic canvas coordinate system.

[0024] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program and implement the panoramic image generation method provided in any embodiment of the present application when executing the computer program.

[0025] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program; the computer program is executed by a processor to make the processor implement the panoramic image generation method provided in any embodiment of the present application.

[0026] The present application provides a panoramic image generation method, device, equipment and storage medium. The panoramic image generation method provided by the present application significantly improves the efficiency and quality of panoramic image generation by optimizing the calibration process and image registration strategy. The method only calibrates the checkerboard of the wide-angle camera to obtain the distortion correction matrix, and saves the separate calibration of each sub-camera of the array camera. Through non-linear distortion correction, the initial panoramic image acquired by the wide-angle camera is converted into a standardized image without geometric distortion, which serves as a global reference benchmark. The traditional method needs to calibrate each camera in the array one by one, while the present method only needs to calibrate a single lens, greatly simplifying the process.

[0027] Meanwhile, the method ensures the consistency of the data of the wide-angle camera and the array camera in time (the same moment of shooting) and space (fixed relative position) by controlling the synchronous acquisition of the wide-angle camera and the array camera. The wide-angle camera provides a global initial panoramic image, and the array camera provides a high-resolution local sub-image set. The overlapping area of the sub-image and the initial panoramic image provides a basis for subsequent registration. The synchronous mechanism avoids misplacement of dynamic scenes and directly provides a global coordinate framework through the wide-angle image.

[0028] The method further determines the projection position, perspective transformation matrix (homography matrix), and scale transformation matrix of the sub-image by extracting feature information (such as SIFT, ORB, etc.) of the sub-image and matching the feature information with the standardized panoramic image. By projecting into a unified panoramic canvas coordinate system, the size and viewing angle of the sub-image are dynamically adjusted to realize seamless splicing. The global information of the wide-angle image is used to guide the registration of the sub-image, avoiding the error accumulation problem in traditional methods that relies on pairwise matching between sub-images.

[0029] Finally, the method establishes a coordinate system based on the standardized panoramic image, projects the sub-image into the canvas through perspective transformation (to solve the viewing angle difference) and scale transformation (to unify the resolution), and combines a fusion algorithm (such as multi-band mixing) to eliminate the seams and generate a high-resolution panoramic image. The size and viewing angle of the sub-image are dynamically adjusted to ensure the balance between local details and global consistency.

[0030] The provided method has at least the following beneficial effects:

[0031] 1. Significantly reduces the calibration complexity: the traditional method needs to calibrate each sub-camera in the array camera individually, and the present method only needs to calibrate a single wide-angle lens, saving more than 90% of the calibration time and computing resources, especially suitable for large multi-camera array systems.

[0032] 2. Improves registration accuracy and image quality: the standardized panoramic image of the wide-angle camera is used as a global reference, and the sub-image registration is based on the same coordinate system, avoiding the cumulative error of traditional pairwise matching between sub-images, significantly reducing the splicing misplacement and ghosting problems.

[0033] 3. Enhances system robustness: the spatio-temporal synchronous acquisition ensures the consistency of the sub-image and the panoramic image in dynamic scenes, avoiding motion blur or object displacement caused by shooting time difference, and improving the splicing success rate of complex scenes (such as pedestrians and vehicles).

[0034] 4. Optimizes resource utilization: the wide-angle camera provides a low-resolution global view, and the array camera provides a high-resolution local sub-image. Through intelligent fusion, the resolution and coverage range of the panoramic image are considered, reducing the dependence on the hardware performance of a single camera.

[0035] 5. Suitable for large-scale application scenarios: can be extended to unmanned aerial photography, vehicle surround view, VR shooting and other fields, especially in scenarios that require rapid deployment (such as temporary monitoring systems), saving the cumbersome multi-camera calibration steps and improving practicality.

[0036] In summary, the method provided solves the pain points of complex calibration and large matching errors in traditional panorama generation through single-lens calibration, spatio-temporal synchronous acquisition and global-local collaborative registration strategy, realizes efficient and high-precision panorama image generation, and has wide application prospects and commercial value.

[0037] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 is a step schematic flow chart of the panorama image generation method provided by an embodiment of the present application;

[0040] Figure 2 is a structure schematic block diagram of the panorama image generation device provided by an embodiment of the present application;

[0041] Figure 3 is a structure schematic block diagram of the computer device provided by an embodiment of the present application.

[0042] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] The flow chart shown in the drawings is only an example description, and does not necessarily include all contents and operations / steps, nor does it necessarily execute in the order described. For example, some operations / steps can be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation.

[0045] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same or similar items with basically the same function and role. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. also do not necessarily mean different.

[0046] It should be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0047] It should also be understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations.

[0048] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] With the rapid development of computer vision technology, array camera technology has been widely used in many fields due to its ability to generate high-resolution, high signal-to-noise ratio, and high dynamic range images through the cooperative work of multiple cameras. Array cameras can effectively improve image quality and provide important data support for 3D reconstruction and panoramic imaging by capturing images from different angles through multiple cameras. However, array cameras need to solve a key problem in practical application: how to accurately calibrate and stitch the images captured by multiple cameras to generate high-quality panoramic images.

[0050] In the prior art, the calibration of array cameras usually needs to calibrate each camera separately, which involves a complex chessboard calibration process. The calibration of each camera needs to obtain its intrinsic and extrinsic parameters, including focal length, principal point, distortion parameters, and the relative position and attitude relationship between cameras, through a chessboard image. However, this method of calibration one by one has the following significant problems: first, the calibration workload is huge, especially when the array camera contains multiple cameras, the calibration process is time-consuming and tedious; second, due to the small differences in the calibration parameters of each camera, the matching error between sub-images is large, and the panoramic image generated by stitching will have obvious stitching seams, geometric distortion, or color inconsistency, which seriously affects the quality and visual effect of the final image.

[0051] In addition, in the prior art, even if the subgraphs are corrected and spliced by calibrating parameters, the splicing process still relies on complex image registration algorithms, and it is difficult to ensure seamless connection between subgraphs in actual application. Especially in dynamic scenes or complex lighting conditions, the geometric transformation relationship and lighting consistency between subgraphs are difficult to accurately match, resulting in problems such as distortion, ghosting or blur in the spliced panoramic image.

[0052] To solve the above problems, the present application proposes an innovative panoramic image generation method, which only calibrates the wide-angle camera, uses the synchronous acquisition data of the wide-angle camera and the array camera, combines feature matching and perspective transformation matrix, and realizes fast calibration and splicing of all subgraphs. This method not only significantly reduces the calibration workload and avoids the cumbersome process of traditional individual calibration method, but also improves the accuracy and efficiency of subgraph splicing through feature matching and transformation matrix optimization, and finally generates a high-resolution, distortion-free panoramic image. This method has significant innovation in technology, which can effectively solve the problems of calibration complexity and splicing error in the prior art, and provides a more efficient and reliable solution for the application of array cameras.

[0053] Please refer to Figure 1 The present application provides a panoramic image generation method. The method is run on a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a notebook computer, a wearable device, or a robot, etc.

[0054] As Figure 1 shown, the provided panoramic image generation method includes steps S101 to S105. Details are as follows:

[0055] Step S101. Calibrate the wide-angle camera according to the checkerboard calibration method to obtain the distortion correction matrix corresponding to the wide-angle camera.

[0056] Specifically, the checkerboard calibration method is a geometric feature-based calibration method that calculates the intrinsic parameters (such as focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (such as camera position and attitude) of the camera by analyzing the corner point positions in the checkerboard image. In the calibration process, the known size of the checkerboard and the accurate extraction of the corner points are critical.

[0057] Unlike traditional methods, this step only calibrates the wide-angle camera, avoiding the cumbersome process of calibrating multiple cameras one by one. The wide-angle camera serves as a reference, and its calibration parameters will be used for subsequent correction and splicing of subgraphs. The distortion coefficients (such as radial distortion and tangential distortion) obtained by calibration are used to generate a distortion correction matrix to eliminate geometric distortion in the image.

[0058] As a chessboard calibration plate with high-precision printing is used to ensure the accuracy of the chessboard size and the clarity of the pattern. Align the wide-angle camera to the chessboard and take multiple images from different angles and distances, ensuring that the chessboard is evenly distributed in the image. Use image processing algorithms (such as the findChessboardCorners function in OpenCV) to extract the chessboard corner points and calculate the sub-pixel level accuracy. Use the calibration algorithm (such as the calibrateCamera function in OpenCV) to calculate the camera's intrinsic parameters and distortion coefficients. According to the distortion coefficients, generate a distortion correction matrix for subsequent image correction.

[0059] The steps only need to calibrate one wide-angle camera, significantly reducing the complexity and time cost of calibration. Through multi-angle shooting and accurate calculation, the accuracy of the calibration parameters is ensured, providing a reliable foundation for subsequent image processing. Single-lens calibration avoids the tedious process of calibrating multiple cameras one by one, especially suitable for large-scale array camera systems.

[0060] Step S102. Control the wide-angle camera and the array camera to perform time-space synchronous acquisition, obtain an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; each sub-image in the sub-image set has an overlapping area with the initial panoramic image.

[0061] Specifically, through hardware or software triggering mechanisms, ensure that the wide-angle camera and the array camera capture images at the same time and in the same space to ensure the time consistency and spatial consistency between images. The wide-angle camera captures a wide-angle image as the basis for the panoramic image, covering the entire scene. The array camera captures multiple sub-images, each covering a part of the initial panoramic image and ensuring sufficient overlapping areas for subsequent stitching.

[0062] By using hardware triggers or software instructions, ensure that the wide-angle camera and the array camera capture images simultaneously. The wide-angle camera captures an initial panoramic image with a wide angle, covering the entire scene. The array camera captures multiple sub-images, each covering a part of the initial panoramic image and ensuring sufficient overlapping areas between adjacent sub-images. Store the captured images and perform preliminary noise removal and format conversion.

[0063] Time-space synchronous acquisition ensures that all images are captured in the same environment, reducing errors caused by time or space differences. The initial panoramic image and the sub-image set provide necessary data support for subsequent stitching and correction. The synchronous acquisition mechanism improves the efficiency of data acquisition, especially suitable for dynamic scenes.

[0064] Step S103. Perform non-linear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion.

[0065] Specifically, the distortion correction matrix obtained in step S101 is used to perform geometric distortion correction on the initial panoramic image to eliminate radial distortion and tangential distortion in the image. The corrected panoramic image has the characteristics of no distortion and geometric consistency, facilitating subsequent stitching and processing.

[0066] If the initial panoramic image is geometrically transformed using the distortion correction matrix, the distortion is eliminated. The bilinear interpolation or bicubic interpolation algorithm is applied to smooth the corrected image to ensure image quality. The corrected image is saved as a standardized panoramic image as a reference for subsequent processing. The corrected image has no geometric distortion, improving the geometric accuracy of the image. The standardized panoramic image provides a unified basis for subsequent stitching and projection, reducing stitching errors. Through interpolation processing, the corrected image is smooth and distortion-free.

[0067] Step S104. Obtain the sub-image feature information of each sub-image in the sub-image set, and obtain the corresponding projection position, perspective transformation matrix and scale transformation matrix of each sub-image in the standardized panoramic image according to the sub-image feature information.

[0068] Specifically, the feature points (such as SIFT and SURF feature points) and descriptors of each sub-image are extracted for matching and positioning. The projection position is the corresponding position of the sub-image in the standardized panoramic image. The perspective transformation matrix is a transformation matrix for projecting the sub-image into the standardized panoramic image, ensuring geometric consistency. The scale transformation matrix is used to adjust the size of the sub-image to match the size of the standardized panoramic image.

[0069] Feature extraction is performed on each sub-image to obtain feature points and descriptors. The feature points of the sub-image are matched with the feature points of the standardized panoramic image to find the corresponding relationship. The perspective transformation matrix and the scale transformation matrix are calculated according to the matching results. The projection position of the sub-image in the standardized panoramic image is determined.

[0070] Through feature matching and transformation matrix calculation, the sub-image is accurately aligned with the standardized panoramic image. The application of perspective transformation and scale transformation matrix reduces the geometric distortion and size inconsistency in the stitching process. Feature extraction and matching algorithm realizes automatic processing, improving efficiency.

[0071] Step S105. Establish a panoramic canvas coordinate system according to the standardized panoramic image, project each sub-image into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the corresponding size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image according to the adjusted panoramic canvas coordinate system.

[0072] Specifically, the panorama canvas coordinate system is a coordinate system established based on the standardized panorama image, which is used to unify the projection and stitching of all sub-images. Each sub-image is projected into the panorama canvas coordinate system according to the perspective transformation matrix and the projection position, and the size is adjusted according to the scale transformation matrix to ensure that the stitched image is seamlessly connected. After stitching all sub-images, a final high-resolution, distortion-free panorama image is generated.

[0073] As the basis of the standardized panorama image, the panorama canvas coordinate system is established. Each sub-image is projected into the panorama canvas coordinate system according to the perspective transformation matrix and the projection position. The size of the sub-image is adjusted according to the scale transformation matrix to ensure consistency with the panorama canvas coordinate system. The stitched image is smoothed using a multi-band fusion or Poisson fusion algorithm to eliminate stitching seams and inconsistent lighting. The fused image is saved as the final panorama image.

[0074] Through projection and adjustment, the seamless connection of sub-images in the panorama canvas coordinate system is ensured, and the stitching seams and geometric distortion are reduced. The final generated panorama image has the characteristics of high resolution, distortion-free, and color consistency, significantly improving the image quality and visual effect. The automated stitching and fusion algorithm improves the processing efficiency and is suitable for large-scale image processing.

[0075] In summary, the panorama image generation method of the present application realizes the rapid calibration and stitching of array camera sub-images by only calibrating the wide-angle camera, combining feature matching and perspective transformation matrix. This method significantly reduces the calibration workload, improves the stitching accuracy and efficiency, and finally generates a high-resolution, distortion-free panorama image. This method has significant innovation in technology, which can effectively solve the problems of calibration complexity and stitching error in the prior art, and provides a more efficient and reliable solution for the application of array cameras. Through further refinement and improvement, the implementation process of this method is more clear, and the technical details are more rich, providing stronger guidance and operability for practical application.

[0076] In some embodiments, the non-linear distortion correction of the initial panorama image according to the distortion correction matrix to generate a standardized panorama image without geometric distortion includes: obtaining corresponding radial distortion coefficients and tangential distortion coefficients in the distortion correction matrix; obtaining the mapping relationship between the original pixel points and the corrected pixel points; traversing all pixel points of the initial panorama image to perform inverse mapping calculation according to the mapping relationship to generate a standardized panorama image that eliminates barrel distortion according to bilinear interpolation.

[0077] In step S101, the distortion correction matrix of the wide-angle camera is obtained by the chessboard calibration method, which contains the radial distortion coefficients (such as k1, k2, k3) and the tangential distortion coefficients (such as p1, p2). These coefficients describe the distortion characteristics of the pixel points in the image, and the radial distortion mainly causes the barrel or pillow distortion of the image edge, while the tangential distortion is caused by the non-parallelism of the lens and the sensor.

[0078] According to the preset distortion model, the corrected pixel points (x', y') corresponding to each original pixel point (x, y) are calculated. The distortion model formula is as follows:

[0079] x' = x (1 + k1r^2 + k2r^4 + k3r^6) + 2p1xy + p2(r^2 + 2x^2);

[0080] y' = y (1 + k1r^2 + k2r^4 + k3r^6) + p1(r^2 + 2y^2) + 2p2xy; where r^2 = x^2 + y^2. x, y are pixel coordinates in the original image. k1, k2, k3 are radial distortion coefficients, which are used to describe the barrel or pillow distortion of the image edge. r is the distance of the pixel point to the center of the image.

[0081] Map from the corrected image space (x', y') to the original image space (x, y) in reverse to avoid the appearance of holes in the corrected image. For each corrected pixel point, calculate its corresponding position in the original image according to the distortion model.

[0082] Since the reverse mapping may result in non-integer coordinates, the bilinear interpolation algorithm is used to calculate the gray value of the corrected pixel point. The bilinear interpolation formula is as follows:

[0083] f(x, y) = (1 - a) (1 - b) f(x1, y1) + a (1 - b) f(x2, y1) + (1 - a) b f(x1, y2) + a b f(x2, y2); where x1, y1 are the top-left pixel coordinates, x2, y2 are the bottom-right pixel coordinates, a and b are the interpolation weights. f(x, y) is the gray value of the interpolated pixel point.

[0084] By correcting the radial distortion and tangential distortion, a standardized panoramic image without geometric distortion is generated, which significantly improves the geometric accuracy of the image. The bilinear interpolation algorithm ensures that the corrected image is smooth and distortion-free, preserving the details of the original image. The standardized panoramic image provides a unified basis for subsequent sub-image stitching and projection, reducing the stitching error. The reverse mapping and interpolation algorithm realizes automatic correction and improves the processing efficiency.

[0085] In some embodiments, the obtaining, according to the subgraph feature information, a projection position, a perspective transformation matrix and a scale transformation matrix of each subgraph in the standardized panoramic image comprises: obtaining a local feature point set of each subgraph and a global feature point set of the standardized panoramic image according to a preset image feature detector; performing a nearest neighbor feature matching on each subgraph to obtain a corresponding relationship between the local feature point set and the global feature point set; screening, according to the corresponding relationship, a matching point pair of the local feature point set that satisfies a homography constraint, and obtaining a homography matrix corresponding to the matching point pair according to a least square method; decomposing the homography matrix to obtain the perspective transformation matrix containing a rotation and translation component and the scale transformation matrix containing a scaling factor; and determining a projection center coordinate of the subgraph in the standardized panoramic image according to a decomposition result corresponding to an eigenvalue of the homography matrix.

[0086] Feature detection algorithms (such as SIFT, SURF or ORB) are used to extract feature points and their descriptors from the subgraph and the standardized panoramic image. The local feature point set is the set of feature points extracted from the subgraph image. The global feature point set is the set of feature points extracted from the standardized panoramic image. Feature matching algorithms (such as FLANN or brute-force matching) are used to calculate the matching relationship between the subgraph feature points and the standardized panoramic image feature points. In the matching process, the distance between the feature descriptors is calculated, and the most matching feature point pair is screened out. RANSAC algorithm is used to screen out the matching point pair that satisfies the homography constraint, and to eliminate the false matching points. According to the screened matching point pair, the homography matrix H is solved by the least square method, and the homography matrix describes the projection relationship between the subgraph and the standardized panoramic image.

[0087] The singular value decomposition (SVD) is performed on the homography matrix H to obtain the rotation matrix R, the translation vector T and the scaling factor s. The perspective transformation matrix M is composed of the rotation matrix and the translation vector, which is used to describe the projection transformation of the subgraph in the standardized panoramic image. The scale transformation matrix S is composed of the scaling factor, which is used to adjust the size of the subgraph. According to the eigenvalue decomposition result of the homography matrix, the projection center coordinates (xc, yc) of the subgraph in the standardized panoramic image are calculated, which are used to determine the specific position of the subgraph in the panoramic image.

[0088] Through the feature detection and matching algorithm, the accurate alignment of the subgraph and the standardized panoramic image is ensured, and the stitching error is reduced. The perspective transformation matrix and the scale transformation matrix obtained by the decomposition of the homography matrix ensure the geometric consistency and size matching of the subgraph in the projection process. The feature extraction, matching and matrix decomposition algorithms realize the automation processing and improve the efficiency. Through the accurate projection and adjustment, the seamless connection of the subgraph in the panoramic image is ensured, and the stitching seam and geometric distortion are reduced. The RANSAC algorithm is used to effectively eliminate the false matching points, which improves the robustness and reliability of the algorithm.

[0089] In some embodiments, the establishing a panorama canvas coordinate system according to the standardized panorama image comprises: taking the geometric center of the standardized panorama image as a coordinate origin, establishing a composite coordinate system of a polar coordinate system and a Cartesian coordinate system; determining a coverage range of the panorama canvas coordinate system according to a field of view angle parameter of the wide-angle camera; establishing a mapping relationship of pixel coordinates to panorama coordinates; setting a canvas boundary constraint condition in the Cartesian coordinate system, and establishing the panorama canvas coordinate system according to the composite coordinate system, the coverage range and the mapping relationship.

[0090] The geometric center point (xc, yc) of the standardized panorama image is determined and taken as a coordinate origin. A composite coordinate system is established, in which a polar coordinate system is used to describe the angle and distance of a pixel point, and a Cartesian coordinate system is used to describe the horizontal and vertical positions of the pixel point.

[0091] The conversion relationship of the composite coordinate system is as follows:

[0092] θ = arctan[(y - yc) / (x - xc)]; x, y are pixel coordinates in the standardized panorama image. R is the distance of the pixel point to the geometric center.

[0093] The field of view angle (FOV) parameter of the wide-angle camera is obtained to determine the coverage range of the panorama canvas coordinate system. For example, if the horizontal field of view angle of the wide-angle camera is 180 degrees, the horizontal coverage range of the panorama canvas coordinate system is 90-90 degrees.

[0094] The pixel coordinates (x, y) of the standardized panorama image are mapped to panorama coordinates (r, θ). The mapping relationship is as follows:

[0095] θ = arctan[(y - yc) / (x - xc)]; Rmax is the maximum radius of the standardized panorama image, used for normalizing the distance.

[0096] The canvas boundary constraint condition is set in the Cartesian coordinate system to ensure that the coverage range of the panorama canvas coordinate system is consistent with the field of view angle of the wide-angle camera. According to the composite coordinate system, the coverage range and the mapping relationship, the panorama canvas coordinate system is established, which is used for unifying the projection and stitching of all sub-images.

[0097] By establishing the composite coordinate system, the coordinate systems of the standardized panorama image and the sub-images are unified, which facilitates subsequent projection and stitching. The coverage range is determined according to the field of view angle parameter of the wide-angle camera to ensure that the panorama canvas coordinate system can completely cover the entire scene. The mapping relationship of pixel coordinates to panorama coordinates simplifies the coordinate conversion process and improves the processing efficiency. The canvas boundary constraint condition ensures the rationality and integrity of the panorama canvas coordinate system, avoiding image overflow or loss.

[0098] In some embodiments, the projecting each of the subgraphs into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and the projection position comprises: performing affine transformation on the pixel point set of each subgraph according to the translation vector corresponding to the perspective transformation matrix and the projection position; establishing a dynamic weight fusion region in the panoramic canvas coordinate system; calculating the projection confidence weight of each subgraph in the overlapping projection region corresponding to the overlapping region; and performing consistency optimization on the pixel gradient field of the overlapping projection region according to a Poisson fusion algorithm, eliminating the joint effect between the subgraphs, and completing the projection of each of the subgraphs into the panoramic canvas coordinate system.

[0099] The perspective transformation matrix M corresponding to the subgraph and the translation vector T corresponding to the projection position are obtained. Affine transformation is performed on each pixel point (x, y) of the subgraph to project it into the panoramic canvas coordinate system. The overlapping region of the subgraph and the panoramic canvas coordinate system is determined, and a dynamic weight fusion region is established. The dynamic weight fusion region is used to process the overlapping part between the subgraphs to ensure seamless splicing. The projection confidence weight of the subgraph in the overlapping projection region is calculated according to the change of the pixel value. The confidence weight formula is: wi=1 / σi^2; σi is the pixel variance of the subgraph in the overlapping region (the projection confidence weight of the i-th subgraph is wi, and the pixel variance is σi). The pixel gradient field of the overlapping projection region is optimized for consistency using the Poisson fusion algorithm to ensure seamless connection between the subgraphs. The Poisson fusion algorithm adjusts the pixel value of the overlapping region by solving the Poisson equation to make it smoothly transition with the surrounding area. Finally, the optimized subgraph is projected into the panoramic canvas coordinate system to generate a seamless panoramic image.

[0100] Through affine transformation, the accurate projection of the subgraph in the panoramic canvas coordinate system is ensured, and geometric distortion is reduced. The dynamic weight fusion and Poisson fusion algorithm eliminate the joint effect between the subgraphs, realizing seamless splicing. The Poisson fusion algorithm ensures the consistency of the pixel value of the overlapping region, improving the quality and visual effect of the panoramic image. The dynamic weight calculation and Poisson fusion algorithm realize automatic processing, improving efficiency and reliability. The confidence weight and Poisson fusion algorithm enhance the robustness of the algorithm, which is suitable for image stitching under complex scenes and lighting conditions.

[0101] In some embodiments, the adjusting the size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix comprises: analyzing anisotropic scaling factors corresponding to the scale transformation matrix; calculating a sub-image resolution adjustment ratio according to the anisotropic scaling factors; performing multiple interpolation resamplings on the sub-image to adjust the original resolution of the sub-image to a preset resolution; determining a tolerance threshold according to the installation position tolerance of the array camera, and determining a aspect ratio constraint condition of the sub-image according to the tolerance threshold to keep the aspect ratio of the sub-image in the scale transformation process consistent with the aspect ratio constraint condition.

[0102] Specifically, the scale transformation matrix S is decomposed to obtain its scaling factors sx and sy in the horizontal and vertical directions. The anisotropic scaling factors represent the resolution adjustment ratios of the sub-image in different directions. The resolution adjustment ratio of the sub-image is calculated as follows: x-direction adjustment ratio = sx / sref; y-direction adjustment ratio = sy / sref; sref is the reference resolution. The sub-image is resampled using an interpolation algorithm (such as bilinear interpolation or bicubic interpolation) to adjust its resolution to the preset resolution. Multiple interpolation resamplings ensure that the adjusted image is smooth and distortion-free. According to the installation position tolerance of the array camera, a tolerance threshold for aspect ratio adjustment is determined. The aspect ratio of the sub-image is ensured to meet the constraint condition during adjustment: |sy / sx - original aspect ratio| ≤ tolerance threshold. When adjusting the resolution, the aspect ratio of the sub-image is ensured to change by no more than the tolerance threshold, avoiding image distortion. The original aspect ratio is the original aspect ratio of the sub-image before the resolution is adjusted. It is calculated from the original width / original height. It is used to ensure that the difference between the adjusted aspect ratio and the original aspect ratio does not exceed the tolerance threshold during the scale transformation process.

[0103] Through the anisotropic scaling factors and interpolation resampling, the resolution adjustment of the sub-image is ensured to be accurate and smooth. The aspect ratio constraint condition avoids image distortion during the scale transformation process, ensuring the geometric consistency of the image. Multiple interpolation resamplings and aspect ratio constraint conditions improve the quality and visual effect of the adjusted image. Analyzing the scaling factors and interpolation resampling algorithm realizes automated processing, improving efficiency. The tolerance threshold and aspect ratio constraint condition enhance the robustness of the algorithm, which is suitable for complex scenes and installation errors.

[0104] In some embodiments, the single-lens calibration of the wide-angle camera according to the checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera comprises: arranging a plurality of checkerboard calibration boards with different poses within the field of view range of the wide-angle camera; acquiring a calibration board image corresponding to each of the calibration boards collected by the wide-angle camera; using Zhang's calibration method to solve a camera intrinsic parameter matrix and a distortion coefficient vector; and constructing the distortion correction matrix according to the camera intrinsic parameter matrix and the distortion coefficient vector.

[0105] Multiple checkerboard calibration boards are arranged at different angles and positions within the field of view of the wide-angle camera, ensuring that the calibration boards cover the entire field of view. The pose of each calibration board should be as diverse as possible to improve the accuracy of the calibration. Use a wide-angle camera to take images of each calibration board, ensuring that the calibration board is clearly visible in the image. Take multiple images covering different angles and distances. Use Zhang's calibration method to extract the corner points of the calibration board image and calculate the intrinsic matrix K and distortion coefficient vector D of the camera. The intrinsic matrix K includes parameters such as focal length, principal point coordinates, and distortion coefficient vector D includes radial and tangential distortion coefficients. According to the intrinsic matrix and distortion coefficient vector, construct the distortion correction matrix C. The distortion correction matrix is used for subsequent image distortion correction to eliminate geometric distortion in the image.

[0106] The arrangement of multiple-angle and diversified calibration boards improves the accuracy of the calibration, ensuring the accuracy of the calibration parameters. Single-lens calibration avoids the tedious process of calibrating multiple cameras one by one, and is particularly suitable for large-scale array camera systems. Zhang's calibration method and the construction of the distortion correction matrix realize automatic processing and improve efficiency. The application of the distortion correction matrix eliminates the geometric distortion in the image, improves the image quality and geometric accuracy. The arrangement of multiple-angle and diversified calibration boards enhances the robustness of the algorithm and is suitable for complex scenes and installation errors.

[0107] In some embodiments, a pre-trained deep learning model is used to extract features of subgraphs and normalized panoramic images: a pre-trained convolutional neural network (such as VGG, ResNet) is used to extract high-dimensional features of subgraphs and normalized panoramic images. The image is input into the network, and the feature map of the last convolutional layer is extracted as the feature descriptor. Calculate the feature similarity between the subgraph and the normalized panoramic image through the feature matching network: use a feature matching network (such as Siamese network or MatchNet) to calculate the similarity between the subgraph features and the normalized panoramic image features. By calculating the cosine similarity or Euclidean distance between the feature vectors, the best matching feature point pair is found. Use a graph optimization algorithm to optimize the matching results and eliminate false matches: model the matching results as a graph structure and use a graph optimization algorithm (such as maximum flow minimum cut or spectral clustering) to eliminate false matches. By optimizing the structure of the matching graph, high-quality matching point pairs are retained. Recalculate the perspective transformation matrix and scale transformation matrix based on the optimized matching point pairs: use the optimized matching point pairs to recalculate the perspective transformation matrix and scale transformation matrix using the least squares method. Ensure the accuracy of projection and size adjustment.

[0108] The high-dimensional features extracted by the deep learning model have stronger discrimination ability, improving the accuracy of feature matching. The graph optimization algorithm effectively eliminates false matching points, improving the reliability of the matching result. The deep learning model and the graph optimization algorithm realize automatic processing, improving the efficiency. The generalization ability of the deep learning model and the robustness of the graph optimization algorithm are suitable for image stitching in complex scenes and lighting conditions.

[0109] In some embodiments, the pixel motion vector between the subgraph and the normalized panoramic image is calculated using the optical flow method: the pixel motion vector between the subgraph and the normalized panoramic image is calculated using the optical flow algorithm (such as Lucas-Kanade or Horn-Schunck). By analyzing the motion of the pixels, the correspondence between the subgraph and the panoramic image is found. The projection position and the perspective transformation matrix of the subgraph are dynamically adjusted according to the pixel motion vector: the projection position and the perspective transformation matrix of the subgraph are dynamically adjusted according to the pixel motion vector calculated by the optical flow method. Ensure the accurate alignment of the subgraph and the panoramic image in the dynamic scene. Use a dynamic weight fusion algorithm to process the overlapping area in the dynamic scene: in the dynamic scene, use a dynamic weight fusion algorithm to process the overlapping area between the subgraphs. According to the pixel motion vector and the time consistency, the fusion weight is dynamically adjusted. Generate a seamless stitched panoramic image in a dynamic scene: through dynamic adjustment and fusion, generate a seamless stitched panoramic image in a dynamic scene. Ensure the real-time and consistency of image stitching in dynamic scenes.

[0110] The optical flow method can capture the pixel motion in the dynamic scene, ensuring the real-time and consistency of image stitching. The projection position and the perspective transformation matrix are dynamically adjusted, improving the accuracy of image stitching in dynamic scenes. The dynamic weight fusion algorithm ensures the seamless stitching of images in dynamic scenes. The optical flow method and the dynamic weight fusion algorithm enhance the robustness of the algorithm, which is suitable for complex dynamic scenes.

[0111] In some embodiments, the subgraph and the normalized panoramic image are decomposed into multiple frequency bands: the subgraph and the normalized panoramic image are decomposed into multiple frequency bands using a multi-band decomposition algorithm (such as Laplacian pyramid or wavelet transform). Each frequency band represents a different frequency component of the image. Feature matching and projection are performed separately on different frequency bands: feature matching and projection are performed separately on each frequency band to ensure accurate alignment of different frequency components. Through multi-band processing, the stitching error of high-frequency and low-frequency components is reduced. Use a multi-band fusion algorithm to fuse the stitching results of different frequency bands: use a multi-band fusion algorithm (such as Laplacian pyramid fusion or wavelet fusion) to fuse the stitching results of different frequency bands. Through multi-band fusion, the stitching seam and lighting inconsistency problems are eliminated. Generate a high-quality seamless stitched panoramic image: through multi-band fusion, generate a high-quality seamless stitched panoramic image. Ensure the consistency of high-frequency details and low-frequency backgrounds of the image.

[0112] The multi-band decomposition and fusion algorithm reduces the stitching error of high-frequency and low-frequency components, and improves the stitching quality. The multi-band fusion algorithm eliminates the stitching seam and the problem of inconsistent lighting, realizing seamless stitching. The multi-band processing preserves the high-frequency details and low-frequency background of the image, improving the visual effect of the image. The multi-band decomposition and fusion algorithm enhances the robustness of the algorithm, and is suitable for image stitching under complex scenes and lighting conditions.

[0113] In some embodiments, semantic segmentation is performed on the subgraphs and the normalized panoramic image using a semantic segmentation model: a pre-trained semantic segmentation model (such as DeepLab or U-Net) is used to perform semantic segmentation on the subgraphs and the normalized panoramic image. The image is segmented into different semantic regions (such as sky, ground, building, etc.). The projection position and perspective transformation matrix of the subgraph are optimized according to the semantic information: the projection position and perspective transformation matrix of the subgraph are optimized according to the semantic segmentation results. Ensure the consistency of different semantic regions in the stitching process. Use a semantic weighted fusion algorithm to process the overlapping area: in the overlapping area, use a semantic weighted fusion algorithm to process the stitching of different semantic regions. According to the weight of the semantic region, dynamically adjust the fusion strategy. Generate a seamless panoramic image with consistent semantics: through semantic segmentation and weighted fusion, generate a seamless panoramic image with consistent semantics. Ensure the natural transition of different semantic regions in the stitching process. The semantic segmentation and weighted fusion algorithm ensures the consistency of different semantic regions in the stitching process, improving the naturalness of the image. The semantic weighted fusion algorithm eliminates the stitching seam and the problem of inconsistent semantics, realizing seamless stitching. Semantic segmentation preserves the semantic information of the image, improving the visual effect and practicality of the image. The semantic segmentation and weighted fusion algorithm enhances the robustness of the algorithm, and is suitable for image stitching under complex scenes and semantic conditions.

[0114] The provided panoramic image generation method significantly improves the efficiency and quality of panoramic image generation by optimizing the calibration process and image registration strategy. The method only calibrates the checkerboard for the wide-angle camera to obtain its distortion correction matrix, eliminating the need for individual calibration of each sub-camera in the array camera. Through non-linear distortion correction, the initial panoramic image captured by the wide-angle camera is converted into a standardized image without geometric distortion, serving as a global reference. Traditional methods require individual calibration of each camera in the array, while this method only requires single-lens calibration, significantly simplifying the process.

[0115] At the same time, the method controls the synchronous acquisition of the wide-angle camera and the array camera to ensure the consistency of their data in time (same time shooting) and space (fixed relative position). The wide-angle camera provides a global initial panoramic image, and the array camera provides a high-resolution set of local subgraphs. The overlapping area between the subgraphs and the initial panoramic image provides the basis for subsequent registration. The synchronous mechanism avoids misplacement in dynamic scenes, and provides a global coordinate framework directly through the wide-angle image.

[0116] The method further extracts feature information (such as SIFT, ORB, etc. feature points) of the subgraph, matches with the standardized panoramic graph, determines the projection position, perspective transformation matrix (homography matrix) and scale transformation matrix of the subgraph. By projecting to a unified panoramic canvas coordinate system, dynamically adjusting the size and viewing angle of the subgraph, seamless splicing is realized. Using the global information of the wide-angle image to guide the subgraph registration, the error accumulation problem in the traditional method of relying on two-by-two matching between subgraphs is avoided.

[0117] Finally, the method establishes a coordinate system with a standardized panoramic graph, projects the subgraph to the canvas through perspective transformation (solves the viewing angle difference) and scale transformation (unifies the resolution), combines with a fusion algorithm (such as multi-band mixing) to eliminate the joint, and generates a high-resolution panoramic image. Dynamically adjust the size and viewing angle of the subgraph to ensure the balance between local details and global consistency.

[0118] The provided method has at least the following beneficial effects:

[0119] 1. Significantly reduce the calibration complexity: the traditional method needs to calibrate each sub-camera in the array camera separately, and the present method only needs to calibrate a single wide-angle lens, saving more than 90% of the calibration time and computing resources, especially suitable for large multi-camera array systems.

[0120] 2. Improve registration accuracy and image quality: use the standardized panoramic graph of the wide-angle camera as a global reference, and the subgraph registration is based on the same coordinate system, avoiding the cumulative error of two-by-two matching between traditional subgraphs, significantly reducing the misregistration and ghosting problems.

[0121] 3. Enhance system robustness: time-space synchronous acquisition ensures the consistency of subgraphs and panoramic graphs in dynamic scenes, avoiding motion blur or object displacement caused by shooting time difference, and improving the success rate of splicing in complex scenes (such as pedestrians and vehicles).

[0122] 4. Optimize resource utilization: the wide-angle camera provides a low-resolution global view, and the array camera provides a high-resolution local subgraph. Through intelligent fusion, the resolution and coverage of the panoramic graph are considered, reducing the dependence on the hardware performance of a single camera.

[0123] 5. Suitable for large-scale application scenarios: can be extended to fields such as unmanned aerial vehicle aerial photography, vehicle surround view, VR photography, etc. Especially in scenarios that require rapid deployment (such as temporary monitoring systems), the cumbersome multi-camera calibration steps are saved, and the practicality is improved.

[0124] In summary, the provided method solves the problems of complex calibration and large matching error in traditional panoramic generation through single-lens calibration, time-space synchronous acquisition and global-local collaborative registration strategy, realizes efficient and high-precision panoramic image generation, and has wide application prospect and commercial value.

[0125] Please refer toFigure 2 , Figure 2 is a structural schematic diagram of a panoramic image generation apparatus 200 provided by an embodiment of the present application. The panoramic image generation apparatus 200 is configured to perform the steps of the panoramic image generation method shown in any embodiment of the present application. The panoramic image generation apparatus 200 can be a single server or a server cluster, or the panoramic image generation apparatus 200 can be a terminal, which can be a handheld terminal, a notebook computer, a wearable device, a robot, or the like.

[0126] As shown in Figure 2 , the panoramic image generation apparatus 200 includes:

[0127] a lens calibration unit 201 configured to perform single-lens calibration on a wide-angle camera according to a checkerboard calibration method, and obtain a distortion correction matrix corresponding to the wide-angle camera;

[0128] a synchronous acquisition unit 202 configured to control the wide-angle camera and an array camera to perform time-space synchronous acquisition, and obtain an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; each sub-image in the sub-image set has an overlapping area with the initial panoramic image;

[0129] a distortion correction unit 203 configured to perform non-linear distortion correction on the initial panoramic image according to the distortion correction matrix, and generate a standardized panoramic image without geometric distortion;

[0130] a feature acquisition unit 204 configured to obtain sub-image feature information of each sub-image in the sub-image set, and obtain a projection position, a perspective transformation matrix, and a scale transformation matrix of each sub-image in the standardized panoramic image according to the sub-image feature information;

[0131] a canvas establishment unit 205 configured to establish a panoramic canvas coordinate system according to the standardized panoramic image, project each sub-image into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image according to the adjusted panoramic canvas coordinate system.

[0132] It should be noted that, for the convenience and brevity of description, the specific working processes of the panoramic image generation apparatus 200 and each module described above can be clearly understood by those skilled in the art, and will not be described herein again. Figure 1 The corresponding processes in the panoramic image generation method embodiments described in the corresponding embodiments can be referred to, and will not be described herein again.

[0133] Figure 1 The corresponding panoramic image generation method can be implemented in the form of a computer program, which can run on a device such as a server, a terminal, or the like. Figure 2The device shown is running on.

[0134] Please refer to Figure 3 , Figure 3 is a structural schematic block diagram of a computer device provided by an embodiment of the present application. The computer device comprises a processor, a memory and a network interface connected through a device bus, wherein the memory can comprise a storage medium and an internal memory.

[0135] The storage medium can store an operating device and a computer program. The computer program comprises program instructions which, when executed, can cause the processor to execute Figure 1 any one of the corresponding panoramic image generation methods.

[0136] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0137] The internal memory provides an environment for the execution of the computer program in the non-volatile storage medium, which, when executed by the processor, can cause the processor to execute any one of the panoramic image generation methods.

[0138] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. A specific computer device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0140] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0141] According to the checkerboard calibration method, a single-lens calibration is performed on the wide-angle camera to obtain a distortion correction matrix corresponding to the wide-angle camera.

[0142] controlling the wide-angle camera and the array camera to perform time-space synchronous acquisition, to obtain an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; wherein each sub-image in the sub-image set has an overlapping area with the initial panoramic image;

[0143] performing non-linear distortion correction on the initial panoramic image according to the distortion correction matrix, to generate a standardized panoramic image without geometric distortion;

[0144] obtaining sub-image feature information of each sub-image in the sub-image set, and obtaining a projection position, a perspective transformation matrix and a scale transformation matrix of each sub-image in the standardized panoramic image according to the sub-image feature information;

[0145] establishing a panoramic canvas coordinate system according to the standardized panoramic image, projecting each sub-image into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjusting the corresponding size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generating a panoramic image according to the adjusted panoramic canvas coordinate system.

[0146] In some embodiments, the non-linear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion comprises: obtaining corresponding radial distortion coefficients and tangential distortion coefficients in the distortion correction matrix; obtaining a mapping relationship between original pixel points and corrected pixel points; and performing inverse mapping calculation on all pixel points of the initial panoramic image according to the mapping relationship, to generate a standardized panoramic image eliminating barrel distortion according to bilinear interpolation.

[0147] In some embodiments, the obtaining of the projection position, the perspective transformation matrix and the scale transformation matrix of each sub-image in the standardized panoramic image according to the sub-image feature information comprises: obtaining a local feature point set of each sub-image and a global feature point set of the standardized panoramic image according to a preset image feature detector; performing a nearest neighbor feature matching on each sub-image to obtain a corresponding relationship between the local feature point set and the global feature point set; filtering a matching point pair satisfying a homography constraint from the local feature point set according to the corresponding relationship, and obtaining a homography matrix corresponding to the matching point pair according to a least squares method; decomposing the homography matrix to obtain the perspective transformation matrix containing a rotation and translation component and the scale transformation matrix containing a scaling factor; and determining a projection center coordinate of the sub-image in the standardized panoramic image according to a decomposition result corresponding to an eigenvalue of the homography matrix.

[0148] In some embodiments, the establishing a panorama canvas coordinate system according to the normalized panorama image comprises: taking a geometric center of the normalized panorama image as a coordinate origin, establishing a compound coordinate system of a polar coordinate system and a Cartesian coordinate system; determining a coverage range of the panorama canvas coordinate system according to a field of view angle parameter of the wide-angle camera; establishing a mapping relationship from a pixel coordinate to a panorama coordinate; setting a canvas boundary constraint condition in the Cartesian coordinate system, and establishing the panorama canvas coordinate system according to the compound coordinate system, the coverage range and the mapping relationship.

[0149] In some embodiments, the projecting each of the sub-images into the panorama canvas coordinate system according to the corresponding perspective transformation matrix and the projection position comprises: performing affine transformation on a pixel point set of each of the sub-images according to a translation vector corresponding to the perspective transformation matrix and the projection position; establishing a dynamic weight fusion region in the panorama canvas coordinate system; calculating a projection confidence weight of each of the sub-images in an overlapping projection region corresponding to the overlapping region; performing consistency optimization on a pixel gradient field of the overlapping projection region according to a Poisson fusion algorithm, eliminating a joint effect between the sub-images, and completing the projection of each of the sub-images into the panorama canvas coordinate system.

[0150] In some embodiments, the adjusting a corresponding size of each of the sub-images in the panorama canvas coordinate system according to the scale transformation matrix comprises: analyzing an anisotropic scaling factor corresponding to the scale transformation matrix; calculating a sub-image resolution adjustment ratio according to the anisotropic scaling factor; performing multiple interpolation resampling on the sub-image to adjust an original resolution corresponding to the sub-image to a preset resolution; determining a length-width ratio constraint condition corresponding to the sub-image according to a tolerance threshold determined according to a mounting position tolerance of the array camera, so as to keep the length-width ratio of the sub-image in the scale transformation process in conformity with the length-width ratio constraint condition.

[0151] In some embodiments, the single-lens calibration of the wide-angle camera according to the checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera comprises: arranging a plurality of checkerboard calibration boards in different poses in a field of view range of the wide-angle camera; acquiring a calibration board image corresponding to each of the calibration boards collected by the wide-angle camera, and using Zhang's calibration method to solve a camera intrinsic parameter matrix and a distortion coefficient vector; and constructing the distortion correction matrix according to the camera intrinsic parameter matrix and the distortion coefficient vector.

[0152] The application also provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to enable the processor to implement the panorama image generation method provided by any of the embodiments. Figure 1 The steps of the panorama image generation method provided by any of the embodiments.

[0153] The computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like.

[0154] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for generating panoramic images, characterized in that, include: The wide-angle camera is calibrated using the checkerboard calibration method to obtain the distortion correction matrix corresponding to the wide-angle camera; The wide-angle camera and the array camera are controlled to perform spatiotemporal synchronous acquisition to obtain an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; wherein each sub-image in the sub-image set has an overlapping area with the initial panoramic image; The initial panoramic image is subjected to nonlinear distortion correction based on the distortion correction matrix to generate a standardized panoramic image without geometric distortion. Obtain the sub-image feature information of each sub-image in the sub-image set, and obtain the projection position, perspective transformation matrix and scale transformation matrix of each sub-image in the standardized panoramic image based on the sub-image feature information; A panoramic canvas coordinate system is established based on the standardized panoramic image. Each sub-image is projected into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position. The size of each sub-image is adjusted in the panoramic canvas coordinate system according to the scale transformation matrix. A panoramic image is generated based on the adjusted panoramic canvas coordinate system.

2. The method according to claim 1, characterized in that, The step of performing nonlinear distortion correction on the initial panoramic image based on the distortion correction matrix to generate a standardized panoramic image without geometric distortion includes: Obtain the radial distortion coefficient and tangential distortion coefficient corresponding to the distortion correction matrix; Obtain the mapping relationship between the original pixels and the corrected pixels; The initial panoramic image is traversed to perform inverse mapping calculations based on the mapping relationship, so as to generate a standardized panoramic image with barrel distortion eliminated by bilinear interpolation.

3. The method according to claim 1, characterized in that, The step of obtaining the projection position, perspective transformation matrix, and scale transformation matrix of each sub-image in the standardized panoramic image based on the sub-image feature information includes: The local feature point set of each sub-image and the global feature point set of the standardized panoramic image are obtained according to the preset image feature detector; For each subgraph, perform nearest neighbor feature matching to obtain the correspondence between the local feature point set and the global feature point set; In the local feature point set, matching point pairs that satisfy the homography constraint are selected according to the correspondence relationship, and the homography matrix corresponding to the matching point is obtained by solving the least squares method. The homography matrix is ​​decomposed to obtain the perspective transformation matrix containing rotation and translation components and the scale transformation matrix containing scaling factors; The projection center coordinates of the sub-image in the standardized panoramic image are determined based on the decomposition results corresponding to the eigenvalues ​​of the homography matrix.

4. The method according to claim 1, characterized in that, The step of establishing a panoramic canvas coordinate system based on the standardized panoramic image includes: Using the geometric center of the standardized panoramic image as the origin, a composite coordinate system of polar and Cartesian coordinates is established. The coverage area of ​​the panoramic canvas coordinate system is determined based on the field of view parameters of the wide-angle camera; Establish a mapping relationship between pixel coordinates and panoramic coordinates; Set canvas boundary constraints in a Cartesian coordinate system, and establish the panoramic canvas coordinate system based on the composite coordinate system, coverage area, and mapping relationship.

5. The method according to claim 1, characterized in that, The step of projecting each sub-image onto the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position includes: Perform an affine transformation on the pixel set of each sub-image based on the perspective transformation matrix and the translation vector corresponding to the projection position; A dynamic weighted fusion region is established in the panoramic canvas coordinate system; For the overlapping projection regions corresponding to the overlapping regions, calculate the projection confidence weight of each sub-graph within the overlapping projection regions; The pixel gradient field of the overlapping projection area is optimized for consistency using the Poisson fusion algorithm to eliminate the seam effect between the sub-images, thus completing the projection of each sub-image onto the panoramic canvas coordinate system.

6. The method according to claim 1, characterized in that, Adjusting the size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix includes: Analyze the anisotropic scaling factor corresponding to the scaling transformation matrix; Calculate the sub-image resolution adjustment ratio based on the anisotropic scaling factor; The sub-image is interpolated and resampled multiple times to adjust the original resolution of the sub-image to a preset resolution; Based on the tolerance threshold determined by the array camera installation position tolerance, the aspect ratio constraint condition corresponding to the sub-image is determined based on the tolerance threshold, so as to maintain the aspect ratio of the sub-image in accordance with the aspect ratio constraint condition during the scaling process.

7. The method according to claim 1, characterized in that, The step of calibrating a wide-angle camera using a single lens according to the checkerboard calibration method to obtain the distortion correction matrix corresponding to the wide-angle camera includes: Multiple checkerboard calibration plates in different poses are arranged within the field of view of the wide-angle camera; The calibration board image corresponding to each calibration board is acquired by the wide-angle camera, and the camera intrinsic parameter matrix and distortion coefficient vector are solved using Zhang's calibration method. The distortion correction matrix is ​​constructed based on the camera intrinsic parameter matrix and the distortion coefficient vector.

8. A panoramic image generation device, characterized in that, The device includes: The lens calibration unit is used to perform single-lens calibration of the wide-angle camera according to the checkerboard calibration method and obtain the distortion correction matrix corresponding to the wide-angle camera. The synchronous acquisition unit is used to control the wide-angle camera and the array camera to perform spatiotemporal synchronous acquisition, and to acquire the initial panoramic image acquired by the wide-angle camera and the sub-image set acquired by the array camera; wherein each sub-image in the sub-image set has an overlapping area with the initial panoramic image; The distortion correction unit is used to perform nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion. The feature acquisition unit is used to acquire the sub-image feature information of each sub-image in the sub-image set, and to acquire the projection position, perspective transformation matrix and scale transformation matrix of each sub-image in the standardized panoramic image based on the sub-image feature information. The canvas establishment unit is used to establish a panoramic canvas coordinate system based on the standardized panoramic image, project each sub-image into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image based on the adjusted panoramic canvas coordinate system.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 1 to 7.

Images