Panoramic image generation method and device, equipment and storage medium

Abstract

The invention provides a panoramic image generation method and device, equipment and a storage medium, and the method comprises the steps: carrying out the single-lens calibration of a wide-angle camera according to a checkerboard calibration method, and obtaining a distortion correction matrix; controlling the wide-angle camera and the array camera to perform time-space synchronous acquisition to obtain an initial panoramic image and a sub-image set; performing nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a distortionless standardized panoramic image; acquiring sub-image feature information of each sub-image in the sub-image set, and acquiring a projection position, a perspective transformation matrix and a scale transformation matrix corresponding to each sub-image in the standardized panoramic image according to the sub-image feature information; and establishing a panoramic canvas coordinate system according to the standardized panoramic image, projecting each sub-image into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and the projection position, adjusting the size corresponding to the sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generating a panoramic image according to the adjusted panoramic canvas coordinate system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a panoramic image generation method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of computer vision technology, array camera technology has been widely used in many fields due to its ability to generate high-resolution, high signal-to-noise ratio, and high dynamic range images through the cooperative work of multiple cameras. Array cameras can effectively improve image quality and provide important data support for 3D reconstruction and panoramic imaging by capturing images from different angles through multiple cameras. However, a key problem needs to be solved in practical applications of array cameras: how to accurately calibrate and stitch the images captured by multiple cameras to generate high-quality panoramic images.

[0003] In the prior art, the calibration of array cameras usually requires individual calibration of each camera, which involves a complex chessboard calibration process. The calibration of each camera requires obtaining its intrinsic and extrinsic parameters, including focal length, principal point, distortion parameters, and the relative position and attitude relationship between cameras, through a chessboard image. However, this individual calibration method has the following significant problems: first, the calibration workload is huge, especially when the array camera contains multiple cameras, the calibration process is time-consuming and tedious; second, due to the small differences in the calibration parameters of each camera, the matching error between subgraphs is large, and the generated panoramic image has obvious stitching seams, geometric distortion, or color inconsistency, which seriously affects the quality and visual effect of the final image.

[0004] Therefore, there is an urgent need for a panoramic image generation method to solve at least one of the above problems. SUMMARY

[0005] The present application provides a panoramic image generation method, device, equipment and storage medium, aiming to solve the problem that in the prior art, the calibration of array cameras usually requires individual calibration of each camera, which involves a complex chessboard calibration process and has a huge calibration workload, and the matching error between subgraphs is large, resulting in poor quality and visual effect of the final image.

[0006] In a first aspect, the present application provides a panoramic image generation method, comprising: single-lens calibration of the wide-angle camera according to the chessboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera; controlling the wide-angle camera and the array camera to perform time and space synchronization acquisition, to obtain an initial panoramic image captured by the wide-angle camera and a set of subgraphs captured by the array camera; wherein each subgraph in the set of subgraphs has an overlapping area with the initial panoramic image; nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion; obtain subgraph feature information of each subgraph in the subgraph set, and obtain a projection position, a perspective transformation matrix, and a scale transformation matrix of each subgraph in the standardized panoramic image according to the subgraph feature information; establish a panoramic canvas coordinate system according to the standardized panoramic image, project each subgraph into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the corresponding size of each subgraph in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image according to the adjusted panoramic canvas coordinate system.

[0007] In some embodiments, the nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion comprises: obtaining corresponding radial distortion coefficients and tangential distortion coefficients in the distortion correction matrix; obtaining a mapping relationship between original pixel points and corrected pixel points; traversing all pixel points of the initial panoramic image to perform inverse mapping calculation according to the mapping relationship, so as to generate a standardized panoramic image eliminating barrel distortion according to bilinear interpolation.

[0008] In some embodiments, the obtaining of the projection position, the perspective transformation matrix, and the scale transformation matrix of each subgraph in the standardized panoramic image according to the subgraph feature information comprises: obtaining a local feature point set of each subgraph and a global feature point set of the standardized panoramic image according to a preset image feature detector; performing neighbor feature matching on each subgraph to obtain a corresponding relationship between the local feature point set and the global feature point set; filtering a matching point pair satisfying a homography constraint from the local feature point set according to the corresponding relationship, and obtaining a homography matrix corresponding to the matching point pair according to a least square method; decomposing the homography matrix to obtain the perspective transformation matrix containing a rotation and translation component and the scale transformation matrix containing a scaling factor; and determining a projection center coordinate of the subgraph in the standardized panoramic image according to a decomposition result corresponding to an eigenvalue of the homography matrix.

[0009] In some embodiments, the establishment of the panoramic canvas coordinate system according to the standardized panoramic image comprises: taking a geometric center of the standardized panoramic image as a coordinate origin to establish a composite coordinate system of a polar coordinate system and a Cartesian coordinate system; determining a coverage range of the panoramic canvas coordinate system according to a field of view angle parameter of the wide-angle camera; establishing a mapping relationship from a pixel coordinate to a panoramic coordinate; setting a canvas boundary constraint condition in the Cartesian coordinate system, and establishing the panoramic canvas coordinate system according to the composite coordinate system, the coverage range, and the mapping relationship.

[0010] In some embodiments, the projecting each of the subgraphs into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and the projection position comprises: performing affine transformation on the pixel point set of each subgraph according to a translation vector corresponding to the perspective transformation matrix and the projection position; establishing a dynamic weight fusion region in the panoramic canvas coordinate system; calculating a projection confidence weight of each subgraph in an overlapping projection region corresponding to the overlapping region; performing consistency optimization on a pixel gradient field of the overlapping projection region according to a Poisson fusion algorithm, eliminating seam effects between the subgraphs, and completing the projecting of each of the subgraphs into the panoramic canvas coordinate system.

[0011] In some embodiments, the adjusting the size of each of the subgraphs in the panoramic canvas coordinate system according to the scale transformation matrix comprises: analyzing anisotropic scaling factors corresponding to the scale transformation matrix; calculating a subgraph resolution adjustment ratio according to the anisotropic scaling factors; performing multiple interpolation resampling on the subgraph to adjust the original resolution of the subgraph to a preset resolution; determining a length-width ratio constraint condition of the subgraph according to a tolerance threshold determined according to the array camera installation position tolerance, so as to keep the length-width ratio of the subgraph in the scale transformation process consistent with the length-width ratio constraint condition.

[0012] In some embodiments, the single-lens calibration of the wide-angle camera according to the checkerboard calibration method to obtain the distortion correction matrix corresponding to the wide-angle camera comprises: arranging a plurality of checkerboard calibration boards in different poses in the field of view range of the wide-angle camera; acquiring a calibration board image corresponding to each of the calibration boards collected by the wide-angle camera, and using Zhang's calibration method to solve a camera intrinsic parameter matrix and a distortion coefficient vector; and constructing the distortion correction matrix according to the camera intrinsic parameter matrix and the distortion coefficient vector.

[0013] In a second aspect, the present application provides a panoramic image generation device, the device comprising: a lens calibration unit configured to calibrate a wide-angle camera according to a checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera; a synchronous acquisition unit configured to control the wide-angle camera and an array camera to perform time-space synchronous acquisition, and to acquire an initial panoramic image collected by the wide-angle camera and a set of subgraphs collected by the array camera; wherein each of the subgraphs in the set of subgraphs has an overlapping region with the initial panoramic image; a distortion correction unit configured to perform non-linear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion; The feature acquisition unit is configured to acquire subgraph feature information of each subgraph in the subgraph set, and acquire a projection position, a perspective transformation matrix and a scale transformation matrix corresponding to each subgraph in the standardized panoramic image according to the subgraph feature information. The canvas establishment unit is configured to establish a panoramic canvas coordinate system according to the standardized panoramic image, project each subgraph into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the size of each subgraph in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image according to the adjusted panoramic canvas coordinate system.

[0014] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program and implement the panoramic image generation method provided in any embodiment of the present application when executing the computer program.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor implement the panoramic image generation method provided in any embodiment of the present application.

[0016] The present application provides a panoramic image generation method, device, equipment and storage medium. The panoramic image generation method provided by the present application significantly improves the efficiency and quality of panoramic image generation by optimizing the calibration process and image registration strategy. The method only calibrates the checkerboard of the wide-angle camera to obtain its distortion correction matrix, and saves the separate calibration of each sub-camera of the array camera. Through nonlinear distortion correction, the initial panoramic image collected by the wide-angle camera is converted into a standardized image without geometric distortion, which serves as a global reference benchmark. The traditional method needs to calibrate each camera in the array one by one, while the present method only needs to calibrate a single lens, greatly simplifying the process.

[0017] At the same time, the method controls the wide-angle camera and the array camera to collect synchronously, ensuring the consistency of the data of the two in time (shooting at the same time) and space (fixed relative position). The wide-angle camera provides a global initial panoramic image, and the array camera provides a high-resolution local subgraph set. The overlapping area of the subgraph and the initial panoramic image provides a basis for subsequent registration. The synchronous mechanism avoids misplacement of dynamic scenes, and directly provides a global coordinate framework through the wide-angle image.

[0018] The method further extracts feature information (such as SIFT, ORB, etc. feature points) of the subgraph, matches with the standardized panoramic graph, determines the projection position, perspective transformation matrix (homography matrix) and scale transformation matrix of the subgraph. By projecting to a unified panoramic canvas coordinate system, dynamically adjusting the size and viewing angle of the subgraph, seamless splicing is realized. Using the global information of the wide-angle image to guide the subgraph registration, the error accumulation problem in the traditional method of relying on pairwise matching between subgraphs is avoided.

[0019] Finally, the method establishes a coordinate system with a standardized panoramic graph, projects the subgraph to the canvas through perspective transformation (solves the viewing angle difference) and scale transformation (unifies the resolution), combines with a fusion algorithm (such as multi-band mixing) to eliminate the joint, and generates a high-resolution panoramic image. Dynamically adjust the size and viewing angle of the subgraph to ensure the balance between local details and global consistency.

[0020] The provided method has at least the following beneficial effects: 1. Significantly reduce the calibration complexity: the traditional method needs to calibrate each sub-camera in the array camera separately, and the present method only needs to calibrate a single wide-angle lens, saving more than 90% of the calibration time and computing resources, especially suitable for large multi-camera array systems.

[0021] 2. Improve registration accuracy and image quality: use the standardized panoramic graph of the wide-angle camera as a global reference, and the subgraph registration is based on the same coordinate system, avoiding the cumulative error of traditional pairwise matching between subgraphs, significantly reducing the misalignment and ghosting problems.

[0022] 3. Enhance system robustness: time-space synchronous acquisition ensures the consistency of subgraphs and panoramic graphs in dynamic scenes, avoiding motion blur or object displacement caused by shooting time difference, and improving the success rate of complex scene (such as pedestrians, vehicles) splicing.

[0023] 4. Optimize resource utilization: the wide-angle camera provides a low-resolution global view, and the array camera provides a high-resolution local subgraph. Through intelligent fusion, the resolution and coverage of the panoramic graph are considered, reducing the dependence on the hardware performance of a single camera.

[0024] 5. Suitable for large-scale application scenarios: can be extended to fields such as unmanned aerial vehicle aerial photography, vehicle surround view, VR photography, etc. Especially in scenarios that require rapid deployment (such as temporary monitoring systems), the tedious multi-camera calibration steps are saved, and the practicality is improved.

[0025] In summary, the provided method solves the problems of complex calibration and large matching error in traditional panoramic generation through single-lens calibration, time-space synchronous acquisition and global-local collaborative registration strategy, realizes efficient and high-precision panoramic image generation, and has wide application prospect and commercial value.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 is a schematic flow chart of steps of a panoramic image generation method provided by an embodiment of the present application; Figure 2 is a schematic structural block diagram of a panoramic image generation apparatus provided by an embodiment of the present application; Figure 3 is a schematic structural block diagram of a computer device provided by an embodiment of the present application.

[0029] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0031] The flow chart shown in the drawings is only an example description, and does not necessarily include all the contents and operations / steps, and does not necessarily be executed in the described order. For example, some operations / steps can be decomposed, combined or partially combined, so that the actual execution order can be changed according to the actual situation.

[0032] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second" and the like. Those skilled in the art can understand that "first", "second" and the like do not limit the quantity and execution order, and "first", "second" and the like do not necessarily mean different.

[0033] It is to be understood that the terms used in the specification herein are for the purpose of describing particular embodiments and do not intend to limit the application. As used in the specification and the appended claims herein, the singular forms "a," "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise.

[0034] It should also be understood that the term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, and includes these combinations.

[0035] Some embodiments of the present application will be described in detail with reference to the drawings. The following embodiments and features of the embodiments described below can be combined with each other without conflict.

[0036] With the rapid development of computer vision technology, array camera technology has been widely used in many fields due to its ability to generate high-resolution, high signal-to-noise ratio, and high dynamic range images through the cooperative work of multiple cameras. Array cameras can effectively improve image quality and provide important data support for 3D reconstruction and panoramic imaging by capturing images from different angles through multiple cameras. However, array cameras need to solve a key problem in practical application: how to accurately calibrate and stitch the images captured by multiple cameras to generate high-quality panoramic images.

[0037] In the prior art, the calibration of array cameras usually needs to calibrate each camera separately, which involves a complex chessboard calibration process. The calibration of each camera needs to obtain its intrinsic and extrinsic parameters, including focal length, principal point, distortion parameters, and the relative position and attitude relationship between cameras, through a chessboard image. However, this method of calibration one by one has the following significant problems: First, the calibration workload is huge, especially when the array camera contains multiple cameras, the calibration process is time-consuming and tedious; second, due to the small differences in the calibration parameters of each camera, the matching error between sub-images is large, and the panoramic image generated by stitching will have obvious stitching seams, geometric distortion, or color inconsistency, which seriously affects the quality and visual effect of the final image.

[0038] In addition, in the prior art, even if the sub-images are corrected and stitched through the calibration parameters, the stitching process still relies on complex image registration algorithms, and it is difficult to ensure seamless connection between sub-images in practical applications. Especially in dynamic scenes or complex lighting conditions, the geometric transformation relationship and lighting consistency between sub-images are difficult to accurately match, resulting in distortion, ghosting or blurring in the stitched panoramic image.

[0039] To solve the above problems, the present application provides an innovative panoramic image generation method. By only calibrating the wide-angle camera, using the synchronous acquisition data of the wide-angle camera and the array camera, combining feature matching and perspective transformation matrix, the fast calibration and stitching of all subgraphs are realized. This method not only significantly reduces the calibration workload and avoids the cumbersome process of traditional individual calibration method, but also improves the accuracy and efficiency of subgraph stitching through feature matching and transformation matrix optimization, and finally generates high-resolution, distortion-free panoramic images. This method has significant innovation in technology, which can effectively solve the problems of calibration complexity and stitching error in the prior art, and provides a more efficient and reliable solution for the application of array cameras.

[0040] Please refer to Figure 1 The present application provides a panoramic image generation method. The method is run on a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a notebook computer, a wearable device, or a robot, etc.

[0041] As Figure 1 shown, the provided panoramic image generation method includes steps S101 to S105. Details are as follows: Step S101. Calibrate the wide-angle camera according to the checkerboard calibration method to obtain the distortion correction matrix corresponding to the wide-angle camera.

[0042] Specifically, the checkerboard calibration method is a geometric feature-based calibration method. By analyzing the corner point position in the checkerboard image, the camera's intrinsic parameters (such as focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (such as camera position and attitude) are calculated. In the calibration process, the known size of the checkerboard and the accurate extraction of the corner points are the key.

[0043] Unlike traditional methods, this step only calibrates the wide-angle camera, avoiding the cumbersome process of calibrating multiple cameras one by one. The wide-angle camera serves as a reference, and its calibration parameters will be used for subsequent subgraph correction and stitching. The distortion coefficients (such as radial distortion and tangential distortion) obtained by calibration are used to generate a distortion correction matrix to eliminate geometric distortion in the image.

[0044] As with high-precision printed checkerboard calibration boards, ensure the accuracy of the checkerboard size and the clarity of the pattern. Align the wide-angle camera to the checkerboard and take multiple images from different angles and distances, ensuring that the checkerboard is evenly distributed in the image. Use image processing algorithms (such as the findChessboardCorners function in OpenCV) to extract the checkerboard corner points and calculate sub-pixel accuracy. Use the calibration algorithm (such as the calibrateCamera function in OpenCV) to calculate the camera's intrinsic parameters and distortion coefficients. Generate a distortion correction matrix based on the distortion coefficients for subsequent image correction.

[0045] The steps only need to calibrate one wide-angle camera, significantly reducing the complexity and time cost of calibration. Through multi-angle shooting and accurate calculation, the accuracy of the calibration parameters is ensured, providing a reliable foundation for subsequent image processing. Single-lens calibration avoids the tedious process of calibrating multiple cameras one by one, especially suitable for large-scale array camera systems.

[0046] Step S102. Control the wide-angle camera and the array camera to perform time-space synchronous acquisition, obtain an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; each sub-image in the sub-image set has an overlapping area with the initial panoramic image.

[0047] Specifically, through hardware or software triggering mechanisms, ensure that the wide-angle camera and the array camera capture images at the same time and in the same space to ensure the temporal consistency and spatial consistency between images. The wide-angle camera captures a wide-angle image as the basis for the panoramic image, covering the entire scene. The array camera captures multiple sub-images, each covering a part of the initial panoramic image and ensuring sufficient overlapping areas for subsequent stitching.

[0048] By using hardware triggers or software instructions, ensure that the wide-angle camera and the array camera capture images simultaneously. The wide-angle camera captures an initial panoramic image with a wide angle, covering the entire scene. The array camera captures multiple sub-images, each covering a part of the initial panoramic image and ensuring sufficient overlapping areas between adjacent sub-images. Store the captured images and perform preliminary noise removal and format conversion.

[0049] Time-space synchronous acquisition ensures that all images are captured in the same environment, reducing errors caused by time or space differences. The initial panoramic image and the sub-image set provide necessary data support for subsequent stitching and correction. The synchronous acquisition mechanism improves the efficiency of data acquisition, especially suitable for dynamic scenes.

[0050] Step S103. Perform non-linear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion.

[0051] Specifically, the distortion correction matrix obtained in step S101 is used to perform geometric distortion correction on the initial panoramic image to eliminate radial distortion and tangential distortion in the image. The corrected panoramic image has the characteristics of no distortion and geometric consistency, facilitating subsequent stitching and processing.

[0052] If the initial panoramic image is geometrically transformed using the distortion correction matrix, the distortion is eliminated. The bilinear interpolation or bicubic interpolation algorithm is applied to smooth the corrected image to ensure image quality. The corrected image is saved as a standardized panoramic image as a reference for subsequent processing. The corrected image has no geometric distortion, improving the geometric accuracy of the image. The standardized panoramic image provides a unified basis for subsequent stitching and projection, reducing stitching errors. Through interpolation processing, the corrected image is smooth and distortion-free.

[0053] Step S104. Obtain the sub-image feature information of each sub-image in the sub-image set, and obtain the corresponding projection position, perspective transformation matrix and scale transformation matrix of each sub-image in the standardized panoramic image according to the sub-image feature information.

[0054] Specifically, the feature points (such as SIFT and SURF feature points) and descriptors of each sub-image are extracted for matching and positioning. The projection position is the corresponding position of the sub-image in the standardized panoramic image. The perspective transformation matrix is a transformation matrix for projecting the sub-image into the standardized panoramic image, ensuring geometric consistency. The scale transformation matrix is used to adjust the size of the sub-image to match the size of the standardized panoramic image.

[0055] Feature extraction is performed on each sub-image to obtain feature points and descriptors. The feature points of the sub-image are matched with the feature points of the standardized panoramic image to find the corresponding relationship. The perspective transformation matrix and the scale transformation matrix are calculated according to the matching results. The projection position of the sub-image in the standardized panoramic image is determined.

[0056] Through feature matching and transformation matrix calculation, the sub-image is accurately aligned with the standardized panoramic image. The application of perspective transformation and scale transformation matrix reduces the geometric distortion and size inconsistency in the stitching process. Feature extraction and matching algorithm realizes automatic processing, improving efficiency.

[0057] Step S105. Establish a panoramic canvas coordinate system according to the standardized panoramic image, project each sub-image into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the corresponding size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image according to the adjusted panoramic canvas coordinate system.

[0058] Specifically, the panorama canvas coordinate system is a coordinate system established based on the standardized panorama image, which is used to unify the projection and stitching of all sub-images. Each sub-image is projected into the panorama canvas coordinate system according to the perspective transformation matrix and the projection position, and the size is adjusted according to the scale transformation matrix to ensure that the stitched image is seamlessly connected. After stitching all sub-images, a final high-resolution, distortion-free panorama image is generated.

[0059] As the basis of the standardized panorama image, the panorama canvas coordinate system is established. Each sub-image is projected into the panorama canvas coordinate system according to the perspective transformation matrix and the projection position. The size of the sub-image is adjusted according to the scale transformation matrix to ensure consistency with the panorama canvas coordinate system. The stitched image is smoothed using a multi-band fusion or Poisson fusion algorithm to eliminate stitching seams and inconsistent lighting. The fused image is saved as the final panorama image.

[0060] Through projection and adjustment, the seamless connection of sub-images in the panorama canvas coordinate system is ensured, and the stitching seams and geometric distortion are reduced. The final generated panorama image has the characteristics of high resolution, distortion-free, and color consistency, significantly improving the image quality and visual effect. The automated stitching and fusion algorithm improves the processing efficiency and is suitable for large-scale image processing.

[0061] In summary, the panorama image generation method of the present application realizes the rapid calibration and stitching of array camera sub-images by only calibrating the wide-angle camera, combined with feature matching and perspective transformation matrix. This method significantly reduces the calibration workload, improves the stitching accuracy and efficiency, and finally generates a high-resolution, distortion-free panorama image. This method has significant innovation in technology, which can effectively solve the problems of calibration complexity and stitching error in the prior art, and provides a more efficient and reliable solution for the application of array cameras. Through further refinement and improvement, the implementation process of this method is more clear, and the technical details are more rich, providing stronger guidance and operability for practical application.

[0062] In some embodiments, the non-linear distortion correction of the initial panorama image according to the distortion correction matrix to generate a standardized panorama image without geometric distortion includes: obtaining corresponding radial distortion coefficients and tangential distortion coefficients in the distortion correction matrix; obtaining the mapping relationship between the original pixel points and the corrected pixel points; traversing all pixel points of the initial panorama image to perform inverse mapping calculation according to the mapping relationship, to generate a standardized panorama image eliminating barrel distortion according to bilinear interpolation.

[0063] In step S101, the distortion correction matrix of the wide-angle camera is obtained by the chessboard calibration method, which contains the radial distortion coefficients (such as k1, k2, k3) and the tangential distortion coefficients (such as p1, p2). These coefficients describe the distortion characteristics of the pixel points in the image, and the radial distortion mainly causes the barrel or pillow distortion of the image edge, while the tangential distortion is caused by the non-parallelism of the lens and the sensor.

[0064] According to the preset distortion model, the corrected pixel points (x', y') corresponding to each original pixel point (x, y) are calculated. The distortion model formula is as follows: x'=x(1+k1r^2+k2r^4+k3r^6)+2p1xy+p2(r^2+2x^2); y'=y(1+k1r^2+k2r^4+k3r^6)+p1(r^2+2y^2)+2p2xy; where r^2=x^2+y^2. x, y are pixel coordinates in the original image. k1, k2, k3 are radial distortion coefficients, which are used to describe the barrel or pillow distortion of the image edge. r is the distance of the pixel point to the center of the image. Map from the corrected image space (x', y') to the original image space (x, y) in reverse to avoid holes in the corrected image. For each corrected pixel point, calculate its corresponding position in the original image according to the distortion model.

[0065] Since the reverse mapping may result in non-integer coordinates, the bilinear interpolation algorithm is used to calculate the gray value of the corrected pixel point. The bilinear interpolation formula is as follows: f(x, y)=(1-α)(1-β)f(x1, y1)+α(1-β)f(x2, y1)+(1-α)βf(x1, y2)+αβf(x2, y2); where x1, y1 are the upper left pixel coordinates, x2, y2 are the lower right pixel coordinates, and α and β are the interpolation weights. f(x, y) is the gray value of the interpolated pixel point.

[0066] By correcting the radial distortion and tangential distortion, a standardized panoramic image without geometric distortion is generated, which significantly improves the geometric accuracy of the image. The bilinear interpolation algorithm ensures that the corrected image is smooth and distortion-free, preserving the details of the original image. The standardized panoramic image provides a unified basis for subsequent sub-image stitching and projection, reducing the stitching error. The reverse mapping and interpolation algorithm realizes automatic correction and improves the processing efficiency.

[0067] In some embodiments, the obtaining, according to the subgraph feature information, a projection position, a perspective transformation matrix and a scale transformation matrix of each subgraph in the standardized panoramic image comprises: obtaining a local feature point set of each subgraph and a global feature point set of the standardized panoramic image according to a preset image feature detector; performing a nearest neighbor feature matching on each subgraph to obtain a corresponding relationship between the local feature point set and the global feature point set; screening, according to the corresponding relationship, a matching point pair of the local feature point set that satisfies a homography constraint, and obtaining a homography matrix corresponding to the matching point pair according to a least square method; decomposing the homography matrix to obtain the perspective transformation matrix containing a rotation and translation component and the scale transformation matrix containing a scaling factor; and determining a projection center coordinate of the subgraph in the standardized panoramic image according to a decomposition result corresponding to an eigenvalue of the homography matrix.

[0068] Feature detection algorithms (such as SIFT, SURF or ORB) are used to extract feature points and their descriptors from the subgraph and the standardized panoramic image. The local feature point set is the set of feature points extracted from the subgraph image. The global feature point set is the set of feature points extracted from the standardized panoramic image. Feature matching algorithms (such as FLANN or brute-force matching) are used to calculate the matching relationship between the subgraph feature points and the standardized panoramic image feature points. In the matching process, the distance between the feature descriptors is calculated, and the most matching feature point pair is screened out. RANSAC algorithm is used to screen out the matching point pair that satisfies the homography constraint, and to eliminate the false matching points. According to the screened matching point pair, the homography matrix H is solved by the least square method, and the homography matrix describes the projection relationship between the subgraph and the standardized panoramic image.

[0069] The homography matrix H is singular value decomposed (SVD) to obtain a rotation matrix R, a translation vector T and a scaling factor s. The perspective transformation matrix M is composed of the rotation matrix and the translation vector, which is used to describe the projection transformation of the subgraph in the standardized panoramic image. The scale transformation matrix S is composed of the scaling factor, which is used to adjust the size of the subgraph. According to the eigenvalue decomposition result of the homography matrix, the projection center coordinates (xc, yc) of the subgraph in the standardized panoramic image are calculated, which are used to determine the specific position of the subgraph in the panoramic image.

[0070] Through the feature detection and matching algorithm, the accurate alignment of the subgraph and the standardized panoramic image is ensured, and the stitching error is reduced. The perspective transformation matrix and the scale transformation matrix obtained by the decomposition of the homography matrix ensure the geometric consistency and size matching of the subgraph in the projection process. The feature extraction, matching and matrix decomposition algorithms realize the automation processing and improve the efficiency. Through the accurate projection and adjustment, the seamless connection of the subgraph in the panoramic image is ensured, and the stitching seam and geometric distortion are reduced. The RANSAC algorithm is used to effectively eliminate the false matching points, which improves the robustness and reliability of the algorithm.

[0071] In some embodiments, the establishing a panorama canvas coordinate system according to the standardized panorama image comprises: taking the geometric center of the standardized panorama image as a coordinate origin, establishing a composite coordinate system of a polar coordinate system and a Cartesian coordinate system; determining a coverage range of the panorama canvas coordinate system according to a field of view angle parameter of the wide-angle camera; establishing a mapping relationship of pixel coordinates to panorama coordinates; setting a canvas boundary constraint condition in the Cartesian coordinate system, and establishing the panorama canvas coordinate system according to the composite coordinate system, the coverage range and the mapping relationship.

[0072] The geometric center point (xc, yc) of the standardized panorama image is determined and taken as the coordinate origin. A composite coordinate system is established, in which the polar coordinate system is used to describe the angle and distance of the pixel point, and the Cartesian coordinate system is used to describe the horizontal and vertical positions of the pixel point.

[0073] The conversion relationship of the composite coordinate system is as follows: θ = arctan[(y - yc) / (x - xc)]; x, y are the pixel coordinates in the standardized panorama image. R is the distance of the pixel point to the geometric center.

[0074] The field of view angle (FOV) parameter of the wide-angle camera is obtained to determine the coverage range of the panorama canvas coordinate system. For example, if the horizontal field of view angle of the wide-angle camera is 180 degrees, the horizontal coverage range of the panorama canvas coordinate system is 90-90 degrees.

[0075] The pixel coordinates (x, y) of the standardized panorama image are mapped to the panorama coordinates (r, θ). The mapping relationship is as follows: θ = arctan[(y - yc) / (x - xc)]; Rmax is the maximum radius of the standardized panorama image, used for normalizing the distance.

[0076] The canvas boundary constraint condition is set in the Cartesian coordinate system to ensure that the coverage range of the panorama canvas coordinate system is consistent with the field of view angle of the wide-angle camera. According to the composite coordinate system, the coverage range and the mapping relationship, the panorama canvas coordinate system is established, which is used for unifying the projection and stitching of all sub-images.

[0077] By establishing the composite coordinate system, the coordinate systems of the standardized panorama image and the sub-images are unified, which facilitates the subsequent projection and stitching. The coverage range is determined according to the field of view angle parameter of the wide-angle camera to ensure that the panorama canvas coordinate system can completely cover the entire scene. The mapping relationship of the pixel coordinates to the panorama coordinates simplifies the coordinate conversion process and improves the processing efficiency. The canvas boundary constraint condition ensures the rationality and integrity of the panorama canvas coordinate system, avoiding image overflow or loss.

[0078] In some embodiments, the projecting each of the subgraphs into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position comprises: performing affine transformation on a pixel point set of each subgraph according to a translation vector corresponding to the perspective transformation matrix and projection position; establishing a dynamic weight fusion region in the panoramic canvas coordinate system; calculating a projection confidence weight of each subgraph in an overlapping projection region corresponding to the overlapping region; and performing consistency optimization on a pixel gradient field of the overlapping projection region according to a Poisson fusion algorithm, eliminating seam effects between the subgraphs, and completing the projecting each of the subgraphs into the panoramic canvas coordinate system.

[0079] The perspective transformation matrix M corresponding to the subgraph and the translation vector T corresponding to the projection position are obtained. Affine transformation is performed on each pixel point (x, y) of the subgraph to project it into the panoramic canvas coordinate system. The overlapping region of the subgraph and the panoramic canvas coordinate system is determined, and a dynamic weight fusion region is established. The dynamic weight fusion region is used to process the overlapping part between the subgraphs to ensure seamless splicing. The projection confidence weight of the subgraph in the overlapping projection region is calculated according to the change of the pixel value. The confidence weight formula is: wi=1 / σi^2; σi is the pixel variance of the subgraph in the overlapping region (the projection confidence weight of the i-th subgraph is wi, and the pixel variance is σi). The pixel gradient field of the overlapping projection region is optimized for consistency using the Poisson fusion algorithm to ensure seamless connection between the subgraphs. The Poisson fusion algorithm adjusts the pixel value of the overlapping region by solving the Poisson equation to make it smoothly transition with the surrounding area. Finally, the optimized subgraph is projected into the panoramic canvas coordinate system to generate a seamless panoramic image.

[0080] Through affine transformation, the accurate projection of the subgraph in the panoramic canvas coordinate system is ensured, and geometric distortion is reduced. The dynamic weight fusion and Poisson fusion algorithm eliminate the seam effects between the subgraphs, realizing seamless splicing. The Poisson fusion algorithm ensures the consistency of the pixel value of the overlapping region, improving the quality and visual effect of the panoramic image. The dynamic weight calculation and Poisson fusion algorithm realize automatic processing, improving efficiency and reliability. The confidence weight and Poisson fusion algorithm enhance the robustness of the algorithm, which is suitable for image stitching under complex scenes and lighting conditions.

[0081] In some embodiments, the adjusting the size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix comprises: analyzing anisotropic scaling factors corresponding to the scale transformation matrix; calculating a sub-image resolution adjustment ratio according to the anisotropic scaling factors; performing multiple interpolation resamplings on the sub-image to adjust the original resolution of the sub-image to a preset resolution; determining a tolerance threshold according to the installation position tolerance of the array camera, and determining a aspect ratio constraint condition of the sub-image according to the tolerance threshold to keep the aspect ratio of the sub-image in the scale transformation process consistent with the aspect ratio constraint condition.

[0082] Specifically, the scale transformation matrix S is decomposed to obtain its scaling factors sx and sy in the horizontal and vertical directions. The anisotropic scaling factors represent the resolution adjustment ratios of the sub-image in different directions. The resolution adjustment ratio of the sub-image is calculated as follows: x-direction adjustment ratio = sx / sref; y-direction adjustment ratio = sy / sref; sref is the reference resolution. The sub-image is resampled using an interpolation algorithm (such as bilinear interpolation or bicubic interpolation) to adjust its resolution to the preset resolution. Multiple interpolation resamplings ensure that the adjusted image is smooth and distortion-free. According to the installation position tolerance of the array camera, a tolerance threshold for aspect ratio adjustment is determined. The aspect ratio of the sub-image is ensured to meet the constraint condition during adjustment: |sy / sx - original aspect ratio| ≤ tolerance threshold. When adjusting the resolution, the aspect ratio of the sub-image is ensured to change by no more than the tolerance threshold, avoiding image distortion. The original aspect ratio is the original aspect ratio of the sub-image before the resolution is adjusted. It is calculated from the original width / original height. It is used to ensure that the difference between the adjusted aspect ratio and the original aspect ratio does not exceed the tolerance threshold during the scale transformation process.

[0083] Through the anisotropic scaling factors and interpolation resampling, the resolution adjustment of the sub-image is ensured to be accurate and smooth. The aspect ratio constraint condition avoids image distortion during the scale transformation process, ensuring the geometric consistency of the image. Multiple interpolation resamplings and aspect ratio constraint conditions improve the quality and visual effect of the adjusted image. Analyzing the scaling factors and interpolation resampling algorithm realizes automated processing, improving efficiency. The tolerance threshold and aspect ratio constraint condition enhance the robustness of the algorithm, which is suitable for complex scenes and installation errors.

[0084] In some embodiments, the single-lens calibration of the wide-angle camera according to the checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera comprises: arranging a plurality of checkerboard calibration boards with different poses within the field of view range of the wide-angle camera; acquiring a calibration board image corresponding to each of the calibration boards collected by the wide-angle camera; using Zhang's calibration method to solve a camera intrinsic parameter matrix and a distortion coefficient vector; and constructing the distortion correction matrix according to the camera intrinsic parameter matrix and the distortion coefficient vector.

[0085] Arrange multiple checkerboard calibration plates at different angles and positions within the field of view of the wide-angle camera to ensure that the calibration plates cover the entire field of view. The pose of each calibration plate should be as diverse as possible to improve calibration accuracy. Use the wide-angle camera to capture an image of each calibration plate, ensuring that the calibration plate is clearly visible in the image. Take multiple images covering different angles and distances. Use Zhang's method to extract the corner points of the calibration plate image and calculate the camera's intrinsic parameter matrix K and distortion coefficient vector D. The intrinsic parameter matrix K includes parameters such as focal length and principal point coordinates, and the distortion coefficient vector D includes radial distortion and tangential distortion coefficients. Based on the intrinsic parameter matrix and distortion coefficient vector, construct the distortion correction matrix C. The distortion correction matrix is ​​used for subsequent image distortion correction to eliminate geometric distortion in the image.

[0086] Multi-angle and diverse calibration plate layouts improve calibration accuracy and ensure the accuracy of calibration parameters. Single-lens calibration avoids the tedious process of calibrating multiple cameras individually, making it particularly suitable for large-scale camera array systems. Zhang's calibration method and the construction of a distortion correction matrix enable automated processing and improve efficiency. The application of the distortion correction matrix eliminates geometric distortion in the image, improving image quality and geometric accuracy. Multi-angle and diverse calibration plate layouts enhance the robustness of the algorithm, making it suitable for complex scenarios and installation errors.

[0087] In some embodiments, a pre-trained deep learning model is used to extract features from the sub-image and the standardized panoramic image. A pre-trained convolutional neural network (e.g., VGG or ResNet) is used to extract high-dimensional features from the sub-image and the standardized panoramic image. The image is input into the network, and the feature map of the last convolutional layer is extracted as a feature descriptor. Feature similarity between the sub-image and the standardized panoramic image is calculated using a feature matching network (e.g., a Siamese network or MatchNet). The cosine similarity or Euclidean distance between the feature vectors is calculated to find the best matching feature point pairs. A graph optimization algorithm is used to optimize the matching results and remove incorrect matching points. The matching results are modeled as a graph structure, and graph optimization algorithms (e.g., maximum flow minimum cut or spectral clustering) are used to remove incorrect matching points. By optimizing the matching graph structure, high-quality matching point pairs are retained. The perspective and scaling transformation matrices are recalculated based on the optimized matching point pairs using the least squares method. This ensures the accuracy of projection and resizing.

[0088] The high-dimensional features extracted by the deep learning model have stronger discrimination ability, improving the accuracy of feature matching. The graph optimization algorithm effectively eliminates false matching points, improving the reliability of the matching result. The deep learning model and the graph optimization algorithm realize automatic processing, improving the efficiency. The generalization ability of the deep learning model and the robustness of the graph optimization algorithm are suitable for image stitching in complex scenes and lighting conditions.

[0089] In some embodiments, the pixel motion vector between the subgraph and the normalized panoramic image is calculated using the optical flow method: the pixel motion vector between the subgraph and the normalized panoramic image is calculated using the optical flow algorithm (such as Lucas-Kanade or Horn-Schunck). By analyzing the motion of the pixels, the correspondence between the subgraph and the panoramic image is found. The projection position and the perspective transformation matrix of the subgraph are dynamically adjusted according to the pixel motion vector: the projection position and the perspective transformation matrix of the subgraph are dynamically adjusted according to the pixel motion vector calculated by the optical flow method. Ensure the accurate alignment of the subgraph and the panoramic image in the dynamic scene. Use a dynamic weight fusion algorithm to process the overlapping area in the dynamic scene: in the dynamic scene, use a dynamic weight fusion algorithm to process the overlapping area between the subgraphs. According to the pixel motion vector and the time consistency, the fusion weight is dynamically adjusted. Generate a seamless stitched panoramic image in a dynamic scene: through dynamic adjustment and fusion, generate a seamless stitched panoramic image in a dynamic scene. Ensure the real-time and consistency of image stitching in dynamic scenes.

[0090] The optical flow method can capture the pixel motion in the dynamic scene, ensuring the real-time and consistency of image stitching. The projection position and the perspective transformation matrix are dynamically adjusted, improving the accuracy of image stitching in dynamic scenes. The dynamic weight fusion algorithm ensures the seamless stitching of images in dynamic scenes. The optical flow method and the dynamic weight fusion algorithm enhance the robustness of the algorithm, which is suitable for complex dynamic scenes.

[0091] In some embodiments, the subgraph and the normalized panoramic image are decomposed into multiple frequency bands: the subgraph and the normalized panoramic image are decomposed into multiple frequency bands using a multi-band decomposition algorithm (such as Laplacian pyramid or wavelet transform). Each frequency band represents a different frequency component of the image. Feature matching and projection are performed separately on different frequency bands: feature matching and projection are performed separately on each frequency band to ensure accurate alignment of different frequency components. Through multi-band processing, the stitching error of high-frequency and low-frequency components is reduced. Use a multi-band fusion algorithm to fuse the stitching results of different frequency bands: use a multi-band fusion algorithm (such as Laplacian pyramid fusion or wavelet fusion) to fuse the stitching results of different frequency bands. Through multi-band fusion, the stitching seam and lighting inconsistency problems are eliminated. Generate a high-quality seamless stitched panoramic image: through multi-band fusion, generate a high-quality seamless stitched panoramic image. Ensure the consistency of high-frequency details and low-frequency backgrounds of the image.

[0092] The multi-band decomposition and fusion algorithm reduces the stitching error of high-frequency and low-frequency components, and improves the stitching quality. The multi-band fusion algorithm eliminates the stitching seam and the problem of inconsistent lighting, realizing seamless stitching. The multi-band processing preserves the high-frequency details and low-frequency background of the image, improving the visual effect of the image. The multi-band decomposition and fusion algorithm enhances the robustness of the algorithm, and is suitable for image stitching under complex scenes and lighting conditions.

[0093] In some embodiments, semantic segmentation is performed on the subgraphs and the normalized panoramic image using a semantic segmentation model: a pre-trained semantic segmentation model (such as DeepLab or U-Net) is used to perform semantic segmentation on the subgraphs and the normalized panoramic image. The image is segmented into different semantic regions (such as sky, ground, building, etc.). The projection position and perspective transformation matrix of the subgraph are optimized according to the semantic information: the projection position and perspective transformation matrix of the subgraph are optimized according to the semantic segmentation results. Ensure the consistency of different semantic regions in the stitching process. Use a semantic weighted fusion algorithm to process the overlapping area: in the overlapping area, use a semantic weighted fusion algorithm to process the stitching of different semantic regions. According to the weight of the semantic region, dynamically adjust the fusion strategy. Generate a seamless panoramic image with consistent semantics: through semantic segmentation and weighted fusion, generate a seamless panoramic image with consistent semantics. Ensure the natural transition of different semantic regions in the stitching process. The semantic segmentation and weighted fusion algorithm ensures the consistency of different semantic regions in the stitching process, improving the naturalness of the image. The semantic weighted fusion algorithm eliminates the stitching seam and the problem of inconsistent semantics, realizing seamless stitching. Semantic segmentation preserves the semantic information of the image, improving the visual effect and practicality of the image. The semantic segmentation and weighted fusion algorithm enhances the robustness of the algorithm, and is suitable for image stitching under complex scenes and semantic conditions.

[0094] The provided panoramic image generation method significantly improves the efficiency and quality of panoramic image generation by optimizing the calibration process and image registration strategy. The method only calibrates the wide-angle camera with a checkerboard to obtain its distortion correction matrix, eliminating the need for individual calibration of each sub-camera in the array camera. Through non-linear distortion correction, the initial panoramic image captured by the wide-angle camera is converted into a standardized image without geometric distortion, serving as a global reference. Traditional methods require individual calibration of each camera in the array, while this method only requires single-lens calibration, significantly simplifying the process.

[0095] At the same time, the method controls the synchronous acquisition of the wide-angle camera and the array camera to ensure the consistency of their data in time (shooting at the same time) and space (fixed relative position). The wide-angle camera provides a global initial panoramic image, and the array camera provides a high-resolution set of local subgraphs. The overlapping area between the subgraphs and the initial panoramic image provides the basis for subsequent registration. The synchronous mechanism avoids misplacement in dynamic scenes, and provides a global coordinate framework directly through the wide-angle image.

[0096] The method further extracts feature information (such as SIFT, ORB, etc.) of the subgraph, matches with the standardized panoramic graph, determines the projection position, perspective transformation matrix (homography matrix) and scale transformation matrix of the subgraph. By projecting to a unified panoramic canvas coordinate system, dynamically adjusting the size and viewing angle of the subgraph, seamless splicing is realized. Using the global information of the wide-angle image to guide the subgraph registration, the error accumulation problem in the traditional method of relying on two-by-two matching between subgraphs is avoided.

[0097] Finally, the method establishes a coordinate system with the standardized panoramic graph, projects the subgraph to the canvas through perspective transformation (solves the viewing angle difference) and scale transformation (unifies the resolution), combines with the fusion algorithm (such as multi-band mixing) to eliminate the joint, and generates a high-resolution panoramic image. Dynamically adjust the size and viewing angle of the subgraph to ensure the balance between local details and global consistency.

[0098] The provided method has at least the following beneficial effects: 1. Significantly reduce the calibration complexity: the traditional method needs to calibrate each sub-camera in the array camera separately, and the present method only needs to calibrate a single wide-angle lens, saving more than 90% of the calibration time and computing resources, especially suitable for large multi-camera array systems.

[0099] 2. Improve registration accuracy and image quality: use the standardized panoramic graph of the wide-angle camera as a global reference, and the subgraph registration is based on the same coordinate system, avoiding the cumulative error of two-by-two matching between traditional subgraphs, significantly reducing the misregistration and ghosting problems.

[0100] 3. Enhance system robustness: time-space synchronous acquisition ensures the consistency of subgraphs and panoramic graphs in dynamic scenes, avoiding motion blur or object displacement caused by shooting time difference, and improving the success rate of splicing in complex scenes (such as pedestrians and vehicles).

[0101] 4. Optimize resource utilization: the wide-angle camera provides a low-resolution global view, and the array camera provides a high-resolution local subgraph. Through intelligent fusion, the resolution and coverage of the panoramic graph are considered, reducing the dependence on the hardware performance of a single camera.

[0102] 5. Suitable for large-scale application scenarios: can be extended to fields such as unmanned aerial vehicle aerial photography, vehicle surround view, VR photography, etc., especially in scenarios that require rapid deployment (such as temporary monitoring systems), saving the cumbersome multi-camera calibration steps and improving practicality.

[0103] In summary, the provided method solves the problems of complex calibration and large matching error in traditional panoramic generation through single-lens calibration, time-space synchronous acquisition and global-local collaborative registration strategy, realizes efficient and high-precision panoramic image generation, and has wide application prospect and commercial value.

[0104] Please refer to Figure 2 ,Figure 2 1 is a schematic diagram of the structure of a panoramic image generation device 200 provided in an embodiment of the present application. The panoramic image generation device 200 is configured to execute the steps of the panoramic image generation method described in any embodiment of the present application. The panoramic image generation device 200 can be a single server or a server cluster, or the panoramic image generation device 200 can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.

[0105] like Figure 2 As shown, the panoramic image generating device 200 includes: The lens calibration unit 201 is configured to perform single-lens calibration on the wide-angle camera according to a checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera; a synchronous acquisition unit 202 configured to control the wide-angle camera and the array camera to perform spatiotemporal synchronous acquisition to obtain an initial panoramic image acquired by the wide-angle camera and a sub-atlas acquired by the array camera; wherein each sub-atlas in the sub-atlas has an overlapping area with the initial panoramic image; a distortion correction unit 203, configured to perform nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion; A feature acquisition unit 204 is configured to acquire sub-image feature information of each sub-image in the sub-image set, and acquire a projection position, a perspective transformation matrix, and a scale transformation matrix corresponding to each sub-image in the standardized panoramic image according to the sub-image feature information; A canvas establishing unit 205 is configured to establish a panoramic canvas coordinate system based on the standardized panoramic image, project each of the sub-images into the panoramic canvas coordinate system according to a corresponding perspective transformation matrix and projection position, adjust the size of each of the sub-images in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image based on the adjusted panoramic canvas coordinate system.

[0106] It should be noted that those skilled in the art will clearly understand that for the convenience and brevity of description, the above-described panoramic image generation device 200 and the specific working process of each module can refer to Figure 1 The corresponding processes in the panoramic image generation method embodiments described in the corresponding embodiments are not repeated here.

[0107] Figure 1 The corresponding panoramic image generation method can be implemented in the form of a computer program. Figure 2 Run on the device shown.

[0108] See also Figure 3 , Figure 3is a structural schematic block diagram of a computer device provided by an embodiment of the present application. The computer device comprises a processor, a memory and a network interface connected through a device bus, wherein the memory can comprise a storage medium and an internal memory.

[0109] The storage medium can store an operating device and a computer program. The computer program comprises program instructions which, when executed, can cause the processor to perform Figure 1 any one of the panoramic image generation methods.

[0110] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.

[0111] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium, and the computer program, when executed by the processor, can cause the processor to perform any one of the panoramic image generation methods.

[0112] The network interface is configured to perform network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. A specific computer device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0114] In one embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps: According to the checkerboard calibration method, a single-lens calibration is performed on the wide-angle camera to obtain a distortion correction matrix corresponding to the wide-angle camera. The wide-angle camera and the array camera are controlled to perform time-space synchronous acquisition, to obtain an initial panoramic image acquired by the wide-angle camera and a sub-image set acquired by the array camera; each sub-image in the sub-image set has an overlapping area with the initial panoramic image; According to the distortion correction matrix, non-linear distortion correction is performed on the initial panoramic image to generate a standardized panoramic image without geometric distortion. Sub-image feature information of each sub-image in the sub-image set is obtained, and a projection position, a perspective transformation matrix and a scale transformation matrix of each sub-image in the standardized panoramic image are obtained according to the sub-image feature information. A panoramic canvas coordinate system is established according to the standardized panoramic image, each sub-image is projected into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, and the corresponding size of each sub-image in the panoramic canvas coordinate system is adjusted according to the scale transformation matrix, and a panoramic image is generated according to the adjusted panoramic canvas coordinate system.

[0115] In some embodiments, the distortion correction matrix is obtained according to the following steps: obtaining a plurality of wide-angle images of the scene from different positions; obtaining a plurality of array images of the scene from different positions; obtaining a plurality of panoramic images of the scene from different positions; and obtaining a plurality of panoramic image distortion correction matrices corresponding to the plurality of panoramic images.

[0116] In some embodiments, the distortion correction matrix is obtained according to the following steps: obtaining a plurality of wide-angle images of the scene from different positions; obtaining a plurality of array images of the scene from different positions; obtaining a plurality of panoramic images of the scene from different positions; and obtaining a plurality of panoramic image distortion correction matrices corresponding to the plurality of panoramic images.

[0117] In some embodiments, the establishing a panorama canvas coordinate system according to the normalized panorama image comprises: taking a geometric center of the normalized panorama image as a coordinate origin, establishing a compound coordinate system of a polar coordinate system and a Cartesian coordinate system; determining a coverage range of the panorama canvas coordinate system according to a field of view angle parameter of the wide-angle camera; establishing a mapping relationship from a pixel coordinate to a panorama coordinate; setting a canvas boundary constraint condition in the Cartesian coordinate system, and establishing the panorama canvas coordinate system according to the compound coordinate system, the coverage range and the mapping relationship.

[0118] In some embodiments, the projecting each of the sub-images into the panorama canvas coordinate system according to the corresponding perspective transformation matrix and the projection position comprises: performing affine transformation on a pixel point set of each of the sub-images according to a translation vector corresponding to the perspective transformation matrix and the projection position; establishing a dynamic weight fusion region in the panorama canvas coordinate system; calculating a projection confidence weight of each of the sub-images in an overlapping projection region corresponding to the overlapping region; performing consistency optimization on a pixel gradient field of the overlapping projection region according to a Poisson fusion algorithm, eliminating a joint effect between the sub-images, and completing the projection of each of the sub-images into the panorama canvas coordinate system.

[0119] In some embodiments, the adjusting a corresponding size of each of the sub-images in the panorama canvas coordinate system according to the scale transformation matrix comprises: analyzing an anisotropic scaling factor corresponding to the scale transformation matrix; calculating a sub-image resolution adjustment ratio according to the anisotropic scaling factor; performing multiple interpolation resampling on the sub-image to adjust an original resolution corresponding to the sub-image to a preset resolution; determining a length-width ratio constraint condition corresponding to the sub-image according to a tolerance threshold determined according to a mounting position tolerance of the array camera, so as to keep the length-width ratio of the sub-image in the scale transformation process in conformity with the length-width ratio constraint condition.

[0120] In some embodiments, the single-lens calibration of the wide-angle camera according to the checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera comprises: arranging a plurality of checkerboard calibration boards in different poses in a field of view range of the wide-angle camera; acquiring a calibration board image corresponding to each of the calibration boards collected by the wide-angle camera, and using Zhang's calibration method to solve a camera intrinsic parameter matrix and a distortion coefficient vector; and constructing the distortion correction matrix according to the camera intrinsic parameter matrix and the distortion coefficient vector.

[0121] The application also provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to enable the processor to implement the panorama image generation method according to any one of the embodiments. Figure 1 The steps of the panorama image generation method according to any one of the embodiments.

[0122] The computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like.

[0123] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for generating a panoramic image, characterized in that: include: Performing single-lens calibration on the wide-angle camera according to the checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera; Controlling the wide-angle camera and the array camera to perform spatiotemporal synchronous acquisition to obtain an initial panoramic image acquired by the wide-angle camera and a sub-atlas acquired by the array camera; wherein each sub-atlas in the sub-atlas has an overlapping area with the initial panoramic image; Performing nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion; Obtaining sub-image feature information of each sub-image in the sub-image set, and obtaining a projection position, a perspective transformation matrix, and a scale transformation matrix corresponding to each sub-image in the standardized panoramic image according to the sub-image feature information; A panoramic canvas coordinate system is established based on the standardized panoramic image, each of the sub-images is projected into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, and the corresponding size of each sub-image is adjusted in the panoramic canvas coordinate system according to the scale transformation matrix, and a panoramic image is generated based on the adjusted panoramic canvas coordinate system.

2. The method according to claim 1, characterized in that The performing nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion includes: Obtaining corresponding radial distortion coefficients and tangential distortion coefficients in the distortion correction matrix; Obtain the mapping relationship between the original pixel points and the corrected pixel points; All pixels of the initial panoramic image are traversed to perform inverse mapping calculation according to the mapping relationship, so as to generate a normalized panoramic image with barrel distortion eliminated according to bilinear interpolation.

3. The method according to claim 1, characterized in that The step of obtaining the projection position, perspective transformation matrix, and scale transformation matrix corresponding to each sub-image in the standardized panoramic image according to the sub-image feature information includes: Acquire a local feature point set of each sub-image and a global feature point set of the normalized panoramic image according to a preset image feature detector; Performing nearest neighbor feature matching on each of the subgraphs to obtain a correspondence between the local feature point set and the global feature point set; Selecting matching point pairs that satisfy the homography constraint in the local feature point set according to the corresponding relationship, and obtaining the homography matrix corresponding to the matching points by solving the problem using the least squares method; Decomposing the homography matrix to obtain the perspective transformation matrix including rotation and translation components and the scale transformation matrix including scaling factors; The projection center coordinates of the sub-image in the normalized panoramic image are determined according to the decomposition results corresponding to the eigenvalues ​​of the homography matrix.

4. The method according to claim 1, wherein The step of establishing a panoramic canvas coordinate system according to the standardized panoramic image includes: Taking the geometric center of the standardized panoramic image as the coordinate origin, establishing a composite coordinate system of a polar coordinate system and a Cartesian coordinate system; Determining a coverage range of the panoramic canvas coordinate system according to a field of view angle parameter of the wide-angle camera; Establish a mapping relationship from pixel coordinates to panoramic coordinates; A canvas boundary constraint condition is set in a Cartesian coordinate system, and the panoramic canvas coordinate system is established according to the composite coordinate system, the coverage range, and the mapping relationship.

5. The method according to claim 1, wherein Projecting each of the sub-images into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position includes: Performing an affine transformation on the pixel point set of each sub-image according to the perspective transformation matrix and the translation vector corresponding to the projection position; Establishing a dynamic weighted fusion region in the panoramic canvas coordinate system; For the overlapping projection area corresponding to the overlapping area, calculating the projection confidence weight of each sub-image in the overlapping projection area; The pixel gradient field of the overlapping projection area is optimized for consistency according to the Poisson fusion algorithm to eliminate the seam effect between the sub-images, and complete the projection of each sub-image into the panoramic canvas coordinate system.

6. The method according to claim 1, characterized in that The adjusting the size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix includes: Analyzing anisotropic scaling factors corresponding to the scale transformation matrix; Calculating a sub-image resolution adjustment ratio according to the anisotropic scaling factor; Performing multiple interpolation resampling on the sub-image to adjust the original resolution corresponding to the sub-image to a preset resolution; A tolerance threshold is determined according to the array camera installation position tolerance, and an aspect ratio constraint condition corresponding to the sub-image is determined according to the tolerance threshold, so as to maintain the aspect ratio of the sub-image in compliance with the aspect ratio constraint condition during the scale transformation process.

7. The method according to claim 1, characterized in that The single-lens calibration of the wide-angle camera according to the checkerboard calibration method to obtain the distortion correction matrix corresponding to the wide-angle camera includes: Arranging a plurality of checkerboard calibration plates of different positions within the field of view of the wide-angle camera; Obtaining a calibration plate image corresponding to each calibration plate captured by the wide-angle camera, and using Zhang's calibration method to solve the camera intrinsic parameter matrix and distortion coefficient vector; The distortion correction matrix is ​​constructed according to the camera intrinsic parameter matrix and the distortion coefficient vector.

8. A panoramic image generating device, characterized in that: The device comprises: A lens calibration unit, configured to perform single-lens calibration on the wide-angle camera according to a checkerboard calibration method to obtain a distortion correction matrix corresponding to the wide-angle camera; a synchronous acquisition unit, configured to control the wide-angle camera and the array camera to perform spatiotemporal synchronous acquisition, to obtain an initial panoramic image acquired by the wide-angle camera and a sub-atlas acquired by the array camera; wherein each sub-atlas in the sub-atlas has an overlapping area with the initial panoramic image; a distortion correction unit, configured to perform nonlinear distortion correction on the initial panoramic image according to the distortion correction matrix to generate a standardized panoramic image without geometric distortion; a feature acquisition unit, configured to acquire sub-image feature information of each sub-image in the sub-image set, and acquire a projection position, a perspective transformation matrix, and a scale transformation matrix corresponding to each sub-image in the standardized panoramic image according to the sub-image feature information; A canvas establishment unit is used to establish a panoramic canvas coordinate system based on the standardized panoramic image, project each of the sub-images into the panoramic canvas coordinate system according to the corresponding perspective transformation matrix and projection position, adjust the corresponding size of each sub-image in the panoramic canvas coordinate system according to the scale transformation matrix, and generate a panoramic image based on the adjusted panoramic canvas coordinate system.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.

Images