Camera panoramic image stitching method

Abstract

The application provides a camera panoramic image splicing method, which comprises the following steps: controlling a rotating camera to shoot 360-degree rotating images under multiple pitch angles; correcting the tilt angles of the images in the same rotating period by using the pitch angles; extracting and matching features of adjacent images in the same rotating period, and calculating a projection transformation matrix between the adjacent images; optimizing the projection transformation matrix of the adjacent images by using a difference method, so that the sum of the re-projection errors of the homonymous points between the adjacent images is minimum; splicing multiple images in the same rotating period into annular panoramic images after projection transformation based on the optimized projection transformation matrix; and splicing multiple annular panoramic images obtained in different rotating periods to obtain a panoramic fusion image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision image processing, and in particular to a camera panoramic image stitching method. BACKGROUND

[0002] Traditional rotation image stitching methods are mostly for images taken vertically, that is, the images are taken perpendicular to the lens optical axis, and then the images are projected onto a cylindrical surface or a spherical surface for stitching. Cameras mounted on an aerial platform are difficult to obtain accurate pitch and yaw angles through a gimbal when observing a scene on the ground at a fixed point in a small-amplitude translation and rotation in the air. When the camera is rotated to take images, the pitch angles of the camera at different periods are different, resulting in serious distortion of the original images. If the traditional panoramic image stitching method is used to project the images onto a cylindrical surface or a spherical surface, the geometric relationship of the ground objects will be severely distorted. The panoramic image obtained by using the horizontal strip stitching method after rectification is still at an inclined angle and cannot restore the overall view of the circular ground and perform inter-period stitching. SUMMARY

[0003] The present application provides a camera panoramic image stitching method to solve the above technical problems.

[0004] One aspect of the present application provides a camera panoramic image stitching method, comprising: controlling a 360° rotation camera to take 360° rotation images at multiple pitch angles, each 360° rotation image at a pitch angle comprising a plurality of images taken in a 360° rotation period; performing tilt angle correction on the images in the same rotation period according to the pitch angle or other attitude parameters; extracting and matching a plurality of feature points from adjacent images, and after eliminating the mis-matching points, calculating the projection transformation matrix between the adjacent images; using the redundancy condition that the images in a period intersect at the beginning and the end, and using a adjustment method to optimize the projection transformation matrix; based on the optimized projection transformation matrix, projecting and transforming the plurality of images in the same rotation period to stitch them into a circular panoramic image; and stitching the plurality of circular panoramic images obtained in different rotation periods to obtain a panoramic fusion image.

[0005] Optionally, the tilt angle correction of the images in the same rotation period using the pitch angle comprises: calculating a tilt angle rotation matrix based on the pitch angle, the tilt angle rotation matrix reflecting the mapping relationship between the image points after the tilt angle correction and the original image points; and using a resampling method to complete the tilt angle correction of the images in the same rotation period based on the tilt angle rotation matrix.

[0006] Optionally, after projecting the multiple images in the same rotation cycle onto a reference plane, the feature extraction and matching are performed on each pair of adjacent images to calculate the projection transformation matrix between the adjacent images, including: performing feature extraction on the adjacent images after the tilt angle correction, and performing feature matching to obtain matching points between the adjacent images; eliminating the mismatched points between the adjacent images to obtain corresponding points between the adjacent images; and calculating the projection transformation matrix between the adjacent images based on the corresponding points between the adjacent images.

[0007] Optionally, the feature extraction and matching performed on the adjacent images after the tilt angle correction to obtain matching points between the adjacent images include: extracting feature points of the adjacent images; calculating the nearest neighbor Euclidean distance and the second nearest neighbor Euclidean distance between the feature points of the adjacent images; calculating the ratio of the nearest neighbor Euclidean distance to the second nearest neighbor Euclidean distance to obtain the similarity of the feature points of the adjacent images; and screening the feature points with a similarity higher than a preset threshold as matching points of the adjacent images.

[0008] Optionally, the elimination of the mismatched points between the adjacent images to obtain corresponding points between the adjacent images includes: repeatedly selecting multiple non-collinear matching points from the matching points of the adjacent images to calculate multiple homography matrices; selecting a homography matrix with the largest number of applicable matching points from the multiple homography matrices as an optimal homography matrix between the adjacent images; and eliminating mismatched points that do not conform to the optimal homography matrix to obtain matching points of the adjacent images.

[0009] Optionally, the optimization of the projection transformation matrix by using a method of adjustment includes: calculating the re-projection coordinates of the corresponding points based on the projection transformation matrix; taking the re-projection coordinates as an initial value as an input of adjustment, taking the first image in the cycle as a reference to optimize the attitude parameters of all images in the cycle to minimize the sum of the re-projection errors of the corresponding points, and recalculating the projection transformation matrix; and iteratively performing the above steps multiple times to obtain an optimal projection transformation matrix.

[0010] Optionally, the projection transformation of the multiple images in the same rotation cycle based on the optimized projection transformation matrix to splice the multiple images into a ring panoramic image includes: projecting the multiple images in the same rotation cycle onto the same reference plane based on the optimized projection transformation matrix; searching for a splicing line between the images; and fusing the multiple images based on the splicing line to eliminate the light difference between the images to obtain the ring panoramic image.

[0011] Optionally, the stitching of the multiple annular panoramic images with different rotation periods to obtain a panoramic fused image comprises: selecting an annular panoramic image with a view angle closest to the center of the annular panoramic image as a reference plane image from the multiple annular panoramic images; extracting homonymic points of the reference plane image and an annular panoramic image adjacent to the reference plane image in a pitch angle, and calculating a projection transformation matrix between the adjacent annular panoramic image and the reference plane image based on the homonymic points; stitching and fusing the adjacent annular panoramic image and the reference plane image based on the projection transformation matrix; and repeating the above steps to sequentially select annular panoramic images of adjacent pitch angles for stitching to obtain the panoramic fused image.

[0012] The above at least one technical scheme adopted in the embodiment of the present application can achieve the following beneficial effects:

[0013] 1. The present application can realize panoramic stitching of 360° rotation images, eliminate distortion interference caused by inclined photography at different pitch angles for image stitching, realize seamless stitching of images within a period and between periods, form a large-range ground monitoring panoramic image, and restore the real geometric relationship of ground objects.

[0014] 2. The present application performs preliminary projection transformation on the images taken by inclination before image registration and stitching, and projects the images to a reference plane to obtain images taken approximately vertically. This scheme reduces the geometric distortion of the images, improves the accuracy of feature extraction and matching of adjacent images, and increases the number of matching feature points. It overcomes the defect that the direct stitching result is an inclined view angle, avoids the problem of serious distortion of the stitched images caused by error transmission in the stitching process, and ensures that the panoramic image after stitching can restore the real appearance of the ground objects.

[0015] 3. The present application designs a process of first stitching images within a period and then stitching images between periods, optimizes the projection transformation matrix of each image within a period based on adjustment method when stitching images within a period, so that the images within a period are accurately and seamlessly stitched together, avoiding the problem of slow calculation caused by global optimization of all images, and improving the calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more completely understand the present application and its advantages, reference will now be made to the following description in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A flowchart schematically showing a camera panoramic image stitching method provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] Embodiments of the present application will be described below with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that the embodiments can be practiced without these specific details. In other instances, well-known structures and techniques have not been described in detail in order to avoid unnecessarily obscuring the concepts of the present application.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, shall be read expansively and without limitation. The terms "comprising," "comprise" and / or "comprised of," and tautological expressions thereof (e.g., "comprising of") will be understood to enable recitations that they do not exclude additional matter.

[0020] All terms used herein (including technical and scientific terms) have meanings that are commonly understood by one of ordinary skill in the art unless otherwise defined. It should be further understood that the terms used herein should be interpreted as having a meaning that is consistent with the understanding of those terms by those having ordinary skill in the art and commonly used by those in the field, and not in an overly literal or overly formal sense unless expressly so defined herein.

[0021] Some of the diagrams and / or flowcharts shown in the drawings are block diagrams and / or flowcharts. It should be understood that some of the blocks in the block diagrams and / or flowcharts, or combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that the instructions executed by the processor can create means for implementing the functions / operations specified in the block diagrams and / or flowcharts.

[0022] Accordingly, the technology of the present application can be implemented in hardware and / or software (including firmware, microcode, etc.). Additionally, the technology of the present application can take the form of a computer program product on a computer-readable medium having instructions executable by an instruction execution system. In the context of the present application, a computer-readable medium can be any medium that can contain, store, communicate, propagate, or transport instructions. For example, the computer-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Specific examples of a computer-readable medium include magnetic storage devices (e.g., magnetic tapes or hard disks (HDDs)), optical storage devices (e.g., compact discs (CD-ROMs)), memories (e.g., random access memories (RAMs) or flash memories), and / or wired / wireless communication links.

[0023] Figure 1 A flowchart of a camera panoramic image stitching method provided by an embodiment of the present application is schematically shown.

[0024] AsFigure 1 As shown, the camera panoramic image stitching method may, for example, include operations S110-S160.

[0025] S110, controlling a 360° rotating camera to capture 360° rotating images at multiple pitch angles, each 360° rotating image including multiple images captured at a 360° rotating period.

[0026] In this embodiment, the camera is placed on a mounting platform, and the camera is rotated by a gimbal or other rotating platform to capture 360° rotating images at multiple pitch angles. The mounting platform provides relatively stable support for the camera, allowing the camera to complete 360° rotation within a certain range of disturbances. Optionally, the mounting platform can be a tethered balloon, a drone, a helicopter, or the like, but is not limited to the above mounting platforms.

[0027] According to the shooting requirements and the field of view of the camera, the interval angle of adjacent images in the same period and adjacent periods is calculated, and multiple images in a period are captured at a certain fixed pitch angle. The pitch angle of the rotating mechanism is then adjusted to complete the image capture in the next period. The images can be numbered during the capture process to mark the adjacency of the images. After all the images are captured, they need to be checked, and the missed areas caused by factors such as drift rotation of the mounting platform need to be recaptured.

[0028] In this embodiment, after the rotating camera is adjusted to a pitch angle, the rotating camera is controlled to capture images at the pitch angle for a rotating period. The images in the rotating period include multiple local images of a ring-shaped area on the ground. If the local images are sequentially stitched according to the positional relationship, the image of the ring-shaped area can be obtained.

[0029] S120, using the pitch angle of the rotating mechanism to correct the tilt angle of the original images in the same rotating period.

[0030] The images are captured at a certain pitch angle and roll angle, and there is a large geometric distortion between the images and the actual scene. Moreover, the distortion directions of adjacent images are not consistent. Direct stitching will cause the distortion of the images to accumulate and deviate from the true ground appearance.

[0031] In this embodiment, before the image registration and stitching, the tilt angle of the images captured at a tilt angle is corrected, and the multiple images in the same rotating period are projected onto a reference plane to obtain images that are approximately captured vertically. This scheme reduces the geometric distortion of the images, improves the accuracy of feature extraction and matching of adjacent images, and increases the number of matching feature points. It overcomes the defect that the direct stitching result is at a tilt angle, avoids the problem of severe distortion of the stitched images caused by error propagation during the stitching process, and ensures that the panoramic image after stitching can restore the true appearance of the ground objects.

[0032] The tilt angle correction of the image according to the pitch angle of each image can be expressed as:

[0033]

[0034] In the formula, (x, y) and (x', y') represent image points on the original image and the image after the tilt angle correction respectively, and R is a tilt angle rotation matrix:

[0035]

[0036] In the formula, θ is the pitch angle.

[0037] In the embodiment, first, a tilt angle rotation matrix is calculated based on the pitch angle, and the tilt angle rotation matrix reflects the mapping relationship between the image points of the image after the tilt angle correction and the image points of the original image; then, the tilt angle correction of the image is completed by using the resampling method based on the tilt angle rotation matrix, so that the images in the same rotation period are projected into the same plane.

[0038] Optionally, when the high-precision IMU is mounted on the mounting platform, the IMU and the azimuth angle of the turntable can be comprehensively used to calculate a more accurate rotation matrix, and the images are projected onto the horizontal plane by using the collinearity equation.

[0039] S130, feature extraction and matching are performed on the adjacent images in the period, and a projection transformation matrix between the adjacent images is calculated.

[0040] To calculate the projection transformation matrix between the images, corresponding homonymic points between the adjacent images need to be determined. Specifically, the extraction and matching of the homonymic points between the adjacent images and the calculation of the projection transformation matrix include operations S131-S133.

[0041] S131, feature extraction is performed on the adjacent images after the tilt angle correction, and feature matching is performed to obtain matching points between the adjacent images.

[0042] The mounting platform itself has disturbances, which cause the distortion sizes and directions between the images to be inconsistent, and the number of images to be spliced is large. In the embodiment, the SURF operator with good robustness and high operation efficiency is used to extract feature points. Optionally, in the case of a small number of images, the SIFT operator with higher accuracy can be used; in different scenarios, the ORB, FAST, etc. operators can also be used.

[0043] In the embodiment, the feature matching can use the nearest neighbor Euclidean distance ratio method, but is not limited to this method. The nearest neighbor Euclidean distance and the second nearest neighbor Euclidean distance between the feature points of the adjacent images in the same rotation period are calculated. The Euclidean distance is defined as follows:

[0044]

[0045] The ratio of the nearest neighbor Euclidean distance to the second nearest neighbor Euclidean distance is calculated, and the similarity is determined according to the ratio. The ratio of the nearest neighbor Euclidean distance to the second nearest neighbor Euclidean distance is expressed as:

[0046]

[0047] In this embodiment, the feature points with a similarity higher than a preset threshold are screened as the matching points of the adjacent images.

[0048] In S132, the mismatched points between the adjacent images are removed. The RANSAC algorithm (Random Sample Consensus) can be used to remove the mismatched points, but the method is not limited to this.

[0049] In this embodiment, a homography matrix of two images is calculated from a plurality of (for example, 4) non-collinear matching points randomly selected from the matching points. The matching points of image A1 and image A2 are (x1, y1) and (x2, y2), respectively. The projection relationship between the images can be expressed by a homography matrix H:

[0050]

[0051] The number of matching points satisfying the model and the cost function W are calculated while the homography matrix is calculated. If H is the optimal matrix, then:

[0052]

[0053] A plurality of homography matrices are calculated from a plurality of non-collinear matching points randomly selected from the adjacent images. The optimal homography matrix, i.e., the matrix with the minimum cost function W, is selected from the plurality of homography matrices. The matching points that do not satisfy the optimal homography matrix, i.e., the mismatched points, are removed, and the corresponding points of the adjacent images are obtained.

[0054] In S133, the projection transformation matrix between the adjacent images is calculated based on the corresponding points between the adjacent images. For example, the homography matrix, the essential matrix, or the like is used as the transformation matrix to establish the mapping relationship between the adjacent images.

[0055] In this embodiment, the homography matrix is used to establish the mapping relationship between the adjacent images, but the method is not limited to this.

[0056] According to the shooting order, the projection transformation matrix is calculated based on the corresponding points between all the adjacent images in the current period. It should be noted that the first image in the period extracts the corresponding points and calculates the projection transformation matrix not only with the second image but also with the last image in the current period.

[0057] In S140, the projection transformation matrix in the same rotation period is optimized by using the adjustment method, so that the sum of the re-projection errors of the corresponding points between the adjacent images is minimized.

[0058] The re-projection coordinates of the homonymous points are calculated using the projection transformation matrix, the redundancy condition of the image end-to-end in the same rotation period is used, and the projection transformation matrix is optimized by using adjustment method to eliminate the cumulative error. The re-projection coordinates of the homonymous points between the images in the same rotation period are calculated based on the projection transformation matrix, and are used as the input of adjustment. The attitude parameters of all images in the period are optimized with the first image as the reference, so that the sum of the re-projection error of the homonymous points is minimized, and the projection transformation matrix is recalculated. The optimal inter-image transformation matrix is obtained by multiple iterations.

[0059] S150, based on the optimized projection transformation matrix, the multiple images in the same rotation period are projected and transformed to be spliced into a ring panoramic image.

[0060] Specifically, S150 includes S151-S153.

[0061] S151, based on the optimized projection transformation matrix, the multiple images in the same rotation period are projected and transformed to be spliced into a ring panoramic image.

[0062] S152, searching for the splicing line between the images.

[0063] In this embodiment, the optimal stitching line searching algorithm is used to find a curve with the smallest difference between the two sides of the image overlap area, which divides the overlap area into two parts.

[0064] S153, based on the splicing line, using a fusion algorithm to fuse the images to eliminate the light difference between the images, and obtaining the ring panoramic image. For example, using multi-band fusion algorithm, weighted average method, PCA algorithm, etc.

[0065] In this embodiment, a Laplace pyramid can be used to perform weighted fusion on the images of the same layer of the overlap area, and a multi-band fusion algorithm is used to fuse the images of different layers of the overlap area according to different rules. However, this method is not limited to this method.

[0066] S160, splicing the multiple ring panoramic images obtained in different rotation periods to obtain a panoramic fusion image.

[0067] Specifically, S160 includes S161-S165.

[0068] S161, performing ring panoramic image splicing on the images of each pitch angle period, and the processing method is the same as S120-S150, to obtain multiple ring panoramic images.

[0069] S162, selecting a ring panoramic image with a line of sight direction closest to the vertical downward view from the multiple ring panoramic images as a reference plane image.

[0070] S163, extracting the reference plane image and the homonymy points of the adjacent annular panoramic image, and calculating the projection transformation matrix between the adjacent annular panoramic image and the reference plane image based on the homonymy points, the processing method being same as S130.

[0071] S164, based on the projection transformation matrix, splicing and fusing the adjacent annular panoramic image and the reference plane image, the processing method being same as S150.

[0072] S165, repeating the above steps, and sequentially selecting the adjacent annular panoramic images of different pitch angles to splice, to obtain the panoramic fused image.

[0073] According to the method provided by the embodiment of the present application, the image splicing in the period is performed first, and then the image splicing between the periods is performed, the projection transformation matrix of each image in the period is adjusted by combining the adjustment method during the image splicing in the period, the images in the period are accurately and seamlessly spliced together, the problem of slow calculation speed caused by the global optimization of all images is avoided, and the calculation efficiency is improved. The method can realize the panoramic splicing of the 360° rotating image, eliminate the distortion interference caused by the inclined photography of the images under different pitch angles, realize the seamless splicing of the images in the period and between the periods, form the large-range ground monitoring panoramic image, and restore the real geometric relationship of the ground objects.

[0074] Those skilled in the art can understand that the features described in various embodiments and / or claims of the present application can be combined or / and integrated, even if such combination or integration is not explicitly described in the present application. In particular, the features described in various embodiments and / or claims of the present application can be combined and / or integrated in various combinations, without departing from the spirit and teaching of the present application. All these combinations and / or integrations fall within the scope of the present application.

[0075] Although the present application has been shown and described with respect to certain exemplary embodiments thereof, it should be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the spirit and scope of the present application as defined by the appended claims and their equivalents. Therefore, the scope of the present application should not be limited to the above-described embodiments, but should be determined by the appended claims only, and should be defined by the equivalents of the appended claims.

Claims

1. A camera panorama image stitching method, characterized in that, The method comprises the following steps: controlling a 360° rotation camera to capture 360° rotation images at multiple pitch angles, each 360° rotation image at a pitch angle comprising multiple images captured in a 360° rotation period; performing tilt angle correction on the images in the same rotation period according to the pitch angles; extracting and matching features of adjacent images in the same rotation period to calculate a projection transformation matrix between the adjacent images; optimizing the projection transformation matrix by using a method of adjustment to minimize the reprojection error of corresponding points between adjacent images; based on the optimized projection transformation matrix, performing projection transformation on the multiple images in the same rotation period to stitch the multiple images into a ring-shaped panoramic image; stitching multiple ring-shaped panoramic images obtained in different rotation periods to obtain a panoramic fusion image; the tilt angle correction on the images in the same rotation period according to the pitch angles comprises: calculating a tilt angle rotation matrix based on the pitch angles, the tilt angle rotation matrix reflecting the mapping relationship between the image points of the tilt angle corrected image and the original image points; based on the tilt angle rotation matrix, performing tilt angle correction on the images in the same rotation period by using a resampling method to project the images in the same rotation period onto the same plane; the optimization of the projection transformation matrix by using the method of adjustment comprises: calculating the reprojection coordinates of the corresponding points based on the projection transformation matrix; taking the reprojection coordinates as the input of adjustment and taking the first image in the period as the reference to optimize the attitude parameters of all images in the period to minimize the reprojection error of the corresponding points, and recalculating the projection transformation matrix; iterating the above steps multiple times to obtain the optimal projection transformation matrix.

2. The method of claim 1, wherein, the extraction and matching of features of adjacent images in the same rotation period to calculate the projection transformation matrix between the adjacent images comprises: extracting features of the adjacent images after tilt angle correction and performing feature matching to obtain matching points between the adjacent images; eliminating the mismatched points between the adjacent images to obtain corresponding points between the adjacent images; calculating the projection transformation matrix between the adjacent images based on the corresponding points between the adjacent images.

3. The method of claim 2, wherein, the extraction and matching of features of the adjacent images after tilt angle correction to obtain matching points between the adjacent images comprises: extracting feature points of adjacent images; calculating the nearest neighbor Euclidean distance and the second nearest neighbor Euclidean distance between the feature points of the adjacent images; calculating the ratio of the nearest neighbor Euclidean distance to the second nearest neighbor Euclidean distance to obtain the similarity of the feature points of the adjacent images; selecting feature points with a similarity higher than a preset threshold as matching points of the adjacent images.

4. The method of claim 2, wherein, the elimination of mismatched points between the adjacent images to obtain corresponding points between the adjacent images comprises: randomly selecting multiple non-collinear matching points multiple times from the adjacent images to calculate multiple homography matrices; selecting a homography matrix with the largest number of applicable matching points from the multiple homography matrices as the optimal homography matrix between the adjacent images; eliminating mismatched points in the matching points that do not conform to the optimal homography matrix to obtain corresponding points of the adjacent images.

5. The method of claim 1, wherein, The method comprises the following steps: projecting the multiple images in the same rotation period to the same reference plane based on the optimized projection transformation matrix; searching for a splicing line between the images; fusing the multiple images based on the splicing line to eliminate the light difference between the images, and obtaining the annular panoramic image.

6. The method of claim 1, wherein, The method for splicing the multiple annular panoramic images obtained in different rotation periods to obtain a panoramic fused image comprises the following steps: selecting an annular panoramic image with a line-of-sight direction closest to the vertical downward view from the multiple annular panoramic images as a reference plane image; extracting homonymous points of the reference plane image and an annular panoramic image adjacent to the reference plane image in the pitch angle, and calculating a projection transformation matrix between the adjacent annular panoramic image and the reference plane image based on the homonymous points; splicing and fusing the adjacent annular panoramic image and the reference plane image based on the projection transformation matrix; repeating the above steps to sequentially select annular panoramic images adjacent in the pitch angle for splicing, and obtaining the panoramic fused image.

Images