Aerial image stitching method and device

Abstract

The present invention proposes a method and device for stitching aerial images; wherein the method includes: constructing an imaging model based on the internal and external parameter matrices of the camera; removing the Z-axis component in the rotation matrix of the imaging model to simplify the imaging model; constructing the correspondence between images based on the simplified imaging model, and obtaining a motion imaging model by solving; stitching images based on the motion imaging model, and constraining the relevant parameters in the motion imaging model during the stitching process so that the stitched image is projected to the X-Y plane to the maximum extent and is projected to an appropriate reference plane. The camera imaging model proposed in the present invention has fewer unknown parameters, which greatly reduces the time for global optimization adjustment. And by analyzing the causes of the errors, adding constraint terms to the objective function of parameter optimization, and correcting the motion parameters of the image, the stitching result is closer to the real image, and a better image stitching result can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of aerial photography and image synthesis, and in particular to a method and device for stitching aerial images. Background Art

[0002] In recent years, with the exploration of aerospace and other fields by human beings and the rapid development of digital camera technology, the stitching technology based on aerial and satellite images has begun to enter the human vision.

[0003] In 2001, Capel studied how to combine the information contained in multiple overlapping images of the same scene to produce high-quality images. The study adopted an 8-parameter homography projection model. On this basis, in 2003, Brown et al. further proposed an object recognition technology based on invariant local features to select matching images, and proposed a probabilistic model to verify it, thereby realizing fully automatic stitching of disordered images. This method has achieved good stitching results. However, the disadvantage is that the obtained stitched images have certain image deformation, especially when the number of images is large, the edge images will be more obviously distorted. Summary of the invention

[0004] In order to solve the technical problems mentioned in the background technology, in one aspect of the present invention, a method for stitching aerial images is proposed, the method comprising: establishing a first mapping relationship between spatial coordinate points and corresponding pixel coordinate points based on a parameter matrix of a camera, a rotation matrix of an observation image plane, and a translation matrix; removing the Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain a second mapping relationship; based on the second mapping relationship, obtaining a motion imaging model by solving the correspondence between corresponding spatial coordinate points in two images obtained by shooting the same point in space from different angles and viewpoints; based on the motion imaging model, constructing a misalignment cost function between multiple images to be stitched, and performing image stitching by executing parameter constraints while minimizing the misalignment cost function, so that the stitched image is projected to the XY plane to the maximum extent and is projected to an appropriate reference plane.

[0005] In one or more embodiments, the first mapping relationship between spatial coordinate points and corresponding pixel coordinate points is established based on the camera's parameter matrix, the rotation matrix of the observation image plane, and the translation matrix, including: based on the camera's parameter matrix, the rotation matrix of the observation image plane, and the translation matrix, a first mapping relationship between corresponding spatial coordinate points in three-dimensional space and projection points on the camera imaging plane is established using homogeneous coordinate transformation.

[0006] In one or more embodiments, removing the Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain the second mapping relationship includes: setting the Z-axis component of the homogeneous coordinates of the spatial coordinate point in the first mapping relationship to 0; constructing a second mapping relationship equivalent to the first mapping relationship, and removing the corresponding homogeneous coordinate point and the Z-axis component of the rotation matrix in the second mapping relationship.

[0007] In one or more embodiments, based on the second mapping relationship, the motion imaging model is obtained by solving the correspondence relationship between corresponding points in two images obtained by shooting the same point in space from different angles and viewpoints, including: based on the second mapping relationship, constructing a third mapping relationship from the first spatial coordinate point observed from the first viewpoint and the first angle to the first imaging plane, and a fourth mapping relationship from the first spatial coordinate point observed from the second viewpoint and the second angle to the second imaging plane; obtaining a first expression for the first coordinate point in the third mapping relationship and a second expression for the first coordinate point in the fourth mapping relationship through inverse transformation; and obtaining the motion imaging model based on the equivalent relationship between the first expression and the second expression.

[0008] In one or more embodiments, image stitching is performed by executing parameter constraints while minimizing the misregistration cost function so that the stitched image is projected to the XY plane to the maximum extent and is projected to an appropriate reference plane, including: constraining the pitch angle and roll angle in the rotation matrix of the imaging model in the process of minimizing the misregistration cost function, and making the pitch angle and roll angle approach 0 so that the stitched image is projected to the XY plane to the maximum extent; and constraining the focal length and image plane offset in the parameter matrix of the camera of the imaging model, and making the focal length approach 1 and the image plane offset approach 0 so that the stitched image is projected to an appropriate reference plane.

[0009] In another aspect of the present invention, an aerial image stitching device is proposed, including: a modeling module, the modeling module is configured to establish a first mapping relationship between a spatial coordinate point and a corresponding pixel coordinate point based on a parameter matrix of a camera, a rotation matrix of an observation image plane, and a translation matrix; a model optimization module, the model optimization module is configured to remove the Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain a second mapping relationship; an imaging model generation module, the imaging model generation module is configured to obtain a motion imaging model by solving the correspondence between corresponding spatial coordinate points in two images obtained by shooting the same point in space from different angles and viewpoints; an image stitching module, the image stitching module is configured to construct a misalignment cost function between multiple images to be stitched, and perform image stitching by executing parameter constraints while minimizing the misalignment cost function, so that the stitched image is projected to the XY plane to the maximum extent and is projected to an appropriate reference plane.

[0010] In one or more embodiments, the modeling module is further configured to: based on the parameter matrix of the camera, the rotation matrix of the observation image plane and the translation moment, use homogeneous coordinate transformation to establish a first mapping relationship from corresponding spatial coordinate points in three-dimensional space to projection points on the camera imaging plane.

[0011] In one or more embodiments, the model optimization module is further configured to: set the Z-axis component of the homogeneous coordinates of the spatial coordinate point in the first mapping relationship to 0; and construct a second mapping relationship equivalent to the first mapping relationship, and reduce the dimensions of the corresponding homogeneous coordinate points and rotation matrices in the second mapping relationship.

[0012] In one or more embodiments, the imaging model generation module is further configured to: construct, based on the second mapping relationship, a third mapping relationship from the first spatial coordinate point observed from the first viewpoint and the first angle to the first imaging plane, and a fourth mapping relationship from the first spatial coordinate point observed from the second viewpoint and the second angle to the second imaging plane; obtain, through inverse transformation, a first expression for the first coordinate point in the third mapping relationship and a second expression for the first coordinate point in the fourth mapping relationship; and obtain a motion imaging model by solving the equivalent relationship between the first expression and the second expression.

[0013] In one or more embodiments, the image stitching module is further configured to: constrain the pitch angle and roll angle in the rotation matrix of the imaging model in the process of minimizing the misregistration cost function, and make the pitch angle and roll angle approach 0 so that the stitched image is projected to the XY plane as much as possible; and constrain the focal length and image plane offset in the parameter matrix of the camera of the imaging model, and make the focal length approach 1 and the image plane offset approach 0 so that the stitched image is projected to an appropriate reference plane.

[0014] The beneficial effects of the present invention include: the method of the present invention proposes a 6-parameter camera imaging model by analyzing the motion characteristics of the camera during drone aerial photography. Compared with the homography matrix model, the model has fewer unknown parameters and greatly reduces the time for global optimization adjustment. By analyzing the causes of errors, constraints are added to the objective function of parameter optimization, and the image motion parameters are corrected, so that the stitching result is closer to the real image, and a better stitching result can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0016] Figure 1 The following is a flowchart of the aerial image stitching method of the present invention;

[0017] Figure 2 A schematic diagram illustrating the principle of constructing a motion forming model of the present invention;

[0018] Figure 3(a) is a map of the cement plant surrounding area obtained by the global stitching method based on Capel's homography matrix imaging model;

[0019] FIG3( b ) is a map of the surrounding area of ​​a cement plant obtained based on the method of the present invention;

[0020] Fig. 4(a) is a map of the plain area obtained by the global stitching method based on Capel's homography matrix imaging model;

[0021] FIG4( b ) is a map of a plain area obtained based on the method of the present invention;

[0022] Figure 5(a) shows the grassland area obtained by the global stitching method based on Capel's homography matrix imaging model;

[0023] FIG5( b ) is a map of a plain area obtained based on the method of the present invention;

[0024] FIG6( a ) is a first reference graph for initializing model parameters;

[0025] FIG6( b ) is a second reference graph for initializing model parameters;

[0026] FIG7( a ) is a map of a plain area obtained by using a global stitching method of Capel's homography matrix imaging model based on the second reference image;

[0027] FIG7( b ) is a plain area map obtained by using the method of the present invention based on the second reference map;

[0028] Figure 8 The figure is a schematic structural diagram of an aerial image stitching device of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.

[0031] The present invention proposes a method for automatically stitching disordered images by selecting matching images based on object recognition technology based on invariant local features and verifying them through a probability model. The method is simpler and faster than the one for stitching aerial images, and the stitched images obtained are smoother and have no obvious distortion. The method of the present invention is as follows:

[0032] See also Figure 1 , Figure 1The flowchart is a workflow of the aerial image stitching method of the present invention. In one embodiment, the aerial image stitching method of the present invention includes step S1, establishing a first mapping relationship between spatial coordinate points and corresponding pixel coordinate points based on the parameter matrix of the camera, the rotation matrix of the observation image plane, and the translation matrix; step S2, removing the Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain a second mapping relationship; step S3, based on the second mapping relationship, obtaining a motion imaging model by solving the corresponding relationship between corresponding spatial coordinate points in two images obtained by shooting the same point in space from different angles and viewpoints; step S4, based on the motion imaging model, constructing a misalignment cost function between multiple images to be stitched, and performing image stitching by minimizing the misalignment cost function while executing parameter constraints, so that the stitched image is projected to the XY plane to the maximum extent and is projected to an appropriate reference plane.

[0033] In this embodiment, the present invention can improve the speed of processing the imaging model and thus improve the speed of image stitching by simplifying the imaging model, making it more suitable for the fast movement characteristics of aerial photography; and then in the process of image stitching, the relevant parameters in the simplified imaging model are constrained to adjust each image that needs to be stitched, so that the stitched image is smoother and no obvious distortion will occur.

[0034] In a further embodiment, for step S1, the present invention obtains the parameter matrix of the camera in the aerial photography device and the rotation matrix and translation matrix of the observation image plane through a gyroscope and other devices, and then uses homogeneous coordinate transformation to establish a first mapping relationship from the corresponding spatial coordinate point in the three-dimensional space to the projection point on the camera imaging plane based on the parameter matrix of the camera, the rotation matrix and the translation matrix of the observation image plane. Among them, the first mapping relationship is the most original imaging model. The formula of the first mapping relationship mentioned above is expressed as follows:

[0035]

[0036] in, is the homogeneous coordinate of the coordinate point Q in three-dimensional space, is the homogeneous coordinate of the projection point q in the image, s is the scale factor, and H is the transformation matrix. Specifically, the transformation matrix H is mainly affected by the camera parameter matrix M and the physical transformation matrix W of the object plane. The elements of the parameter matrix M and the physical transformation matrix W are as follows:

[0037]

[0038] Among them, f x 、f y is the focal length of the camera, c x 、c yis the camera's image plane offset.

[0039] W=[R t]. (3)

[0040] Among them, R is the rotation matrix, t is the translation matrix, and W is used to represent the rotation and translation relationship between the world coordinate system of the three-dimensional space and the camera imaging coordinate system.

[0041] More specifically, the translation matrix t = [t x t y t z ] T , is a 3×3 rotation matrix, where each element is about three rotation parameters The function of , its expansion is shown in formula (4):

[0042]

[0043] Where ψ is the roll angle, is the pitch angle, and κ is the yaw angle. From formula (1), we can see that the pixel coordinates of any point in the world coordinate system on the image can be determined by the internal and external parameters of the camera during imaging. Then the first mapping relationship can be further expressed as:

[0044]

[0045] That is, the relationship between the pixel coordinates of any point on the image and the world coordinates is:

[0046]

[0047] In fact, since the aerial images are taken at a high distance, the undulations of the ground surface or the depth of the objects can be ignored, and the photographed ground scene can be approximately regarded as a plane. The method of the present invention uses this feature to simplify the first mapping relationship.

[0048] In a further embodiment, for step S2, the present invention simplifies the first mapping relationship by setting the Z-axis component of the homogeneous coordinates of the spatial coordinate point in the first mapping relationship to 0; then constructing a second mapping relationship equivalent to the first mapping relationship, and removing the Z-axis component of the corresponding homogeneous coordinate point and the rotation matrix in the second mapping relationship. The formula transformation process is as follows:

[0049]

[0050] remember is a simplified form of the external parameter matrix W, is a 3×3 matrix. Then the second mapping relationship is expressed as:

[0051]

[0052] The first mapping relationship and the second mapping relationship mentioned above are both imaging models established under static conditions. In order to adapt to the characteristics of aerial photography motion, the present invention also needs to construct a motion imaging model based on the imaging model under static conditions.

[0053] In a further embodiment, for step S3, the present invention constructs a third mapping relationship from the first spatial coordinate point observed from the first viewpoint and the first angle to the first imaging plane, and a fourth mapping relationship from the first spatial coordinate point observed from the second viewpoint and the second angle to the second imaging plane based on the second mapping relationship; obtains the first expression of the first coordinate point in the third mapping relationship and the second expression of the first coordinate point in the fourth mapping relationship through inverse transformation; obtains the motion imaging model based on the equivalent relationship between the first expression and the second expression. For a clearer explanation of step S3, please refer to Figure 2 , Figure 2 The schematic diagram is a schematic diagram illustrating the principle of constructing a motion forming model of the present invention. The specific process of constructing a motion imaging model of the present invention is as follows:

[0054] In the world coordinate system, the camera's motion imaging model can be determined by taking two photos of the same point in space from different angles and viewpoints, and based on the relationship between the pixel coordinates of the corresponding points in the two images. The specific process includes:

[0055] Let P be a point on the earth plane. O and O' are the camera shooting positions, i.e., two imaging viewpoints. Imaging planes I1 and I2 represent two aerial images obtained by shooting the ground from viewpoints O and O', and p1 and p2 are the pixel points of point P in imaging planes I1 and I2, whose three-dimensional homogeneous coordinates are (x1, y1, 1) and (x2, y2, 1) respectively. Based on the imaging model represented by (8), the coordinates of the pixel points p1 and p2 of the spatial coordinate point P (X, Y, Z) under viewpoint O and viewpoint O' can be expressed as:

[0056]

[0057]

[0058] In the formula, S is the scale factor, M represents the intrinsic parameter matrix of the camera, and Represents the simplified external parameter matrix of the camera when imaging at viewpoint O and viewpoint O', [r 11 r 21 ] and [r 12 r 22] are the first two columns of the rotation matrix that transforms the world coordinate system into the coordinate system of the image planes I1 and I2. t1 and t2 are the coordinates of viewpoint O and viewpoint O' in the world coordinate system, that is, the translation of the optical center of the camera relative to the ground. Inversely transform formulas (9) and (10) to obtain formula (11):

[0059]

[0060] Therefore, for any point in the world coordinate system, there is an equal relationship between the image point coordinates p1(x1, y1) and p2(x2, y2) in the two images as expressed by equation (12):

[0061]

[0062] Formula (12) can be simplified by removing the variable s to obtain formula (13):

[0063]

[0064] Therefore, the motion imaging model can be further simplified as formula (14):

[0065]

[0066] In a further embodiment, after obtaining the motion imaging model, for step S4, the present invention constrains the pitch angle and roll angle in the rotation matrix of the imaging model in the process of minimizing the misalignment cost function, and makes the pitch angle and roll angle approach 0 so that the stitched image is projected to the XY plane to the maximum extent; and constrains the focal length and image plane offset in the parameter matrix of the camera of the imaging model, and makes the focal length approach 1 and the image plane offset approach 0 so that the stitched image is projected to an appropriate reference plane. Among them, the misalignment cost function is used for image stitching, and the image stitching and constraint process of the present invention is as follows:

[0067] Any pair of pixels q on image m and image n ni ,q mi There is a transformation relationship as shown in formula (15) (16):

[0068]

[0069]

[0070] in, represents the transformation matrix of the nth image, Represents the transformation matrix of the mth image. i=1,2...L mn L mn is the number of pixel pairs between image n and image m. ni =[xni y ni 1] T ,q mi =[x mi y mi 1] T are the midpoints q' in space respectively ni =[x' ni y' ni 1] T ,q' mi =[x' mi y' mi 1] T Projected pixel points on image n and image m.

[0071] Due to q ni ,q mi is a pair of pixels, q' ni With q' mi In fact, they are the same point in space, so the error between pixel pairs can be expressed as ε i :

[0072]

[0073] Then the objective function of parameter optimization can be expressed as f(p):

[0074]

[0075] in, is the parameter vector, M = 6K + 4. By minimizing the function f(p), the four camera internal parameters and the six external parameters of each image can be solved.

[0076] In order to make the stitching result as close to the real image as possible, the unknown variables are analyzed and constraints are added. In the process of minimizing f(p), as the pitch angle ψ and the roll angle As the image projection on the XY plane decreases, the error accumulation f(p) also decreases, but the stitching result will be far away from the real image due to its rotation around the X and Y axes. In order to make the stitching image closer to the real image, the pitch angle ψ and roll angle Add constraints to maximize the projection of the stitched image onto the XY plane, making ψ→0. Avoid image rotation around the X and Y axes. Yaw angle κ∈[-π,π]. Translation variable t z Indicates the height of the camera, that is, the zoom of the image. z The smaller it is, the lower the altitude of the drone is, and the smaller the image needs to be scaled. zThe larger the value is, the higher the altitude of the drone is. The smaller the size of the same surface object is in the image, that is, the larger the image needs to be magnified. z Add constraints to the variables so that t z →1, so that the stitching result is finally projected onto the appropriate reference plane. x and t y represents the translation in the X and Y directions, t x ∈(-∞,+∞), t y ∈(-∞,+∞). For the camera internal parameters, f x →1,f y →1,c x →0,c y →0. In summary, add constraint Ereg(p) to the objective function f(p):

[0077]

[0078]

[0079] Among them, α is the rotation parameter ψ and The larger the α is, the smoother the image stitching result will be and the smaller the deformation will be, but the error between images will increase. β adjusts the size of the stitching result. The smaller β represents the variable t z When the proportion is reduced and f(p) is minimized, t z The value of α and β will also decrease, shrinking the final stitching result. By properly adjusting α and β, better stitching results will be obtained.

[0080] Through the method steps in the above embodiments, the present invention obtains a motion imaging model with fewer parameters based on a simplified imaging model, so that the processing speed of the motion imaging model in the subsequent stitching process is faster, and in the process of stitching images, by adding parameter constraints, the obtained stitched image is smoother and does not have obvious distortion. In order to verify the present invention, multiple groups of comparative experiments were also carried out.

[0081] Test 1

[0082] In Experiment 1 of the present invention, three groups of images were used for stitching experiments to compare the inventive method with the global stitching method based on the homography matrix imaging model proposed by Capel. Among them, the first group of images Data1 was collected in the surrounding area of ​​a cement plant, with a pixel resolution of 3680×2456. The second group Data2 is an aerial image of a plain with an image resolution of 3840×5760. The third group of images Data3 was taken in a grassland area, with a corresponding image resolution of 736×491. The three image sets have 61, 66 and 30 images, respectively. Figures 3(a) and 3(b), Figures 4(a) and 4(b), and Figures 5(a) and 5(b) respectively give the stitching results of the three groups of image data. Image (a) in each group of images is obtained by the global stitching method of Capel's homography matrix imaging model, and image (b) is obtained by stitching according to the method of the present invention. Comparing the three groups of experimental results, it can be seen that the image obtained by the stitching method proposed in this paper is smoother than the image obtained by the stitching method proposed by Capel, and the image distortion is relatively smaller.

[0083] In order to more intuitively compare the distortion of the stitching images obtained by the method of the present invention and Capel's global stitching method based on the homography matrix imaging model, the present invention evaluates the stitching effect of the image by calculating the difference between the coordinates of the corresponding pixel points of the two. The average difference err calculation formula is as shown in formula (21).

[0084]

[0085] In addition, the present invention also tests the time and accuracy of the splicing process of the two methods. Please refer to Table 1 for specific results. Table 1 is as follows:

[0086] Table 1

[0087]

[0088] It can be seen from Table 1 that the time required for the method proposed in the present invention is much less than the time required for Capel's stitching method. The main reason for the time reduction is that for K images, the imaging model proposed in the present invention has 6K+4 unknown parameters (6-parameter camera imaging model), while the homography matrix imaging model has 8K. In the process of global optimization and adjustment, the speed of algorithm iteration convergence is inversely proportional to the number of unknown parameters. The more unknown parameters there are, the slower the algorithm convergence speed, so the stitching method of the present invention has a faster processing speed. Secondly, it can be seen from the rightmost column of Table 1 that compared with Capel's stitching method, the average error value of the stitching result of the method of the present invention is smaller, that is, in terms of image stitching accuracy, the method of the present invention is better.

[0089] In further experiments, in order to verify the robustness of the method of the present invention, the present invention also conducted experiments in which different reference images were selected to obtain different initialization parameters, and different optimization results obtained by comparing different initialization parameters.

[0090] Test 2

[0091] The present invention uses two images in Data2 as reference images to conduct stitching experiments to verify the influence of the reference images on the method of the present invention. The two reference images are shown in Figure 6(a) and Figure 6(b) respectively. Taking Figure 6(a) as the reference image, the images obtained by Capel's method and the method of the present invention are shown in Figure 4(a) and Figure 4(b) respectively; taking Figure 6(b) as the reference image, the images obtained by Capel's method and the method of the present invention are shown in Figure 7(a) and Figure 7(b) respectively.

[0092] By comparing Figure 4(a) with Figure 7(a), it can be seen that the stitching result using Capel's eight-parameter model is greatly affected by the reference image, and Figure 7(a) undergoes a larger deformation compared to Figure 4(a); while by comparing Figure 4(b) with Figure 7(b), it can be seen that the results obtained by the method of the present invention are less different, which verifies that the present invention has good robustness in the selection of reference images.

[0093] It can be seen from the above experiment 2 that the stitching effect of the Capel method depends largely on the selection of the reference image. Once the shooting plane of the reference image deviates from the ground level, such as when the camera shoots the reference image with an obvious tilt or the ground height of the area covered by the reference image fluctuates greatly, the stitching result obtained by the Capel optimization method is severely distorted. However, in the parameter optimization model with constraint items proposed by the present invention, the reference image has less influence on the stitching effect and has good robustness.

[0094] Based on the above test results, during the imaging process of aerial images, changes in shooting angles caused by the movement of drones in the air and changes in light intensity under external conditions may cause errors in image stitching. The method of the present invention analyzes the motion characteristics of the camera during drone aerial photography and proposes a 6-parameter camera imaging model. Compared with the homography matrix model, the model has fewer unknown parameters and greatly reduces the time for global optimization adjustment. And by analyzing the causes of errors, adding constraints to the objective function of parameter optimization, and correcting the image motion parameters, the stitching result is closer to the real image, and a better stitching result is obtained.

[0095] Based on the aerial image stitching methods in the above embodiments, the present invention also proposes an aerial image stitching device. Figure 8 As shown, Figure 8The present invention is a schematic diagram of the structure of an aerial image stitching device. The aerial image stitching device of the present invention includes: a modeling module, the modeling module is configured to establish a first mapping relationship between a spatial coordinate point and a corresponding pixel coordinate point based on a parameter matrix of a camera, a rotation matrix of an observation image plane, and a translation matrix; a model optimization module, the model optimization module is configured to remove the Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain a second mapping relationship; an imaging model generation module, the imaging model generation module is configured to obtain a motion imaging model by solving the corresponding relationship between corresponding spatial coordinate points in two images obtained by shooting the same point in space from different angles and viewpoints; an image stitching module, the image stitching module is configured to construct a misalignment cost function between multiple images to be stitched, and perform image stitching by executing parameter constraints while minimizing the misalignment cost function, so that the stitched image is projected to the XY plane to the maximum extent and is projected to an appropriate reference plane.

[0096] In a further embodiment, the modeling module is configured to: based on the camera parameter matrix, the rotation matrix of the observation image plane and the translation moment, use homogeneous coordinate transformation to establish a first mapping relationship from corresponding spatial coordinate points in three-dimensional space to projection points on the camera imaging plane.

[0097] In a further embodiment, the model optimization module is configured to: set the Z-axis component of the homogeneous coordinates of the spatial coordinate points in the first mapping relationship to 0; and construct a second mapping relationship equivalent to the first mapping relationship, and reduce the dimensions of the corresponding homogeneous coordinate points and rotation matrices in the second mapping relationship.

[0098] In a further embodiment, the imaging model generation module is configured to: construct, based on the second mapping relationship, a third mapping relationship from the first spatial coordinate point observed from the first viewpoint and the first angle to the first imaging plane, and a fourth mapping relationship from the first spatial coordinate point observed from the second viewpoint and the second angle to the second imaging plane; obtain, through inverse transformation, a first expression of the first coordinate point in the third mapping relationship and a second expression of the first coordinate point in the fourth mapping relationship; and obtain the motion imaging model by solving the equivalent relationship between the first expression and the second expression.

[0099] In a further embodiment, the image stitching module is configured to: constrain the pitch angle and roll angle in the rotation matrix of the imaging model in the process of minimizing the misalignment cost function, and make the pitch angle and roll angle approach 0 so that the stitched image is projected to the XY plane as much as possible; and constrain the focal length and image plane offset in the parameter matrix of the camera of the imaging model, and make the focal length approach 1 and the image plane offset approach 0 so that the stitched image is projected to an appropriate reference plane.

[0100] The device of the present invention generates a 6-parameter camera imaging model by utilizing the motion characteristics of the camera during drone aerial photography. Compared with the homography matrix model, the model has fewer unknown parameters and greatly reduces the time for global optimization adjustment. By analyzing the causes of errors, the image splicing module adds constraint terms to the objective function of parameter optimization, and corrects the image motion parameters, so that the splicing result is closer to the real image, and a better splicing result is obtained.

[0101] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0102] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.

[0103] The serial numbers of the embodiments disclosed in the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0104] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the embodiments of the present invention as above, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for stitching aerial images, characterized in that: The method comprises: A first mapping relationship between a spatial coordinate point and a corresponding pixel coordinate point is established based on a parameter matrix of the camera, a rotation matrix of the observation image plane, and a translation matrix; Removing the Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain a second mapping relationship; Based on the second mapping relationship, a motion imaging model is obtained by solving a correspondence relationship between corresponding spatial coordinate points in two images obtained by shooting the same point in space from different angles and viewpoints; Based on the motion imaging model, construct a misalignment cost function between multiple images to be stitched, and perform image stitching by executing parameter constraints while minimizing the misalignment cost function, so that the stitched image is projected to the XY plane to the maximum extent and is projected to an appropriate reference plane; The method of establishing a first mapping relationship between spatial coordinate points and corresponding pixel coordinate points based on the parameter matrix of the camera, the rotation matrix of the observation image plane, and the translation matrix comprises: establishing a first mapping relationship between corresponding spatial coordinate points in three-dimensional space and projection points on the camera imaging plane by using homogeneous coordinate transformation based on the parameter matrix of the camera, the rotation matrix of the observation image plane, and the translation matrix; The removing the Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain the second mapping relationship includes: setting the Z-axis component of the homogeneous coordinates of the spatial coordinate point in the first mapping relationship to 0; constructing a second mapping relationship equivalent to the first mapping relationship, and removing the Z-axis component of the corresponding homogeneous coordinate point and the rotation matrix in the second mapping relationship; The motion imaging model is obtained by solving the correspondence between corresponding points in two images obtained by shooting the same point in space from different angles and viewpoints based on the second mapping relationship, including: based on the second mapping relationship, constructing a third mapping relationship from a first spatial coordinate point observed from a first viewpoint and a first angle to a first imaging plane, and a fourth mapping relationship from a first spatial coordinate point observed from a second viewpoint and a second angle to a second imaging plane; obtaining a first expression of the first spatial coordinate point in the third mapping relationship and a second expression of the first spatial coordinate point in the fourth mapping relationship through inverse transformation; and obtaining the motion imaging model by solving the equivalent relationship between the first expression and the second expression.

2. The aerial image stitching method according to claim 1, characterized in that: The image stitching is performed by executing parameter constraints while minimizing the misregistration cost function so that the stitched image is maximally projected onto the XY plane and onto an appropriate reference plane, including: In the process of minimizing the misalignment cost function, constraining the pitch angle and the roll angle in the rotation matrix of the motion imaging model, and making the pitch angle and the roll angle approach 0 so that the stitched image is projected onto the XY plane to the maximum extent; and Constraining the focal length and image plane offset in the parameter matrix of the camera of the motion imaging model, and making the focal length approach 1 and the image plane offset approach 0 so that the stitched image is projected onto a proper reference plane.

3. An aerial image stitching device, characterized in that: include: A modeling module, wherein the modeling module is configured to establish a first mapping relationship between a spatial coordinate point and a corresponding pixel coordinate point based on a parameter matrix of the camera, a rotation matrix of an observation image plane, and a translation matrix; A model optimization module, wherein the model optimization module is configured to remove a Z-axis component in the rotation matrix to simplify the first mapping relationship to obtain a second mapping relationship; An imaging model generation module, the imaging model generation module is configured to obtain a motion imaging model by solving a correspondence relationship between corresponding spatial coordinate points in two images obtained by shooting the same point in space from different angles and viewpoints; An image stitching module, the image stitching module is configured to construct a misalignment cost function between a plurality of images to be stitched, and stitch the images by executing parameter constraints while minimizing the misalignment cost function, so that the stitched image is maximally projected onto the XY plane and onto an appropriate reference plane; The modeling module is further configured to: establish a first mapping relationship from corresponding spatial coordinate points in three-dimensional space to projection points on the camera imaging plane using homogeneous coordinate transformation based on the parameter matrix of the camera, the rotation matrix of the observation image plane and the translation moment; The model optimization module is further configured to: set the Z-axis component of the homogeneous coordinates of the spatial coordinate point in the first mapping relationship to 0; and construct a second mapping relationship equivalent to the first mapping relationship, and remove the Z-axis component of the corresponding homogeneous coordinate point and the rotation matrix in the second mapping relationship; The imaging model generation module is further configured to: construct, based on the second mapping relationship, a third mapping relationship from the first spatial coordinate point observed at the first viewpoint and the first angle to the first imaging plane, and a fourth mapping relationship from the first spatial coordinate point observed at the second viewpoint and the second angle to the second imaging plane; The first expression of the first spatial coordinate point in the third mapping relationship and the second expression of the first spatial coordinate point in the fourth mapping relationship are respectively obtained through inverse transformation; and the motion imaging model is obtained by solving the equivalent relationship between the first expression and the second expression.

4. The aerial image stitching device according to claim 3, characterized in that: The image stitching module is further configured to: In the process of minimizing the misalignment cost function, constraining the pitch angle and the roll angle in the rotation matrix of the motion imaging model, and making the pitch angle and the roll angle approach 0 so that the stitched image is projected onto the XY plane to the maximum extent; and Constraining the focal length and image plane offset in the parameter matrix of the camera of the motion imaging model, and making the focal length approach 1 and the image plane offset approach 0 so that the stitched image is projected onto a proper reference plane.

Images