A method for rapid generation of orthophotos based on scene aerial images and sparse point clouds

Through sparse point cloud processing based on aerial images, orthophotos are directly generated, which solves the problems of complicated steps and time-consuming in the existing technology, and achieves rapid and hole-free orthophoto generation, which is suitable for a variety of application scenarios.

CN114998496BActive Publication Date: 2024-11-26BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210592065.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-27
Publication Date
2024-11-26
Estimated Expiration
2042-05-27

AI Technical Summary

Technical Problem

The prior art produces orthophotos, which are complicated and time-consuming, especially when mapping based on three-dimensional models, are prone to hole problems.

Method used

By generating sparse point clouds based on aerial images, and using the SfM algorithm to obtain the camera pose and preliminary mapping map, block processing and texture mapping, orthophotos are directly generated, and the dense point cloud generation and triangular gridization steps are omitted.

Benefits of technology

It realizes rapid generation of orthophotos, reduces generation time, avoids hole problems, and reduces costs, and is suitable for a variety of application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114998496B_ABST
    Figure CN114998496B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of image processing and environmental modeling and simulation, and provides a method for quickly generating orthophotos based on aerial photos of scenes and sparse point clouds. The method for quickly generating orthophotos includes: generating sparse point clouds and camera poses using aerial photos; straightening the camera point cloud and the sparse point cloud and obtaining a preliminary mapping map; dividing the preliminary mapping map into blocks and finding the original point cloud and aerial photo corresponding to each block; finding the most suitable aerial photo for the block mapping map and performing texture mapping; merging the block mapping results to obtain an orthophoto. The present invention generates orthophotos through aerial photos, without the need to purchase professional equipment, greatly reducing limitations and being more economical and practical; orthophotos can be generated using only sparse point clouds, without the need to generate dense point clouds and three-dimensional grids, effectively shortening the time; while ensuring the resolution of the image, it provides technical support for other subsequent applications such as smart cities and three-dimensional reconstruction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of image processing and environmental modeling and simulation, and in particular to a method for quickly generating orthophotos based on scene aerial images and sparse point clouds. Background Art

[0002] Automated scene modeling based on aerial images to form high-resolution scene models is an important requirement in many fields. Among them, orthophotos, as one of the final output products of modeling, are remote sensing images with orthophoto properties, which can be used to make various image maps and measurements. How to generate higher-definition and more accurate orthophotos under limited camera conditions and modeling time is the key to various applications.

[0003] At present, orthophoto production is mainly carried out with special equipment. For example, optical mechanical correctors can be used to produce center-projected aerial photographs of flat ground; orthophoto projection devices, such as various optical mechanical differential correctors and CNC analytical orthophoto projectors, can be used to produce aerial or aerospace photographs with large fluctuations. However, special equipment is expensive, and when merging data with other data sources, due to the inconsistency of the data source, further steps such as target matching and corresponding correction are required, and the subsequent steps are complicated and time-consuming.

[0004] In recent years, the rapid development of drones and drone-mounted cameras has led to rapid progress in various applications based on aerial images, such as scene 3D modeling. Most existing 3D modeling software can also generate orthophotos, mainly by reconstructing 3D scenes from aerial images and then further generating orthophotos. However, the steps of 3D scene reconstruction are complicated, including sparse point cloud generation based on aerial images, dense point cloud generation from sparse point clouds, dense point cloud triangulation to generate mesh models, scene texture mapping to generate scene models, and further generation of orthophotos based on scene models. There are not only many redundant steps, but also the steps of dense point cloud generation and scene meshing are time-consuming. At the same time, since the orthophoto is mapped on the basis of the 3D model, when the model has holes in some locations where the texture is relatively simple, it will also cause holes in the orthophoto. Summary of the invention

[0005] In view of this, the present invention provides a method for quickly generating orthophotos based on a scene aerial photo and its sparse point cloud, so as to solve the problem in the prior art that an orthophoto of a scene cannot be generated as quickly and completely as possible using aerial photos.

[0006] The present invention provides a method for quickly generating orthophotos based on scene aerial images and sparse point clouds, comprising:

[0007] S1 uses aerial images to generate sparse point clouds and camera poses;

[0008] S2 aligns the camera point cloud and the sparse point cloud and obtains a preliminary mapping;

[0009] S3 divides the preliminary map into blocks and finds the original point cloud and aerial image corresponding to each block;

[0010] S4 finds the most suitable aerial image for the block mapping image and performs texture mapping;

[0011] S5 merges the block mapping results to obtain the orthophoto.

[0012] Furthermore, the S1 includes:

[0013] S11 prepares the aerial image to be processed;

[0014] S12 uses the SfM (Structure from motion) algorithm to perform aerial triangulation to obtain the camera pose and scene sparse point cloud.

[0015] Further, the S2 includes:

[0016] S21 uses the camera pose to fit the camera plane and calculate the rotation matrix;

[0017] S22 uses a rotation matrix to straighten the sparse point cloud, maps the sparse point cloud to the same plane, and obtains a preliminary mapping image.

[0018] Further, the S21 includes:

[0019] S211 reads the camera position coordinates obtained by the SfM algorithm, each coordinate is a camera point in three-dimensional space, performs least squares plane fitting on all camera points, and obtains the camera plane equation, which is in the form of z=ax+by+c;

[0020] S212 obtains the normal vector A(a, b, -1) of the camera plane based on the camera plane equation, performs a dot product of the normal vector of the camera plane and the unit vector B(0, 0, 1) in the positive direction of the Z axis, and calculates the rotation angle α according to the dot product result:

[0021] A·B=|A|*|B|*cosα;

[0022] S213 obtains a rotation axis vector C based on the cross product of the normal vector of the camera plane and the unit vector in the positive direction of the Z axis. The expression of the rotation axis vector is as follows:

[0023] A×B=C

[0024] Among them, C(C x ,C y ,C z ) is the rotation axis vector;

[0025] S214 uses the rotation angle and the rotation axis vector to finally obtain the rotation matrix M, which is:

[0026]

[0027] Where α is the rotation angle, C(C x ,C y ,C z ) is the rotation axis vector, and M is the rotation matrix.

[0028] Further, the S22 includes:

[0029] S221 performs rotation calculation on all points of the sparse point cloud based on the rotation matrix, so as to straighten the sparse point cloud. The point P (P x ,P y ,P z ) After calculation, the point P′(P′) in the straightened point cloud model is obtained. x ,P′ y ,P′ z ), the coordinate calculation method of the point cloud model before and after alignment is as follows:

[0030] P′ x =M[0][0]*P x +M[0][1]*P y +M[0][2]*P z

[0031] P′ y =M[1][0]*P x +M[1][1]*P y +M[1][2]*P z

[0032] P′ z =M[2][0]*P x +M[2][1]*P y +M[2][2]*P z

[0033] S222 obtains the z value of each point in the point cloud model based on the straightened point cloud model, and uses the z value as the height of each point;

[0034] S223 determines the size of the orthophoto according to the scale of the sparse point cloud and actual application requirements; traverses the x value and y value of each point in the straightened point cloud model, determines the value range of x and y, obtains xmin, xmax, ymin, ymax, which are the minimum and maximum values ​​of x, the minimum and maximum values ​​of y, respectively, and determines the image size of the generated orthophoto in combination with the actual requirements and the number of points in the point cloud model.

[0035]

[0036]

[0037] Where num is the number of points in the point cloud model, k is a coefficient determined according to actual needs, X and Y represent the image size, and the minimum value of X and Y, Min(X,Y), is obtained according to the above formula. The maximum value of X and Y, Max(X,Y), is calculated proportionally through the ratio of the value range of the point cloud x value and y value, and the image size of the orthophoto is obtained;

[0038] S224 determines the coordinate position (i, j) of the point in the orthophoto image corresponding to the point through the position information represented by the x value and y value of the point in the point cloud model, completes mapping the sparse point cloud to the same plane, and obtains a preliminary mapping map.

[0039]

[0040]

[0041] in, Indicates the correspondence between the unit length in the point cloud model and the pixel coordinate system.

[0042] S225 counts the number of valid cells in the edge rows and columns of the preliminary mapping image. If the valid number is less than the set threshold, it means that the number of point clouds generated in this area is too small and is not suitable for orthophoto generation. Such edges are cropped and the mapping image size is modified.

[0043] Further, the S3 includes:

[0044] S31 divides the map into blocks;

[0045] S311 specifies that the block division meets the following conditions:

[0046] (1) Each divided block contains at least 10 sparse points;

[0047] (2) The area of ​​the divided block scene is not larger than the area of ​​the scene in a single aerial photo. Under the premise of meeting condition (1), the block scene is subdivided;

[0048] (3) Based on the actual aerial image size and the mapping image size, the mapping image of size p*q is divided into m*n blocks of size r*s, where m*r=p, n*s=q. If a block does not meet the requirement of 10 sparse points, the block is merged with the block with the least sparse points among the four adjacent blocks until condition (1) is met.

[0049] S312: for each point in each block, find the corresponding original sparse point cloud based on the mapping relationship between the point cloud and the mapping graph, and obtain the corresponding original sparse point cloud block;

[0050] S32 finds the original aerial image corresponding to the original sparse point cloud for each block;

[0051] S321 finds the original corresponding aerial atlas of each sparse point and its position in each aerial image from the SfM result file, and finds multiple aerial images containing corresponding points of all points in the same block;

[0052] S322 reads the coordinate values ​​of all points in the aerial image corresponding to all points in the same block, retains the maximum and minimum coordinate values ​​in the aerial image, and captures the aerial image within the coordinate range to obtain a screenshot of the original position of the aerial image corresponding to a sparse point cloud block.

[0053] Further, the S4 includes:

[0054] S41 finds the aerial photo with the smallest deflection and the most center corresponding to each block;

[0055] S411 obtains the rotation angle of the camera relative to the ground during the aerial photography process according to the POS data of the aerial photos or the camera pose obtained by the SfM algorithm, and finds several aerial photos with the smallest rotation angle;

[0056] S412 Considering that the center of the aerial image has the smallest distortion and is closest to the top-down view, the original position screenshot of the aerial image corresponding to the block is combined to find the aerial image whose original position screenshot is closest to the center of the aerial image;

[0057] S42 finds all points in the aerial image corresponding to each sparse point in the block map;

[0058] S421 finds the correspondence between the points in the sparse point cloud and the feature points in the aerial photo from the SfM result file, combines the mapping relationship between the points in the sparse point cloud and the points in the block mapping image, and finds the position of each sparse point in the mapping image of the different blocks in the corresponding feature point of the target aerial photo;

[0059] S422 uses one-to-one matching point pairs to form a corresponding point set;

[0060] S43 calculates a homography matrix H between the block map and the aerial image using the corresponding point set;

[0061] S431 imports the point pairs of the corresponding target aerial image and the block mapping image into the calculation equation group of the homography matrix H;

[0062] S432 solves the calculation equation group to obtain a homography matrix H to describe the coordinate transformation relationship between the mapping image and the aerial image;

[0063] S44 uses the homography matrix H to fill the aerial image texture into the mapping image.

[0064] S441 calculates each pixel Q(Q x ,Q y ) After the homography transformation using the homography matrix, the position Q′(Q′) of the mapping image is x ,Q′ y ), the specific calculation is as follows:

[0065] Let V1 be a three-dimensional column vector, where V1[0] = Q y , V1[1]=Q x , V1[2]=1, let V2 be the three-dimensional column vector after transformation by the homography matrix H:

[0066] V2=H*V1

[0067] The position coordinates of the points after homography transformation are:

[0068] Q′ x =V2[1] / V2[2]

[0069] Q′ y =V2[0] / V2[2]

[0070] S442 replaces the RGB values ​​of the pixels in the transformed positions in the mapping image with the RGB values ​​of the pixels in the aerial image screenshot, and fills the original aerial image texture into the mapping image.

[0071] Further, the S5 includes:

[0072] S51 removes the portion of the block filling result corresponding to each block that exceeds the original corresponding block, retains only the filling result of the original block part, and obtains a single block mapping result after removing the redundant portion;

[0073] S52 combines all the individual block mapping results according to the original block division method to obtain an orthophoto.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. The present invention does not require a special projection device and can generate orthophotos through aerial photos, which greatly reduces limitations and is more economical and practical.

[0076] 2. The present invention omits the time-consuming processes of dense point cloud generation, triangulation and texture mapping in traditional three-dimensional scene reconstruction, and only uses sparse point clouds to complete the generation of orthophotos, effectively shortening the time.

[0077] 3. The present invention can set some parameters according to actual needs, such as the size of the final orthophoto, etc., which not only ensures the resolution of the image, but also provides technical support for other subsequent applications such as smart cities and three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 It is a flow chart of a method for quickly generating orthophotos based on scene aerial images and sparse point clouds provided by the present invention;

[0080] Figure 2 It is a detailed flow chart of a method for quickly generating orthophotos based on scene aerial images and sparse point clouds provided by the present invention;

[0081] Figure 3 It is an aerial photo provided by the present invention;

[0082] Figure 4 is a schematic diagram of a scene sparse point cloud generated by the SfM algorithm provided by the present invention;

[0083] Figure 5 It is the camera pose and sparse point cloud side view provided by the present invention;

[0084] Figure 6 It is a side view of the sparse point cloud after rotation and straightening provided by the present invention;

[0085] Figure 7 is a preliminary mapping diagram provided by the present invention;

[0086] Figure 8 It is a schematic diagram of the block division result provided by the present invention;

[0087] Fig. 9 is a schematic diagram of a sparse point cloud corresponding to a certain mapping block provided by the present invention;

[0088] Fig.10 It is a schematic diagram of an original aerial image corresponding to a sparse point cloud block provided by the present invention;

[0089] Fig.11 It is a screenshot of the original position of the aerial image corresponding to a sparse point cloud block provided by the present invention;

[0090] Fig.12 It is a schematic diagram of filling the original aerial image texture into the mapping image provided by the present invention;

[0091] Fig.13 is a schematic diagram of the mapping result after removing the redundant parts provided by the present invention;

[0092] Fig.14 It is a schematic diagram of the orthophoto provided by the present invention;

[0093] Fig.15 It is a schematic diagram of orthophotos of other software showing holes in the water area provided by the present invention. DETAILED DESCRIPTION

[0094] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0095] A method for quickly generating orthophotos based on a scene aerial image and its sparse point cloud according to the present invention will be described in detail below with reference to the accompanying drawings.

[0096] Figure 1 The present invention provides a flowchart of a method for quickly generating orthophotos based on scene aerial images and sparse point clouds.

[0097] Figure 2 It is a detailed flow chart of a method for quickly generating orthophotos based on scene aerial images and sparse point clouds provided by the present invention.

[0098] like Figure 1 As shown, the orthophoto rapid generation method includes:

[0099] S1 uses aerial images to generate sparse point clouds and camera poses;

[0100] S11 prepares the aerial image to be processed;

[0101] Figure 3 The present invention provides an aerial picture.

[0102] Figure 4 It is a schematic diagram of a scene sparse point cloud generated by the SfM algorithm provided by the present invention.

[0103] Figure 5 It is a side view of the camera pose and sparse point cloud provided by the present invention.

[0104] S12 uses the SfM (Structure from motion) algorithm to perform aerial triangulation to obtain the camera pose and scene sparse point cloud.

[0105] S2 aligns the camera point cloud and the sparse point cloud and obtains a preliminary mapping;

[0106] S21 uses the camera pose to fit the camera plane and calculate the rotation matrix;

[0107] S21 includes:

[0108] S211 reads the camera position coordinates obtained by the SfM algorithm, each coordinate is a camera point in three-dimensional space, performs least squares plane fitting on all camera points, and obtains the camera plane equation, which is in the form of z=ax+by+c;

[0109] S212 obtains the normal vector A(a, b, -1) of the camera plane based on the camera plane equation, performs a dot product of the normal vector of the camera plane and the unit vector B(0, 0, 1) in the positive direction of the Z axis, and calculates the rotation angle α according to the dot product result:

[0110] A·B=|A|*|B|*cosα;

[0111] S213 obtains the rotation axis vector C based on the cross product of the normal vector of the camera plane and the unit vector in the positive direction of the Z axis. The expression of the rotation axis vector is as follows:

[0112] A×B=C

[0113] Among them, C(C x ,C y ,C z ) is the rotation axis vector,

[0114] S214 uses the rotation angle and the rotation axis vector to finally obtain the rotation matrix M, which is:

[0115]

[0116] Where α is the rotation angle, C(C x ,C y ,C z ) is the rotation axis vector, and M is the rotation matrix.

[0117] S22 uses a rotation matrix to straighten the sparse point cloud, maps the sparse point cloud to the same plane, and obtains a preliminary mapping image.

[0118] S22 includes:

[0119] Figure 6 This is a side view of the sparse point cloud after rotation and correction provided by the present invention.

[0120] S221 performs rotation calculation on all points of the sparse point cloud based on the rotation matrix, thereby straightening the sparse point cloud.x ,P y ,P z ) After calculation, the point P′(P′) in the straightened point cloud model is obtained. x ,P′ y ,P′ z ), the coordinate calculation method of the point cloud model before and after alignment is as follows:

[0121] P′ x =M[0][0]*P x +M[0][1]*P y +M[0][2]*P z

[0122] P′ y =M[1][0]*P x +M[1][1]*P y +M[1][2]*P z

[0123] P′ z =M[2][0]*P x +M[2][1]*P y +M[2][2]*P z

[0124] The side view of the sparse point cloud after rotation and correction is as follows Figure 6 As shown, compared to Figure 5 , it can be seen that the point cloud model has been straightened.

[0125] S222 obtains the z value of each point in the point cloud model based on the rectified point cloud model, and uses it as the height of each point;

[0126] S223 determines the size of the orthophoto according to the scale of the sparse point cloud and actual application requirements; traverses the x value and y value of each point in the straightened point cloud model, determines the value range of x and y, obtains xmin, xmax, ymin, ymax, which are the minimum and maximum values ​​of x, the minimum and maximum values ​​of y, respectively, and determines the image size of the generated orthophoto in combination with the actual requirements and the number of points in the point cloud model.

[0127]

[0128]

[0129] Where num is the number of points in the point cloud model, k is a coefficient determined according to actual needs, X and Y represent the image size, and the minimum value of X and Y, Min(X,Y), is obtained according to the above formula. The maximum value of X and Y, Max(X,Y), is calculated proportionally through the ratio of the value range of the point cloud x value and y value, and the image size of the orthophoto is obtained;

[0130] Figure 7 It is a preliminary mapping diagram provided by the present invention.

[0131] S224 determines the coordinate position (i, j) of the point in the orthophoto image corresponding to the point through the position information represented by the x value and y value of the point in the point cloud model, completes mapping the sparse point cloud to the same plane, and obtains a preliminary mapping map.

[0132]

[0133]

[0134] in, Indicates the correspondence between the unit length in the point cloud model and the pixel coordinate system. Due to the limitation of image size and the difference in point cloud height, a certain pixel unit (i, j) usually corresponds to multiple point sets with similar x and y values. The present invention sets the RGB value of the pixel unit to be equal to the RGB value of the point with the maximum z value in the set. In this way, the content observed in the mapped plane only retains the points with higher height in the point cloud model, thereby completing the preliminary mapping of the point cloud to the plane;

[0135] S225 counts the number of valid cells in the edge rows and columns of the preliminary mapping image. If the valid number is less than a certain threshold, it means that the number of point clouds generated in this area is too small and is not suitable for orthophoto generation. Such edges are cropped and the mapping image size is modified.

[0136] S3 divides the preliminary map into blocks and finds the original point cloud and aerial image corresponding to each block;

[0137] S3 includes:

[0138] S31 divides the map into blocks,

[0139] Figure 8 It is a schematic diagram of the block division result provided by the present invention.

[0140] Fig. 9 It is a schematic diagram of a sparse point cloud corresponding to a certain mapping block provided by the present invention.

[0141] S311 specifies that the block division meets the following conditions:

[0142] (1) Each divided block contains at least 10 sparse points;

[0143] (2) The area of ​​the divided block scene is not larger than the area of ​​the scene in a single aerial photo. Under the premise of meeting condition (1), the block scene is subdivided;

[0144] (3) Combining the actual aerial image size and the mapping image size, the mapping image of size p*q can be divided into m*n blocks of size r*s, where m*r=p, n*s=q. If a block does not meet the requirement of 10 sparse points, the block is merged with the block with the least sparse points among the four adjacent blocks until condition (1) is met. The division result is shown in the figure. Figure 8 As shown;

[0145] like Fig. 9 As shown,

[0146] S312: for each point in each block, find the corresponding original sparse point cloud based on the mapping relationship between the point cloud and the mapping graph, and obtain the corresponding original sparse point cloud block;

[0147] S32 finds the original aerial image corresponding to the original sparse point cloud for each block.

[0148] Fig.10 It is a schematic diagram of an original aerial image corresponding to a sparse point cloud block provided by the present invention;

[0149] S321 finds the original corresponding aerial atlas of each sparse point and its position in each aerial image from the SfM result file, and finds multiple aerial images containing corresponding points of all points in the same block;

[0150] Specifically, each point in the sparse point cloud is obtained by matching and triangulating the corresponding feature points detected in the aerial image in the previous SfM process. Therefore, the aerial atlas to which each sparse point originally corresponds and its position in each aerial image can be found from the SfM result file. Usually, one point corresponds to multiple aerial images. Since all points in the same block are close to each other, they usually appear together in multiple aerial images.

[0151] Fig.11 It is a screenshot of the original position of the aerial image corresponding to a sparse point cloud block provided by the present invention.

[0152] S322 reads the coordinate values ​​of all points in the aerial image corresponding to all points in the same block, retains the maximum and minimum coordinate values ​​in the aerial image, and captures the aerial image within the coordinate range to obtain a screenshot of the original position of the aerial image corresponding to a sparse point cloud block.

[0153] For all points in the same aerial image corresponding to all points in the same block, read their coordinate values ​​in the aerial image, retain the maximum and minimum coordinate values, and intercept the aerial image within the coordinate range to obtain the original position screenshot of the aerial image corresponding to a sparse point cloud block. Fig.11 shown.

[0154] S4 finds the most suitable aerial image for the block mapping image and performs texture mapping;

[0155] S4 includes:

[0156] S41 finds the aerial photo with the smallest deflection and the most center corresponding to each block;

[0157] S411 obtains the rotation angle of the camera relative to the ground during the aerial photography process according to the POS data of the aerial photos or the camera pose obtained by the SfM algorithm, and finds several aerial photos with the smallest rotation angle;

[0158] Taking oblique photography as an example, oblique photography uses multiple cameras on the same drone to collect images from different angles, such as vertical and multiple oblique angles. The corresponding common deflection angles include 0°, 30°, 45°, etc. Considering that the orthophoto is similar to looking down at the model from a vertical angle, the aerial image with the smallest deflection of each block, such as the aerial image taken at a deflection angle of 0°, is found as the initial goal.

[0159] S412 Considering that the center of the aerial image has the smallest distortion and is closest to the top-down view, the original position screenshot of the aerial image corresponding to the block is combined to find the aerial image whose original position screenshot is closest to the center of the aerial image;

[0160] Since there is a certain area overlap between aerial images, there may be multiple aerial images with similar deflection that capture the same block, such as multiple aerial images taken when the deflection angle is 0°. Considering that the position of the block in the aerial image is different due to different camera positions during shooting, for example, the same block may be distributed in the upper left corner, the middle, and the lower right corner of the aerial image in three adjacent aerial images of the same channel. Given that the distortion in the middle of the aerial image is the smallest and closest to the top-down view, the original position screenshot of the aerial image corresponding to the block is combined to find the aerial image whose original position screenshot is closest to the center of the aerial image, which is the target aerial image.

[0161] S42 finds all points in the aerial image corresponding to each sparse point in the block map;

[0162] S421 finds the correspondence between the points in the sparse point cloud and the feature points in the aerial photo from the SfM result file, combines the mapping relationship between the points in the sparse point cloud and the points in the block mapping image, and finds the position of each sparse point in the mapping image of the different blocks in the corresponding feature point of the target aerial photo;

[0163] S422 uses one-to-one matching point pairs to form a corresponding point set;

[0164] S43 calculates a homography matrix H between the block map and the aerial image using the corresponding point set;

[0165] S431 imports the corresponding point pairs of the aerial image and the block map into the calculation equation group of the homography matrix H;

[0166] S432 solves the calculation equation group to obtain a homography matrix H to describe the coordinate transformation relationship between the mapping image and the aerial image;

[0167] Specifically, homography is a concept in projective geometry, also known as projective transformation. It maps points on a projective plane to another projective plane, and maps straight lines to straight lines, with the property of preserving lines, so that two images can be associated through the homography matrix. The homography matrix can be applied in image correction, image stitching, and perspective conversion. The homography matrix can be calculated from a series of corresponding points on two projective planes. The corresponding point pairs in step S422 are imported into the calculation equation group of the homography matrix to obtain the homography matrix H, which can be used to describe the conversion relationship between the mapping image and the aerial image.

[0168] Fig.12 It is a schematic diagram provided by the present invention for filling the original aerial image texture into the mapping image.

[0169] S44 uses the homography matrix H to fill the aerial image texture into the mapping image.

[0170] S441 calculates each pixel Q(Q x ,Q y ) After the homography transformation using the homography matrix, the position Q′(Q′) of the mapping image is x ,Q′ y ), the specific calculation is as follows:

[0171] Let V1 be a three-dimensional column vector, where V1[0] = Q y , V1[1]=Q x , V1[2]=1, let V2 be the three-dimensional column vector after transformation by the homography matrix H:

[0172] V2=H*V1

[0173] The position coordinates of the points after homography transformation are:

[0174] Q′ x =V2[1] / V2[2]

[0175] Q′ y =V2[0] / V2[2]

[0176] S442 replaces the RGB values ​​of the pixels in the transformed positions in the mapping image with the RGB values ​​of the pixels in the aerial image screenshot, and fills the original aerial image texture into the mapping image.

[0177] S5 merges the block mapping results to obtain the orthophoto.

[0178] Fig.13It is a schematic diagram of the mapping result after removing the redundant parts provided by the present invention.

[0179] S51 removes the portion of the block filling result corresponding to each block that exceeds the original corresponding block, retains only the filling result of the original block part, and obtains a single block mapping result after removing the redundant portion;

[0180] Fig.14 It is a schematic diagram of the orthophoto provided by the present invention.

[0181] Fig.15 It is a schematic diagram of orthophotos of other software showing holes in the water area provided by the present invention.

[0182] S52 combines the mapping results of all the individual blocks according to the original block division method to obtain an orthophoto.

[0183] Because other modeling software need to perform dense point cloud reconstruction, meshing and texture mapping before orthophoto generation, such as Fig.15 The water area with a single texture usually produces model holes, which eventually leads to holes in the water area of ​​the orthophoto, affecting the effect of the orthophoto. However, the present invention uses the aerial image texture filling method to generate the orthophoto, omitting the model generation step, which can effectively reduce the generation time and also Fig.14 As shown in the figure, the generation of holes is avoided.

[0184] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0185] The above is an embodiment of the device of the present invention, which can be used to execute the embodiment of the method of the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the embodiment of the method of the present invention.

[0186] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0187] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for quickly generating orthophotos based on scene aerial images and sparse point clouds, characterized in that: include: S1 uses aerial images to generate sparse point clouds and camera poses; S2 aligns the camera point cloud and the sparse point cloud and obtains a preliminary mapping; S3 divides the preliminary map into blocks and finds the original point cloud and aerial image corresponding to each block; S4 finds the most suitable aerial image for the block mapping image and performs texture mapping; S5 merges the block mapping results to obtain an orthophoto; The S1 includes: S11 prepares the aerial image to be processed; S12 uses the SfM algorithm to perform aerial triangulation to obtain the camera pose and scene sparse point cloud; The S2 includes: S21 uses the camera pose to fit the camera plane and calculate the rotation matrix; S22 uses a rotation matrix to straighten the sparse point cloud, maps the sparse point cloud to the same plane, and obtains a preliminary mapping image; The S3 includes: S31 divides the map into blocks; S311 specifies that the block division meets the following conditions: (1) Each divided block contains at least 10 sparse points; (2) The area of ​​the divided block scene is not larger than the area of ​​the scene in a single aerial photo. Under the premise of meeting condition (1), the block scene is subdivided; (3) Based on the actual aerial image size and the mapping image size, the mapping image of size p*q is divided into m*n blocks of size r*s, where m*r=p, n*s=q. If a block does not meet the requirement of 10 sparse points, the block is merged with the block with the least sparse points among the four adjacent blocks until condition (1) is met. S312: for each point in each block, find the corresponding original sparse point cloud based on the mapping relationship between the point cloud and the mapping graph, and obtain the corresponding original sparse point cloud block; S32 finds the original aerial image corresponding to the original sparse point cloud for each block; S321 finds the original corresponding aerial atlas of each sparse point and its position in each aerial image from the SfM result file, and finds multiple aerial images containing all the corresponding points in the same block; S322: for all points in the aerial image corresponding to all points in the same block, read their coordinate values, retain the maximum and minimum values ​​of their coordinate values ​​in the aerial image, intercept the aerial image within the coordinate range, and obtain an original position screenshot of the aerial image corresponding to a sparse point cloud block; The S4 includes: S41 finds the aerial photo with the smallest deflection and the most center corresponding to each block; S411 obtains the rotation angle of the camera relative to the ground during the aerial photography process according to the POS data of the aerial photography or the camera pose obtained by the SfM algorithm, and finds several aerial photography images with the smallest rotation angle; S412 Considering that the center of the aerial image has the smallest distortion and is closest to the bird's-eye view, the original position screenshot of the aerial image corresponding to the block is combined to find the aerial image whose original position screenshot is closest to the center of the aerial image; S42 finds all points in the aerial image corresponding to each sparse point in the block map; S421 finds the correspondence between the points in the sparse point cloud and the feature points in the aerial photo from the SfM result file, combines the mapping relationship between the points in the sparse point cloud and the points in the block mapping image, and finds the position of each sparse point in the mapping image of the different blocks in the corresponding feature point of the target aerial photo; S422 uses one-to-one matching point pairs to form a corresponding point set; S43 calculates a homography matrix H between the block map and the aerial image using the corresponding point set; S431 imports the point pairs of the corresponding target aerial image and the block mapping image into the calculation equation group of the homography matrix H; S432 solves the calculation equation group to obtain a homography matrix H to describe the coordinate transformation relationship between the mapping image and the aerial image; S44 uses the homography matrix H to fill the aerial image texture into the mapping image. S441 calculates each pixel Q(Q x ,Q y ) After the homography transformation using the homography matrix, the position Q of the mapping image ′ (Q ′ x ,Q ′ y ), the specific calculation is as follows: Let V1 be a three-dimensional column vector, where V1[0] = Q y , V1[1]=Q x , V1[2]=1, let V2 be the three-dimensional column vector after transformation by the homography matrix H: V2=H*V1 The position coordinates of the points after homography transformation are: <h2 style=";text-align:left;direction:ltr">Q<h2 style=";text-align:left;direction:ltr"> ′ <h2 style=";text-align:left;direction:ltr"> x <h2 style=";text-align:left;direction:ltr"> (V2[1] / V2[2]) <h2 style=";text-align:left;direction:ltr">Q<h2 style=";text-align:left;direction:ltr"> ′ <h2 style=";text-align:left;direction:ltr"> y <h2 style=";text-align:left;direction:ltr"> =V2[0] / V2[2] S442 replaces the RGB values ​​of the pixels in the transformed positions in the mapping image with the RGB values ​​of the pixels in the aerial image screenshot, and fills the original aerial image texture into the mapping image; The S5 includes: S51: removing the portion of the block filling result corresponding to each block that exceeds the original corresponding block, retaining only the filling result of the original block part, and obtaining a single block mapping result after removing the redundant portion; S52 combines all the individual block mapping results according to the original block division method to obtain an orthophoto.

2. The method for rapidly generating orthophotos according to claim 1, characterized in that: The S21 includes: S211 reads the camera position coordinates obtained by the SfM algorithm, each coordinate is a camera point in three-dimensional space, performs least squares plane fitting on all camera points, and obtains the camera plane equation, which is in the form of z=ax+by+c; S212 obtains the normal vector A(a, b, -1) of the camera plane based on the camera plane equation, performs a dot product of the normal vector of the camera plane and the unit vector B(0, 0, 1) in the positive direction of the Z axis, and calculates the rotation angle α according to the dot product result: A·B=|A|*|B|*cosα; S213 obtains a rotation axis vector C based on the cross product of the normal vector of the camera plane and the unit vector in the positive direction of the Z axis. The expression of the rotation axis vector is as follows: A×B=C Among them, C(C x ,C y ,C z ) is the rotation axis vector; S214 uses the rotation angle and the rotation axis vector to finally obtain the rotation matrix M, which is: M= Where α is the rotation angle, C(C x ,C y ,C z ) is the rotation axis vector, and M is the rotation matrix.

3. The method for quickly generating orthophotos according to claim 2, characterized in that: The S22 includes: S221 performs rotation calculation on all points of the sparse point cloud based on the rotation matrix, so as to straighten the sparse point cloud. The point P (P x ,P y ,P z ) After calculation, the point P in the straightened point cloud model is obtained ′ (P x ′ ,P y ′ ,P z ′ ), the coordinate calculation method of the point cloud model before and after alignment is as follows: P x ′ =M[0][0]*P x +M[0][1]*P y +M[0][2]*P z P y ′ =M[1][0]*P x +M[1][1]*P y +M[1][2]*P z P z ′ =M[2][0]*P x +M[2][1]*P y +M[2][2]*P z S222 obtains the z value of each point in the point cloud model based on the straightened point cloud model, and uses the z value as the height of each point; S223 determines the size of the orthophoto according to the scale of the sparse point cloud and actual application requirements; traverses the x value and y value of each point in the straightened point cloud model, determines the value range of x and y, obtains xmin, xmax, ymin, ymax, which are the minimum and maximum values ​​of x, the minimum and maximum values ​​of y, respectively, and determines the image size of the generated orthophoto in combination with the actual requirements and the number of points in the point cloud model. Where num is the number of points in the point cloud model, k is a coefficient determined according to actual needs, X and Y represent the image size, and the minimum value of X and Y, Min(X,Y), is obtained according to the above formula. The maximum value of X and Y, Max(X,Y), is calculated proportionally through the ratio of the value range of the point cloud x value and y value, and the image size of the orthophoto is obtained; S224 determines the coordinate position (i, j) of the point in the orthophoto image corresponding to the point through the position information represented by the x value and y value of the point in the point cloud model, completes mapping the sparse point cloud to the same plane, and obtains a preliminary mapping map. in, Indicates the correspondence between the unit length in the point cloud model and the pixel coordinate system; S225 counts the number of valid cells in the edge rows and columns of the preliminary mapping image. If the valid number is less than the set threshold, it means that the number of point clouds generated in this area is too small and is not suitable for orthophoto generation. Such edges are cropped and the mapping image size is modified.