Orthorectified Image Block Registration Method and Device for UAV from Whole to Part

Abstract

The present invention discloses a method and device for orthophoto block registration of unmanned aerial vehicles from overall to local, belonging to the field of remote sensing image processing. First, the present invention uses the fast Fourier transform algorithm to convert the orthophoto of the unmanned aerial vehicle from the image domain to the frequency domain, calculates the pixel frequency domain offset of the phase correlation to obtain the image displacement, and then completes the overall registration. Then, the orthophoto of the unmanned aerial vehicle is cropped, the whole image is cropped into small image blocks, and the scale-invariant feature transform algorithm based on point features is used to perform local registration on these image blocks. Finally, the Poisson fusion method is used for image block stitching, and then the entire registration process is completed. The present invention can effectively reduce the registration error caused by the distortion of two-phase images, improve the registration accuracy, and provide strong technical support for subsequent analysis applications such as change detection and geological survey.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and particularly to a method and device for orthophoto image block registration of an unmanned aerial vehicle from overall to local Background Art

[0002] With the development of unmanned aerial vehicle (UAV) aerial photogrammetry technology in recent years, UAV images have gradually become an indispensable part of the image data sources in the remote sensing field. UAV orthophotos can objectively reflect the current situation of the ground through real images and rich colors, and are widely used in aspects such as image fusion, geological survey, and change detection of illegal buildings. Image registration is the key technical support for the above aspects, and its purpose is to eliminate or weaken the image spatial structure differences between the images to be registered and the reference image caused by factors such as shooting posture, terrain undulation, and image projection. Therefore, image registration is an essential link in remote sensing applications.

[0003] Currently, the registration methods based on gray scale and features have been developed and matured. However, due to the large range and high resolution of UAV orthophotos, the ground object information is extremely complex. At the same time, there are different degrees of local distortions in two-phase images. When directly registering orthophotos using the above methods, a large number of false matches will occur, and the registration errors caused by local distortions cannot be corrected, resulting in low registration accuracy or even registration failure. Although the registration methods based on polynomials and spline functions can handle the registration errors caused by local distortions of two-phase images, the above two registration methods require a sufficient number of evenly distributed control points, which are time-consuming, laborious, and costly. Therefore, it is necessary to provide a method for orthophoto image block registration of an unmanned aerial vehicle from overall to local to solve the above technical problems. Summary of the Invention

[0004] The present invention provides a method and device for orthophoto image block registration of an unmanned aerial vehicle from overall to local, which can effectively reduce the registration errors caused by the distortions of two-phase images, improve the registration accuracy, and provide strong technical support for subsequent analysis applications such as change detection and geological survey.

[0005] The first aspect of the present invention provides a method for orthophoto image block registration of an unmanned aerial vehicle from overall to local, including the following steps:

[0006] Step 1, using the fast Fourier transform algorithm to transform the UAV orthophoto from the image domain to the frequency domain, obtaining the displacement of the UAV orthophoto by calculating the pixel frequency domain offset of phase correlation, and performing overall registration on the UAV orthophoto according to the displacement;

[0007] Step 2: Perform block processing on the orthoimage of the drone after global registration to obtain multiple orthoimage blocks of the drone, and perform local registration on each orthoimage block of the drone using the scale-invariant feature transform algorithm based on point features;

[0008] Step 3: Use the Poisson fusion algorithm to splice the multiple orthoimage blocks of the drone after local registration to obtain the finally registered orthoimage of the drone.

[0009] Optionally, in an embodiment of the present invention, the specific steps of Step 1 include:

[0010] Calculate the center point of the orthoimage of the drone, determine the central matching window, and calculate the phase-correlated pixel frequency-domain offset estimation within the central matching window using the fast Fourier transform algorithm. Let the displacement vector be (μ 0 , ν 0 ), then the displacement between the orthoimage of the drone and the reference image is:

[0011] f 2 (x, y) = f 1 (x - μ 0 , y - ν 0 )

[0012] where f 2 is the orthoimage of the drone, and f 1 is the reference image;

[0013] f 1 and f 2 The corresponding Fourier transforms are F 1 and F 2 , then the displacement between the orthoimage of the drone and the reference image in the frequency domain is:

[0014]

[0015] f 1 and f 2 The cross-power spectral density between them is:

[0016]

[0017] where F 2 * (ε, η) represents the conjugate complex number of F 2 (ε, η), perform inverse Fourier transform on the cross-power spectral density image to obtain the impulse response in the image domain. The distance between the spike at the registration point of the orthoimage of the drone and the spectral center position represents the pixel coordinate displacement vector (μ 0 , ν 0 ) on the orthoimage of the drone.

[0018] Optionally, in an embodiment of the present invention, after obtaining the pixel coordinate displacement vector, the following steps are further included:

[0019] Perform displacement detection on the pixel coordinate displacement vector, perform frequency domain offset estimation again on the registered orthoimage of the UAV, and determine whether the displacement detection condition that the pixel coordinate displacement is zero displacement is satisfied. If the displacement detection condition is not satisfied, re-perform frequency domain offset estimation and determine whether the displacement detection condition is satisfied until the maximum number of iterations is reached;

[0020] If the initial central matching window cannot meet the displacement detection condition, move the central window to the left by the length of the window size, and then determine whether the displacement detection condition is satisfied again. If it is still not satisfied, change the position of the matching window clockwise around the center of the UAV orthoimage;

[0021] After the displacement detection condition is passed, eliminate the displacement that is uniquely greater than the preset threshold according to the threshold condition. If the displacement does not meet the threshold condition, re-select the matching window.

[0022] Optionally, in an embodiment of the present invention, step 2 specifically includes:

[0023] Perform multi-scale scaling on the UAV orthoimage block using the scale space, where the scale space is obtained by performing a convolution operation on the Gaussian function and the UAV orthoimage block;

[0024] To extract feature points at each scale, after the scale space pyramid is constructed, subtract two adjacent layers at the same scale to generate the scale space of difference of Gaussian;

[0025] In the scale space of difference of Gaussian, find the local extreme points in the UAV orthoimage block. If the obtained local extreme points are the maximum or minimum values in the adjacent 3-scale neighborhoods, it is considered that the obtained local extreme points are extreme points, and then use the Taylor expansion formula to perform curve fitting on the scale space difference of Gaussian function to obtain the positions of the feature points;

[0026] Use the pixel information in the neighborhood of the feature points to describe the feature points, construct the sift descriptor, perform rough feature matching on the feature points using the KNN algorithm, and perform fine matching using the PROSAC algorithm. The PROSAC algorithm first sorts the feature points in descending order according to the similarity, and then performs iterative calculations on the point pairs with strong similarity to solve the model parameters, and then obtains the homography matrix to perform projective transformation on the image to complete the local registration of each UAV orthoimage block.

[0027] Optionally, in an embodiment of the present invention, step 3 is further included:

[0028] The Poisson fusion algorithm makes the gradient field of the two-dimensional function f to be solved as similar as possible to the reference gradient field ν in the region Ω under the Dirichlet boundary condition. The expression is:

[0029]

[0030] where f represents the orthophoto image block of the drone, and f * represents the orthophoto image of the drone to be stitched. When the values of the two are the same on the function boundary, the Dirichlet boundary condition is satisfied. In the formula represents the gradient operator. If the solution of the above formula can satisfy the following Euler-Lagrange equation, the following formula is the Poisson equation. The expression is:

[0031] Δf = div(v(x)), (x, y) ∈ Ω, f| Ω = f * | Ω

[0032] where represents the Laplace operator, represents the divergence of v = (u, v). When the orthophoto image block of the drone is fused, the guidance field of Poisson fusion is the gradient field of the area set for the orthophoto image block of the drone to be fused, and the Poisson equation is solved to achieve the fusion of the orthophoto image block of the drone.

[0033] The second aspect of the embodiments of the present invention provides a drone orthophoto image block registration device from the whole to the local, including:

[0034] The overall registration module is used to convert the drone orthophoto image from the image domain to the frequency domain by using the fast Fourier transform algorithm, obtain the displacement of the drone orthophoto image by calculating the pixel frequency domain offset of the phase correlation, and perform overall registration on the drone orthophoto image according to the displacement;

[0035] The local registration module is used to perform block processing on the overall registered drone orthophoto image to obtain multiple drone orthophoto image blocks, and perform local registration on each drone orthophoto image block by using the scale-invariant feature transform algorithm based on point features;

[0036] The fusion module is used to splice the image blocks of multiple locally registered drone orthophoto image blocks by using the Poisson fusion algorithm to obtain the finally registered drone orthophoto image.

[0037] Optionally, in an embodiment of the present invention, the overall registration module is specifically used for:

[0038] Calculate the center point of the drone orthophoto image, determine the center matching window, and calculate the phase correlation pixel frequency domain offset estimation in the center matching window by using the fast Fourier transform algorithm. Let the displacement vector be (μ0 , ν 0 ), then the displacement between the orthophoto of the UAV and the reference image is:

[0039] f 2 (x, y) = f 1 (x - μ 0 , y - ν 0 )

[0040] where f 2 is the orthophoto of the UAV, and f 1 is the reference image;

[0041] f 1 and f 2 The corresponding Fourier transforms are F 1 and F 2 , then the displacement between the orthophoto of the UAV and the reference image in the frequency domain is:

[0042]

[0043] f 1 and f 2 The cross-power spectral density between them is:

[0044]

[0045] where F 2 * (ε, η) represents the conjugate complex number of F 2 (ε, η), and the inverse Fourier transform of the cross-power spectral density image is performed to obtain the impulse response in the image domain. The distance between the peak at the registration point of the orthophoto of the UAV and the spectral center position represents the pixel coordinate displacement vector (μ 0 , ν 0 ) on the orthophoto of the UAV.

[0046] Optionally, in an embodiment of the present invention, the overall registration module is further configured to:

[0047] Perform displacement detection on the pixel coordinate displacement vector, perform frequency domain offset estimation on the registered orthophoto of the UAV again, determine whether the displacement detection condition that the pixel coordinate displacement is zero displacement is satisfied. If the displacement detection condition is not satisfied, re-perform frequency domain offset estimation and determine whether the displacement detection condition is satisfied until the maximum number of iterations;

[0048] If the initial center matching window cannot meet the displacement detection condition, move the center window to the left by the length of the window size, and then determine whether the displacement detection condition is satisfied again. If it is still not satisfied, change the position of the matching window clockwise around the center of the orthophoto of the UAV;

[0049] After the displacement detection condition is passed, eliminate the displacement that is uniquely greater than the preset threshold according to the threshold condition. If the displacement does not meet the threshold condition, reselect the matching window.

[0050] Optionally, in an embodiment of the present invention, the local registration module is specifically configured to:

[0051] Perform multi-scale scaling on the orthophoto image block of the UAV using a scale space, where the scale space is obtained by performing a convolution operation on the Gaussian function and the orthophoto image block of the UAV;

[0052] In order to extract feature points at each scale, after the scale space pyramid is constructed, subtract two adjacent layers at the same scale to generate a scale space of difference of Gaussians;

[0053] In the scale space of difference of Gaussians, find the local extreme points in the orthophoto image block of the UAV. If the obtained local extreme points are the maximum or minimum values in the adjacent 3-scale neighborhoods, it is considered that the obtained local extreme points are extreme points, and then use the Taylor expansion formula to perform curve fitting on the scale space Gaussian difference function to obtain the positions of the feature points;

[0054] Use the pixel information in the neighborhood of the feature points to describe the feature points, construct a SIFT descriptor, use the KNN algorithm to perform rough feature matching on the feature points, and use the PROSAC algorithm for fine matching. The PROSAC algorithm first sorts the feature points in descending order according to the similarity, and then performs iterative calculations on the point pairs with strong similarity to solve the model parameters, and then obtains the homography matrix to perform projective transformation on the image to complete the local registration of each orthophoto image block of the UAV.

[0055] Optionally, in an embodiment of the present invention, the fusion module is specifically configured to:

[0056] The Poisson fusion algorithm makes the gradient field of the two-dimensional function f to be solved as similar as possible to the reference gradient field ν in the region Ω under the condition of satisfying the Dirichlet boundary condition. The expression is:

[0057]

[0058] Among them, f represents the orthophoto image block of the UAV, and f * represents the orthophoto image of the UAV to be stitched. When the values of the two are the same at the function boundary, the Dirichlet boundary condition is satisfied. In the formula represents the gradient operator. If the solution of the above formula can satisfy the following Euler-Lagrange equation, the following formula is the Poisson equation. The expression is:

[0059] Δf = div(v(x)), (x, y) ∈ Ω, f| Ω = f * |Ω

[0060] Among them, represents the Laplace operator, represents the divergence of v=(u, v). When fusing the orthophoto image blocks of the UAV, the guidance field of Poisson fusion is the gradient field of the area set for the orthophoto image blocks of the UAV to be fused, and the Poisson equation is solved to achieve the fusion of the orthophoto image blocks of the UAV.

[0061] The third aspect of the embodiments of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the program to implement the method for registering orthophoto image blocks of the UAV from the whole to the local as described in the above embodiments.

[0062] The fourth aspect of the embodiments of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the program is executed by a processor, it implements the method for registering orthophoto image blocks of the UAV from the whole to the local as described above.

[0063] The fourth aspect of the embodiments of the present invention provides a computer program product, including a computer program, and the computer program is executed to implement the method for registering orthophoto image blocks of the UAV from the whole to the local as described in the above embodiments.

[0064] The method and device for registering orthophoto image blocks of the UAV from the whole to the local in the embodiments of the present invention have the following beneficial effects:

[0065] (1) In the process of registering the orthophoto images of the UAV in the present invention, there is no need to manually select control points, realizing the automation of the registration process and saving time and energy.

[0066] (2) The present invention comprehensively considers from the perspectives of the whole and the local, and uses pixel frequency domain offset estimation by fast Fourier transform and the sift algorithm based on point features for global registration and local registration respectively, effectively reducing the local registration error caused by different degrees of distortion of the orthophoto images in two periods and improving the registration accuracy.

[0067] (3) Compared with the registration results obtained by the registration methods based on polynomials and spline functions, the registration results obtained in the present invention have higher peak signal-to-noise ratio, higher structural similarity, greater mutual information, and lower mean square error, and the obtained registration results are better.

[0068] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. Description of the Drawings

[0069] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, wherein:

[0070] Figure 1 It is a flowchart of a method for orthophoto block registration of an unmanned aerial vehicle from overall to local according to an embodiment of the present invention;

[0071] Figure 2 It is a framework diagram of a method for orthophoto block registration of an unmanned aerial vehicle from overall to local according to an embodiment of the present invention;

[0072] Figure 3 It is an experimental area diagram of an embodiment of the present invention: (a) Image in the T1 period; (b) Image in the T2 period;

[0073] Figure 4 It is a local registration error diagram of an embodiment of the present invention;

[0074] Figure 5 It is a difference feature diagram of the eastern part of the image of an embodiment of the present invention:

[0075] Figure 6 It is a difference feature diagram of the southern part of the image of an embodiment of the present invention:

[0076] Figure 7 It is a difference feature diagram of the western part of the image of an embodiment of the present invention:

[0077] Figure 8 It is a difference feature diagram of the northern part of the image of an embodiment of the present invention:

[0078] Figure 9 It is a difference feature diagram of the middle part of the image of an embodiment of the present invention:

[0079] Figure 10 It is a structural schematic diagram of an orthophoto block registration device of an unmanned aerial vehicle from overall to local according to an embodiment of the present invention. Detailed implementation manners

[0080] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0081] Figure 1 It is a flowchart of a method for orthophoto block registration of an unmanned aerial vehicle from overall to local according to an embodiment of the present invention.

[0082] As Figure 1 shown, the method for orthophoto block registration of an unmanned aerial vehicle from overall to local includes the following steps:

[0083] Step 1: Use the Fast Fourier Transform (FFTW) algorithm to transform the orthophoto of the UAV from the image domain to the frequency domain. Calculate the displacement of the orthophoto of the UAV by computing the frequency-domain offset of the pixels related to the phase. Then, perform global registration on the orthophoto of the UAV according to the displacement.

[0084] In the embodiment of the present invention, Step 1 specifically includes:

[0085] Calculate the center point of the orthophoto of the UAV, determine the central matching window, and calculate the estimated frequency-domain offset of the pixels related to the phase within the central matching window using the Fast Fourier Transform algorithm. Let the displacement vector be (μ 0 , ν 0 ), then the displacement between the orthophoto of the UAV and the reference image is:

[0086] f 2 (x, y) = f 1 (x - μ 0 , y - ν 0 )

[0087] where f 2 is the orthophoto of the UAV, and f 1 is the reference image;

[0088] f 1 and f 2 corresponding Fourier transforms are F 1 and F 2 , then the displacement between the orthophoto of the UAV and the reference image in the frequency domain is:

[0089]

[0090] f 1 and f 2 the cross power spectrum (CPS) between them is:

[0091]

[0092] where F 2 * (ε, η) represents the conjugate complex number of F 2 (ε, η). Perform inverse Fourier transform on the cross power spectrum image to obtain the impulse response in the image domain. There is an obvious peak at the registration point of the image. The distance between the peak at the registration point of the orthophoto of the UAV and the spectral center position represents the pixel coordinate displacement vector (μ 0 , ν 0 ) on the orthophoto of the UAV.

[0093] Optionally, in an embodiment of the present invention, after obtaining the pixel coordinate displacement vector, the following steps are further included:

[0094] Perform displacement detection on the pixel coordinate displacement vector, perform frequency domain offset estimation on the registered orthophoto of the UAV again, and determine whether the displacement detection condition that the pixel coordinate displacement is zero displacement is satisfied. If the displacement detection condition is not satisfied, perform frequency domain offset estimation again and determine whether the displacement detection condition is satisfied until the maximum number of iterations is reached;

[0095] If the initial central matching window cannot meet the displacement detection condition, move the central window to the left by the length of the window size, and then determine whether the displacement detection condition is satisfied again. If it is still not satisfied, change the position of the matching window clockwise around the center of the UAV orthophoto;

[0096] After the displacement detection condition is passed, eliminate the displacement that is uniquely greater than the preset threshold according to the threshold condition. If the displacement does not meet the threshold condition, reselect the matching window.

[0097] Specifically, as Figure 2 shown, after obtaining the coordinate displacement vector by using the Fourier displacement theorem, the present invention performs displacement detection, and the frequency domain offset estimation on the corrected target subset image must result in zero displacement. The calculation formula is as follows:

[0098] f 2 (x,y)-f 1 (x-μ 0 ,y-ν 0 )=0

[0099] If subsequent calculations still produce non-zero X / Y shifts, the found match is considered a false positive and rejected. In this case, the offset should be recalculated, and then the evaluation of whether the shift amount is zero is repeated. This will continue until the maximum number of iterations. If the initial central matching window cannot meet the displacement detection, move the central window to the left by the length of the window size, and repeat the above steps again. If it still does not meet the displacement detection, change the position of the matching window clockwise around the image center.

[0100] After displacement detection, perform a threshold check to eliminate unrealistic large displacements, where r and c represent displacements in the x and y directions respectively, and N is the number of pixels that meet the conditions. The calculation formula is as follows:

[0101]

[0102] If the displacement does not meet the threshold condition, the matching window should be reselected as in the previous step.

[0103] Step 2: Perform block processing on the orthoimage of the UAV after global registration to obtain multiple orthoimage blocks of the UAV, and perform local registration on each orthoimage block of the UAV using the Scale-Invariant Feature Transform (SIFT) algorithm based on point features.

[0104] In an embodiment of the present invention, the GDAL library is used to perform block processing on the orthoimage of the UAV after global registration, and then the SIFT (Scale-Invariant Feature Transform) algorithm based on point features is used to perform local registration on the image blocks.

[0105] Optionally, in an embodiment of the present invention, Step 2 specifically includes:

[0106] Perform multi-scale scaling on the orthoimage block of the UAV using the scale space, where the scale space is obtained by performing a convolution operation on the Gaussian function and the orthoimage block of the UAV;

[0107] In order to extract feature points at each scale, after the scale space pyramid is constructed, the difference is taken between two adjacent layers at the same scale to generate the Difference of Gaussian (DoG) scale space;

[0108] In the DoG scale space, find the local extreme points in the orthoimage block of the UAV. If the obtained local extreme points are the maximum or minimum values in the adjacent 3-scale neighborhoods, it is considered that the obtained local extreme points are extreme points, and then the position of the feature points is obtained by curve fitting the scale space Gaussian difference function using the Taylor expansion;

[0109] Use the pixel information in the neighborhood of the feature points to describe the feature points, construct the SIFT descriptor, perform rough feature matching on the feature points using the k-Nearest Neighbor (KNN) algorithm, and perform fine matching using the Progressive Sampling with Approximate Clustering (PROSAC) algorithm. The PROSAC algorithm first sorts the feature points in descending order according to the similarity, and then performs iterative calculations on the point pairs with strong similarity to solve the model parameters, and then obtains the homography matrix to perform projective transformation on the image to complete the local registration of each orthoimage block of the UAV.

[0110] Specifically, GDAL is a typical image processing library. Using this library, the orthoimage of the UAV is cropped into image blocks with coordinates, and then the SIFT algorithm based on point features is used to perform local registration on these image blocks. In order to be able to extract the feature points in the image, this algorithm usually performs multi-scale scaling on the image using the scale space, and the scale space is obtained by performing a convolution operation on the Gaussian function and the original image. The formula is as follows:

[0111] L(x, y, σ) = G(x, y, σ) * I(x, y)

[0112] In the above formula, G(x, y, σ) represents a Gaussian function whose scale can be transformed, and its mathematical expression is:

[0113]

[0114] σ represents the scale space factor, which represents the number of times the original image to be registered is Gaussian blurred. The smaller σ is, the fewer the number of blurring times, the higher the clarity of the image, and it represents the detailed features of the image; on the contrary, it means that the original image is Gaussian blurred more times, the lower the clarity of the image, and it represents the general features of the image.

[0115] To extract the feature points at each scale, after the scale space pyramid is constructed, the difference is taken between two adjacent layers at the same scale to generate a Difference of Gauss (DoG) scale space. The calculation formula is as follows:

[0116] D(x, y, σ) = (G(x, y, kσ) - G(x, y, σ)) * I(x, y)

[0117] = L(x, y, kσ) - L(x, y, σ)

[0118] In the above formula: D(x, y, σ) is the response value image, I(x, y) is the original image, σ is the smoothing factor, and the symbol * represents the convolution operation. In the DoG space, the local extreme points in the image can be obtained, and the extreme points are potential feature points. If this point is the maximum or minimum value in the adjacent 3-scale neighborhood, then this point is considered an extreme point. Then, the position of the feature point can be obtained by using the Taylor expansion formula to fit the curve of the scale space DoG function. The calculation formula is as follows:

[0119]

[0120] In the above formula, X = (x, y, σ) T Let its derivative be zero, then the position of the extreme point can be obtained After obtaining the extreme points, the pixel information in the neighborhood of the feature points is used to describe this point, construct a sift descriptor, and use the KNN algorithm to perform rough matching of the features of the feature points. Using the rough matching of feature points can quickly eliminate invalid feature point pairs. However, there are still many false matches among the feature points after rough screening, and fine matching of features is required. The present invention uses the PROSAC algorithm for fine matching. The PROSAC algorithm first sorts the feature points in descending order according to the similarity, and then performs iterative calculations on the point pairs with strong similarity to solve the model parameters, and then obtains the homography matrix to perform projective transformation on the image to complete the local registration of the orthophoto image block of the unmanned aerial vehicle.

[0121] Step 3: Use the Poisson fusion algorithm to splice multiple orthophoto image blocks of the UAV after local registration to obtain the finally registered orthophoto image of the UAV.

[0122] Optionally, in an embodiment of the present invention, Step 3 further includes:

[0123] Under the condition of satisfying the Dirichlet boundary condition, the Poisson fusion algorithm makes the gradient field of the two-dimensional function f to be solved as similar as possible to the reference gradient field ν in the region Ω, and the expression is:

[0124]

[0125] where f represents the orthophoto image block of the UAV, and f * represents the orthophoto image of the UAV to be spliced. When their values are the same on the function boundary, the Dirichlet boundary condition is satisfied. In the formula represents the gradient operator. If the solution of the above formula can satisfy the following Euler-Lagrange equation, the following formula is the Poisson equation, and the expression is:

[0126] Δf = div(v(x)), (x, y) ∈ Ω, f| Ω = f * | Ω

[0127] where represents the Laplace operator, represents the divergence of v = (u, v). When the orthophoto image blocks of the UAV are fused, the guidance field of the Poisson fusion is the gradient field of the area where the orthophoto image blocks of the UAV to be fused are set, and solving the Poisson equation realizes the seamless fusion of the orthophoto image blocks of the UAV.

[0128] The following uses a specific embodiment to detail the method for registering orthophoto images of UAVs in a whole-then-local manner of the present invention.

[0129] The example research area is located in Tancheng Sub-district, Tancheng County, Linyi City, Shandong Province, China. The data source is the dual-period images of Huating area taken by the DJI Phantom 4Pro V2.0 UAV on February 17, 2022 and July 16, 2022. The DJI Pilot software is used to generate images in real time, and two-period orthophoto images with a spatial resolution of about 9.3 cm are obtained as Figure 3 shown. There are various types of building complexes such as commercial areas, urban villages, and high-class residential areas in the experimental area. Due to reasons such as shooting postures and terrain undulations, the orthophoto images of the two periods have different degrees of distortion in different regions, posing challenges to the registration technology. The resulting local registration errors are as Figure 4 shown, as shown by Figure 4It can be seen that different degrees of registration errors in different regions need to be corrected when registering two - period orthophotos.

[0130] The method for block - wise registration of UAV orthophotos from overall to local includes:

[0131] Step 1: Overall registration of UAV orthophotos. By calculating the center point of the entire orthophoto, determine the position and size of the central matching window. Extend a template window with a size of 256×256 or 512×512 pixels from the image center to obtain a subset image of the image. Use the Western Fast Fourier Algorithm in the matching window to estimate the pixel frequency - domain offset through phase correlation, and then calculate the overall offset.

[0132] After successfully calculating the sub - pixel X / Y displacement, displacement detection should be performed. Re - estimating the frequency - domain offset for the corrected target subset image must result in zero displacement. If subsequent calculations still produce non - zero X / Y shifts, the found match is identified as a false positive and rejected. In this case, the offset should be recalculated, and then the evaluation of whether the shift amount is zero is repeated. This will continue until the maximum number of iterations is reached. The experimental area is set to 5 iterations. If the displacement still exists, the registration of the corresponding geographical location fails, and the central matching window will be changed. After displacement detection, threshold checking is applied to eliminate unrealistic large displacements. This can be achieved by rejecting displacements exceeding a certain vector length. The experimental area is set to 15 reference image pixels. If the displacement does not meet the threshold condition, the matching window should be re - selected as in the previous step.

[0133] If the initial central matching window cannot meet the displacement detection and threshold detection, move the central window to the left by the length of the window size, and repeat the above steps. If it still does not meet the displacement detection, change the position of the matching window clockwise around the image center until the overall registration is completed.

[0134] Step 2: Image block division using the GDAL library and local registration using the SIFT algorithm based on point features. Use the GDAL library to divide the two - period UAV orthophotos into blocks. Each image will generate 525 image blocks with a size of 1024×1024 and an overlap of 50%. Use the SIFT algorithm based on point features to perform local registration on the 525 image blocks in a cyclic manner. Use the KNN algorithm to perform rough feature matching on the feature points, set the KNN Euclidean metric to 0.75 times the original distance, use the PROSAC algorithm for fine matching, set the matching threshold to 5 reference image pixels. First, sort the feature points in descending order according to this similarity, and then perform iterative calculations on the point pairs with strong similarity to solve the model parameters, and then obtain the homography matrix to perform projective transformation on the image to complete the local registration of the UAV orthophoto blocks.

[0135] Step 3: Use the Poisson fusion algorithm to stitch the 525 UAV image patches after local registration into a whole orthophoto in a coordinate loop manner to complete the final registration work.

[0136] Use the difference feature map to reflect the registration results of the two-period images. The difference feature maps of different regions of the images obtained by the unregistered, polynomial registration method, spline registration method, and the block registration method of first global and then local are respectively as Figures 5 to 9 shown. In the difference feature map, the dark part represents the registration error. Since the resolution of the UAV orthophoto is high, the illumination, shadow, seasonal changes of trees, and changes in car parking will also be reflected in the difference feature map. Therefore, only focus on the edge changes of buildings to visually analyze the registration effect. From Figures 5 to 9 it can be seen that there are large registration errors between the two-period images when unregistered, and the high-brightness noise is significant. Compared with the polynomial registration method and the spline registration method, the block registration method of first global and then local proposed by the present invention can better reduce the different degrees of registration errors in different regions. At the same time, the present invention does not require a large number of uniformly distributed control points to be manually selected, saving time and effort. To quantitatively evaluate the accuracy of the three algorithms, four indexes for evaluating image similarity, namely root mean square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and mutual information (MI), are used to evaluate the registration results of the two images, as shown in Table 1. Compared with the polynomial registration method and the spline registration method, the present invention has a higher peak signal-to-noise ratio, higher structural similarity, greater mutual information, and lower mean square error, and obtains better registration results, and can be effectively applied to the registration of UAV orthophotos.

[0137] Table 1

[0138] Method RMSE PSNR SSIM MI Unregistered 0.013260498 85.67960664 0.999996315 0.488367122 Polynomial registration 0.012526520 86.17419476 0.999996935 0.488482308 Spline registration 0.011021658 87.28586523 0.999997451 0.488795579 Block registration 0.010605595 87.62010278 0.999997595 0.488882689

[0139] The block registration method of UAV orthophotos of first global and then local proposed according to the embodiment of the present invention uses the fast Fourier transform algorithm to convert the UAV orthophoto from the image domain to the frequency domain, calculates the phase-related pixel frequency domain offset to estimate the image displacement, and then completes the global registration. On this basis, the UAV orthophoto is cropped, the whole image is cropped into small image patches, and then the sift algorithm based on point features is used to perform local registration on these image patches. Finally, the Poisson fusion method is used for image patch stitching, and then the entire registration process is completed. The present invention can effectively reduce the registration error caused by the distortion of the two-period images, improve the registration accuracy, and provide a strong technical support for subsequent analysis applications such as change detection and geological survey.

[0140] Next, describe the block registration device of UAV orthophotos of first global and then local proposed according to the embodiment of the present invention with reference to the accompanying drawings.

[0141] Figure 10 Schematic structural diagram of an orthophoto image block registration device for drones from overall to local according to an embodiment of the present invention.

[0142] As Figure 10 shown, the orthophoto image block registration device 10 for drones from overall to local includes: an overall registration module 100, a local registration module 200, and a fusion module 300.

[0143] Among them, the overall registration module 100 is used to convert the orthophoto image of the drone from the image domain to the frequency domain by using the fast Fourier transform algorithm, obtain the displacement of the orthophoto image of the drone by calculating the pixel frequency domain offset of phase correlation, and perform overall registration on the orthophoto image of the drone according to the displacement. The local registration module 200 is used to perform block processing on the orthophoto image of the drone after overall registration to obtain a plurality of orthophoto image blocks of the drone, and perform local registration on each orthophoto image block by using the scale-invariant feature transform algorithm based on point features. The fusion module 300 is used to perform image block stitching on the plurality of orthophoto image blocks after local registration by using the Poisson fusion algorithm to obtain the finally registered orthophoto image of the drone.

[0144] Optionally, in an embodiment of the present invention, the overall registration module 100 is specifically used for:

[0145] Calculate the center point of the orthophoto image of the drone, determine the center matching window, calculate the phase correlation pixel frequency domain offset estimation within the center matching window by using the fast Fourier transform algorithm, and set the displacement vector as (μ 0 , ν 0 ), then the displacement between the orthophoto image of the drone and the reference image is:

[0146] f 2 (x, y) = f 1 (x - μ 0 , y - ν 0 )

[0147] Among them, f 2 is the orthophoto image of the drone, and f 1 is the reference image;

[0148] f 1 and f 2 The corresponding Fourier transforms are F 1 and F 2 , then the displacement between the orthophoto image of the drone and the reference image in the frequency domain is:

[0149]

[0150] f 1 and f 2 The cross-power spectral density between them is:

[0151]

[0152] Among them, F 2 * (ε, η) represents the conjugate complex number of F 2 (ε, η). The inverse Fourier transform is performed on the cross-power spectral density image to obtain the impulse response in the image domain. The distance between the spike at the registration point of the UAV orthoimage and the spectral center position represents the pixel coordinate displacement vector (μ 0 , ν 0 ) on the UAV orthoimage.

[0153] Optionally, in an embodiment of the present invention, the overall registration module 100 is further configured to:

[0154] Perform displacement detection on the pixel coordinate displacement vector, perform frequency domain offset estimation on the registered UAV orthoimage again, determine whether the displacement detection condition of zero displacement of the pixel coordinate is satisfied. If the displacement detection condition is not satisfied, re-perform frequency domain offset estimation and determine whether the displacement detection condition is satisfied until the maximum number of iterations;

[0155] If the initial center matching window cannot meet the displacement detection condition, move the center window to the left by the length of the window size, and then determine whether the displacement detection condition is satisfied again. If it is still not satisfied, change the position of the matching window clockwise around the center of the UAV orthoimage;

[0156] After the displacement detection condition is passed, eliminate the displacement that is uniquely greater than the preset threshold according to the threshold condition. If the displacement does not meet the threshold condition, reselect the matching window.

[0157] Optionally, in an embodiment of the present invention, the local registration module 200 is specifically configured to:

[0158] Perform multi-scale scaling on the UAV orthoimage block using the scale space, where the scale space is obtained by performing a convolution operation on the Gaussian function and the UAV orthoimage block;

[0159] In order to extract the feature points at each scale, after the scale space pyramid is constructed, the difference is taken between two adjacent layers at the same scale to generate the scale space of difference of Gaussians;

[0160] In the scale space of difference of Gaussians, find the local extreme points in the UAV orthoimage block. If the obtained local extreme points are the maximum or minimum values in the adjacent 3-scale neighborhoods, it is considered that the obtained local extreme points are extreme points, and then the position of the feature points is obtained by performing curve fitting on the scale space difference of Gaussians function using the Taylor expansion;

[0161] Use the pixel information in the neighborhood of feature points to describe the feature points, construct the SIFT descriptor, perform rough feature matching on the feature points using the KNN algorithm, and perform fine matching using the PROSAC algorithm. The PROSAC algorithm first sorts the feature points in descending order according to similarity, and then iteratively calculates the point pairs with strong similarity to solve the model parameters, and then obtains the homography matrix to perform projective transformation on the image, completing the local registration of each orthophoto block of the UAV.

[0162] Optionally, in an embodiment of the present invention, the fusion module 300 is specifically configured to:

[0163] Under the condition of satisfying the Dirichlet boundary condition, the Poisson fusion algorithm makes the gradient field of the two-dimensional function f to be solved in the region Ω as similar as possible to the reference gradient field ν, and the expression is:

[0164]

[0165] Among them, f represents the orthophoto block of the UAV, and f * represents the orthophoto of the UAV to be spliced. When their values are the same on the function boundary, the Dirichlet boundary condition is satisfied. In the formula represents the gradient operator. If the solution of the above formula can satisfy the following Euler-Lagrange equation, the following formula is the Poisson equation, and the expression is:

[0166] Δf = div(v(x)), (x, y) ∈ Ω, f| Ω = f * | Ω

[0167] Among them, represents the Laplace operator, represents the divergence of v = (u, v). When the orthophoto blocks of the UAV are fused, the guiding field of the Poisson fusion is the gradient field of the area set for the orthophoto blocks of the UAV to be fused, and the Poisson equation is solved to achieve the fusion of the orthophoto blocks of the UAV.

[0168] It should be noted that the foregoing explanation of the embodiment of the orthophoto block registration method for UAVs from the whole to the local also applies to the orthophoto block registration device for UAVs from the whole to the local in this embodiment, and will not be elaborated here.

[0169] The orthophoto image block registration device for UAVs that first performs global and then local registration according to the embodiments of the present invention uses the fast Fourier transform algorithm to convert the UAV orthophoto image from the image domain to the frequency domain, calculates the pixel frequency domain offset of the phase correlation to obtain the image displacement, and thus completes the global registration. On this basis, the UAV orthophoto image is cropped, the entire image is cropped into small image blocks, and then the local registration of these image blocks is performed using the SIFT algorithm based on point features. Finally, the Poisson fusion method is used for image block stitching, and thus the entire registration process is completed. The present invention can effectively reduce the registration error caused by the distortion of two-phase images, improve the registration accuracy, and provide strong technical support for subsequent analysis applications such as change detection and geological survey.

[0170] Furthermore, an electronic device according to an embodiment of the present invention includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the program to implement the method for orthophoto image block registration of UAVs that first performs global and then local registration as described in the above embodiment.

[0171] Furthermore, an embodiment of the present invention provides a computer-readable storage medium that stores a computer program, and when the program is executed by a processor, it implements the method for orthophoto image block registration of UAVs that first performs global and then local registration as described above.

[0172] In addition, an embodiment of the present invention also provides a computer program product, including a computer program, and the computer program is executed to implement the method for orthophoto image block registration of UAVs that first performs global and then local registration as described in the above embodiment.

[0173] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0174] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0175] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment, or portion of code including one or N executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations where functions may be executed not in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

Claims

1. A method for block registration of UAV orthophotos, first overall and then local, characterized in that: The following steps are involved: Step 1, using a fast Fourier transform algorithm to convert the drone orthophoto from the image domain to the frequency domain, and obtaining the displacement of the drone orthophoto by calculating the phase-related pixel frequency domain offset, and performing overall registration of the drone orthophoto according to the displacement; Step 2, the UAV orthophoto image after overall registration is processed into blocks to obtain multiple UAV orthophoto blocks, and each UAV orthophoto block is locally registered using a scale-invariant feature conversion algorithm based on point features; The step 2 specifically includes: The UAV orthophoto block is multi-scaled using a scale space, where the scale space is obtained by convolution operation of a Gaussian function and the UAV orthophoto block; In order to extract feature points at each scale, after the scale space pyramid is constructed, two adjacent layers at the same scale are subtracted to generate a Gaussian difference scale space; In the Gaussian difference scale space, the local extreme point in the drone orthophoto block is obtained. If the local extreme point is the maximum or minimum value in the three adjacent scale neighborhoods, it is considered to be the extreme point. Then, the Gaussian difference function in the scale space is used to perform curve fitting using the Taylor expansion to obtain the position of the feature point. The feature points are described using the pixel information of the feature point neighborhood, and the SIFT descriptor is constructed. The KNN algorithm is used to perform rough feature matching on the feature points, and the PROSAC algorithm is used for precise matching. The PROSAC algorithm first sorts the feature points in descending order according to their similarity, and then iteratively calculates the point pairs with strong similarity to solve the model parameters, and then obtains the homography matrix to perform projection transformation on the image, completing the local registration of each drone orthophoto block; Step 3: Use the Poisson fusion algorithm to stitch the multiple UAV orthophoto blocks after local registration to obtain the final registered UAV orthophoto; The step 3 further comprises: The Poisson fusion algorithm makes the gradient field of the two-dimensional function f to be solved in the region Ω as similar as possible to the reference gradient field ν under the Dirichlet boundary condition. The expression is: Among them, f represents the drone orthophoto block, f * represents the drone orthophoto to be stitched. When the values ​​of the two are the same on the function boundary, the Dirichlet boundary condition is satisfied. represents the gradient operator. If the solution of the above equation can satisfy the following Euler-Lagrange equation, then the following equation is the Poisson equation, expressed as: Δf=div(v(x)),(x,y)∈Ω,f Ω =f * Ω in, represents the Laplace operator, It represents the divergence of v = (u, v). When the UAV orthophoto blocks are fused, the guidance field of Poisson fusion is the gradient field of the set area of ​​the UAV orthophoto blocks to be fused. The Poisson equation is solved to realize the fusion of UAV orthophoto blocks.

2. The method according to claim 1, characterized in that The step 1 specifically includes: Calculate the center point of the UAV orthophoto, determine the center matching window, and use the fast Fourier transform algorithm to calculate the phase-related pixel frequency domain offset estimation in the center matching window. Assume that the displacement vector is (μ0, ν0), then the displacement between the UAV orthophoto and the reference image is: f2(x,y)=f1(x-μ0,y-ν0) Among them, f2 is the UAV orthophoto, and f1 is the reference image; The Fourier transforms corresponding to f1 and f2 are F1 and F2, and the displacement between the UAV orthophoto and the reference image in the frequency domain is: The mutual power spectrum between f1 and f2 is: Among them, F2 * (ε,η) represents the conjugate complex number of F2(ε,η), and the inverse Fourier transform of the cross-power spectrum image is performed to obtain the impulse response in the image domain. The distance between the peak at the registration point of the UAV orthophoto and the center position of the spectrum represents the pixel coordinate displacement vector (μ0,ν0) on the UAV orthophoto.

3. The method according to claim 2, characterized in that After obtaining the pixel coordinate displacement vector, the method further includes: Perform displacement detection on the pixel coordinate displacement vector, perform frequency domain offset estimation again on the registered drone orthophoto, and determine whether the displacement detection condition that the pixel coordinate displacement is zero displacement is met. If the displacement detection condition is not met, re-perform frequency domain offset estimation and determine whether the displacement detection condition is met until the maximum number of iterations is reached; If the initial center matching window cannot meet the displacement detection condition, the center window is moved to the left by the length of the window size, and it is determined again whether the displacement detection condition is met. If it is still not met, the matching window position is changed clockwise around the center of the drone orthophoto; After the displacement detection condition is passed, the only displacement greater than the preset threshold is eliminated according to the threshold condition. If the displacement does not meet the threshold condition, the matching window is reselected.

4. A device for registering orthophotos of drones in blocks by first overall and then local, characterized in that: include: An overall registration module is used to convert the drone orthophoto from the image domain to the frequency domain using a fast Fourier transform algorithm, obtain the displacement of the drone orthophoto by calculating the phase-related pixel frequency domain offset, and perform overall registration on the drone orthophoto according to the displacement; The local registration module is used to process the drone orthophoto after overall registration in blocks to obtain multiple drone orthophoto blocks, and to perform local registration on each drone orthophoto block using a scale-invariant feature conversion algorithm based on point features. The steps of performing local registration specifically include: The UAV orthophoto block is multi-scaled using a scale space, where the scale space is obtained by convolution operation of a Gaussian function and the UAV orthophoto block; In order to extract feature points at each scale, after the scale space pyramid is constructed, two adjacent layers at the same scale are subtracted to generate a Gaussian difference scale space; In the Gaussian difference scale space, the local extreme point in the drone orthophoto block is obtained. If the local extreme point is the maximum or minimum value in the three adjacent scale neighborhoods, it is considered to be the extreme point. Then, the Gaussian difference function in the scale space is used to perform curve fitting using the Taylor expansion to obtain the position of the feature point. The feature points are described using the pixel information of the feature point neighborhood, and the SIFT descriptor is constructed. The KNN algorithm is used to perform rough feature matching on the feature points, and the PROSAC algorithm is used for precise matching. The PROSAC algorithm first sorts the feature points in descending order according to their similarity, and then iteratively calculates the point pairs with strong similarity to solve the model parameters, and then obtains the homography matrix to perform projection transformation on the image, completing the local registration of each drone orthophoto block; The fusion module is used to stitch the multiple UAV orthophoto blocks after local registration using the Poisson fusion algorithm to obtain the final registered UAV orthophoto. The stitching of the image blocks using the Poisson fusion algorithm also includes: The Poisson fusion algorithm makes the gradient field of the two-dimensional function f to be solved in the region Ω as similar as possible to the reference gradient field ν under the Dirichlet boundary condition. The expression is: Among them, f represents the drone orthophoto block, f * represents the drone orthophoto to be stitched. When the values ​​of the two are the same on the function boundary, the Dirichlet boundary condition is satisfied. represents the gradient operator. If the solution of the above equation can satisfy the following Euler-Lagrange equation, then the following equation is the Poisson equation, expressed as: Δf=div(v(x)),(x,y)∈Ω,f Ω =f * Ω in, represents the Laplace operator, It represents the divergence of v = (u, v). When the UAV orthophoto blocks are fused, the guidance field of Poisson fusion is the gradient field of the set area of ​​the UAV orthophoto blocks to be fused. The Poisson equation is solved to realize the fusion of UAV orthophoto blocks.

5. The device according to claim 4, characterized in that The overall registration module is specifically used for: Calculate the center point of the UAV orthophoto, determine the center matching window, and use the fast Fourier transform algorithm to calculate the phase-related pixel frequency domain offset estimation in the center matching window. Assume that the displacement vector is (μ0, ν0), then the displacement between the UAV orthophoto and the reference image is: f2(x,y)=f1(x-μ0,y-ν0) Among them, f2 is the UAV orthophoto, and f1 is the reference image; The Fourier transforms corresponding to f1 and f2 are F1 and F2, and the displacement between the UAV orthophoto and the reference image in the frequency domain is: The mutual power spectrum between f1 and f2 is: Among them, F2 * (ε,η) represents the conjugate complex number of F2(ε,η), and the inverse Fourier transform of the cross-power spectrum image is performed to obtain the impulse response in the image domain. The distance between the peak at the registration point of the UAV orthophoto and the center position of the spectrum represents the pixel coordinate displacement vector (μ0,ν0) on the UAV orthophoto.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the overall-first-local-later drone orthophoto block registration method as described in any one of claims 1 to 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the overall-first-local-later drone orthophoto block registration method as described in any one of claims 1 to 3.

8. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the overall-first-local-later drone orthophoto block registration method as described in any one of claims 1-3.

Images