An unmanned aerial vehicle aerial image registration method, system, device and storage medium based on unsupervised learning

Abstract

The application relates to a kind of unmanned aerial vehicle aerial image registration method, system, equipment and storage medium based on unsupervised learning, the method comprises the following steps: establishing data set: obtaining continuous frame image of unmanned aerial vehicle aerial photography, the adjacent two frames of continuous frame image are formed into a group of image pairs;Optimization registration algorithm network: the image pair is input into registration algorithm network to optimize, and the numerical value of ternary loss function is used to determine that registration algorithm network optimization is completed;Complete image registration: the image pair to be registered is input into the registration algorithm network of optimization completion to obtain the registered image.Compared with the prior art, the application has the advantages that the similarity between images can be accurately described, and accurate image registration can be performed.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a method, system, device and storage medium for unmanned aerial vehicle (UAV) aerial image registration based on unsupervised learning. Background Art

[0002] With the rapid development of modern sensor technology, portable and miniaturized sensors have gradually emerged. Aerial photography using sensors in conjunction with drones is gaining increasing popularity. Image registration, as a key technology, is widely used in the field of drone aerial imagery. Image registration is the process of finding a transformation between two images to align the source and target images. Specifically, it maps the images to the same coordinate system by finding spatial correspondences between them. Drone aerial image registration has applications in various fields, including change detection, drone positioning, and image stitching.

[0003] In recent years, neural networks have been widely used in image registration tasks. Chinese patent publication number CN115082530A discloses a 3D image registration method, system, device, and storage medium based on unsupervised learning. This method addresses the technical issues of dense and fuzzy optical flow in unlabeled images, making manual annotation impossible, and automatically generated datasets often deviating from practical needs. However, the patent does not address or address issues related to image registration, such as the small variations and differences in illumination between consecutive frames.

[0004] In general, existing technical solutions do not effectively address two specific issues in drone aerial image registration. First, the varying illumination between consecutive frames of drone aerial imagery creates significant challenges for the registration network in describing the similarities between images. This contrast variation makes it difficult for the registration network to describe the similarities between images. Second, for some consecutive frames of drone imagery, the differences between the images are relatively small, making it difficult for existing unsupervised learning algorithms to handle these variations. Therefore, achieving more precise registration without falling into trivial solutions is a major challenge for unsupervised learning, making precise image registration difficult. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a UAV aerial image registration method, system, device and storage medium based on unsupervised learning. By introducing a sparse domain consistency network into the unsupervised learning architecture, the present invention can register UAV aerial image pairs without labeled training samples, and avoid the non-convergence problem caused by changes in illumination contrast of consecutive frames of the UAV and the trivial solution problem caused by too small differences between consecutive frames.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] The present invention discloses a UAV aerial image registration method based on unsupervised learning, comprising the following steps:

[0008] Establish a data set: obtain continuous frame images taken by drones and establish a data set. The two adjacent frames of the continuous frame images are combined into a group of image pairs. Each image pair includes a source image S and a target image T.

[0009] Optimize the registration algorithm network: the registration algorithm network includes a feature extraction network, a consistency network, a regression network and a spatial transformation network; the image is input into the feature extraction network to extract features to obtain a feature map; the feature map generates a first correlation matrix C; the first correlation matrix C is input into the consistency network to obtain a second correlation matrix C′; the second correlation matrix is ​​input into the regression network to obtain a two-dimensional affine matrix θ ts ; The source image S and the two-dimensional affine matrix θ ts Input into the spatial transformation network to obtain the geometrically transformed image S′;

[0010] The geometrically transformed image S′, the source image S, and the target image T are input into the similarity network to optimize the ternary loss function. If the value of the ternary loss function stops decreasing, the iteration is stopped and the registration algorithm network optimization is completed.

[0011] Complete image registration: Input the image pair to be registered into the optimized registration algorithm network to obtain the registered image.

[0012] Furthermore, the specific steps of generating the first correlation matrix C from the feature graph include:

[0013] Input the source image S and target image T into the feature extraction network for feature extraction to obtain the source feature map f s and the target feature map f t ;

[0014] For source feature map f s Each point f s (i, j), in the target feature map f t Take out the n points with the highest cosine similarity with point (i, j) and record them as f t (k, l);

[0015] According to f s (i, j) and f t (k, l) Calculate the sparsity correlation matrix

[0016] The source feature map f s and target feature map f t The order of taking points is exchanged, and the target feature map ft Every point f t (i, j), in the source feature map f s Take out the n points with the highest cosine similarity of point (i, j) and record them as f S (k, l), according to f t (i, j) and f S (k, l) calculate the sparsity correlation matrix C T-＞S ;

[0017] Add the two obtained sparse correlation matrices to obtain the first correlation matrix C;

[0018] C=C Sr =C S-＞T +C T-＞S

[0019] Furthermore, the second correlation matrix C' is obtained by implementing domain consistency through a four-dimensional convolutional network.

[0020] Furthermore, the four-dimensional convolutional network includes three four-dimensional sparse convolutional layers, wherein:

[0021] The first four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and has 10 output channels;

[0022] The second four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and has 10 output channels;

[0023] The third four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and an output channel of 1.

[0024] Furthermore, the regression network includes a first two-dimensional convolution layer connected in sequence, using an 11*11 convolution kernel and an output channel of 225; a second two-dimensional convolution layer using a 7*7 convolution kernel and an output channel of 128; a third two-dimensional convolution layer using a 5*5 convolution kernel and an output channel of 64; and a fourth activation function layer activated by a relu function; wherein, each two-dimensional convolution layer is connected to a regularization layer.

[0025] Furthermore, the two-dimensional affine matrix θ ts Contains 6 set transformation parameters a1, a2, a3, a4, a5, a6, expressed as:

[0026]

[0027] Furthermore, the optimized ternary loss function is realized by calculating the parameters of the ternary loss function through a similarity network, wherein the similarity network includes an advanced feature description network HardNet and a similarity NCC function, and the two-dimensional affine matrix θ tsThe feature descriptor des(S') and similarity coefficient NCC are calculated in turn through the advanced feature description network HardNet and the similarity NCC function.

[0028] On the other hand, the present invention proposes a UAV aerial image registration system based on unsupervised learning, the system comprising:

[0029] The dataset module is used to obtain continuous frame images taken by drones and establish a dataset. Two adjacent frames of continuous frame images are combined into a group of image pairs. Each image pair includes a source image S and a target image T.

[0030] Registration algorithm network module: used to optimize the registration algorithm network, including: inputting the image into the feature extraction network to extract features to obtain a feature map; generating a first correlation matrix C from the feature map; inputting the first correlation matrix C into the consistency network to obtain a second correlation matrix C'; inputting the second correlation matrix into the regression network to obtain a two-dimensional affine matrix θ ts ; The source image S and the two-dimensional affine matrix θ ts Input into the spatial transformation network to obtain the geometrically transformed image S';

[0031] The geometrically transformed image S', the source image S and the target image T are input into the similarity network to optimize the ternary loss function. If the value of the ternary loss function stops decreasing, the iteration is stopped and the registration algorithm network optimization is completed.

[0032] Image registration module: used to input the image pair to be registered into the optimized registration algorithm network to obtain the registered image.

[0033] On the other hand, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.

[0034] On the other hand, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) The present invention proposes to establish image pairs using adjacent images. By relatively processing the images and introducing a sparse domain consistency network into the unsupervised learning architecture, the non-convergence problem caused by changes in illumination contrast in consecutive frames of drones is avoided. The present invention can achieve the description of similar images.

[0037] 2) The present invention inputs the geometrically changed image S', the source image S, and the target image T into the similarity network, and integrates the advanced feature description network HardNet into the similarity metric NCC. For consecutive frames with illumination contrast, it has better description similarity and solves the non-convergence problem caused by the change of illumination contrast in consecutive frames of the image.

[0038] 3) This paper designs a new ternary loss function and introduces a dynamic margin to avoid the failure of the loss function, thus avoiding the problem of the training network falling into the trivial solution caused by the small difference between consecutive frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the method flow of the present invention;

[0040] Figure 2 is a schematic diagram of the source image and target image in the data set of the present invention;

[0041] Figure 3 1 is a flow chart of an optimized registration network method according to an embodiment of the present invention;

[0042] Figure 4 2 is a schematic diagram of the overall framework of the unsupervised learning registration method according to an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of a similarity network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0045] On the one hand, the present invention discloses a method for registration of UAV aerial images based on unsupervised learning, referring to Figure 1 , including the following steps:

[0046] Establish a data set: obtain continuous frame images taken by drones and establish a data set. The two adjacent frames of the continuous frame images are combined into a group of image pairs. Figure 2 , each image pair includes a source image S and a target image T;

[0047] Optimize the registration algorithm network: the registration algorithm network includes a feature extraction network, a consistency network, a regression network and a spatial transformation network; the image is input into the feature extraction network to extract features to obtain a feature map; the feature map generates a first correlation matrix C; the first correlation matrix C is input into the consistency network to obtain a second correlation matrix C; the second correlation matrix is ​​input into the regression network to obtain a two-dimensional affine matrix θ ts ; The source image S and the two-dimensional affine matrix θ ts Input into the spatial transformation network to obtain the geometrically transformed image S′;

[0048] The geometrically transformed image S′, the source image S, and the target image T are input into the similarity network to optimize the ternary loss function. If the value of the ternary loss function stops decreasing, the iteration is stopped and the registration algorithm network optimization is completed.

[0049] Complete image registration: Input the image pair to be registered into the optimized registration algorithm network to obtain the registered image.

[0050] For a more detailed description of the above steps, refer to Figure 3 , including the following specific steps:

[0051] S1. Obtain continuous frame images from drone aerial photography and establish a dataset. Group two adjacent frames of the continuous frame images into a set of image pairs. Each pair contains a source image S and a target image T. The training set for this example contains 8,000 400-pixel * 400-pixel image pairs.

[0052] S2: Input the source image S and the target image T into the feature extraction network. In this embodiment, the backbone network ResNet-101 is used for feature extraction to obtain the source feature map f s and the target feature map f t .

[0053] S3, according to the source feature map f s and the target feature map f t , for each point f in the source feature map s (i, j), in the target feature map f t Take out the n points with the highest cosine similarity with point (i, j) and record them as f t (k, l), n can be selected as 10;

[0054] According to f s (i, j) and f t (k, l) Calculate the sparsity correlation matrix The formula is as follows:

[0055]

[0056] The source feature map f s and target feature map f t The order of taking points is exchanged, and the target feature map f t Every point f t (i, j), in the source feature map f s Take out the n points with the highest cosine similarity of point (i, j) and record them as f S (k, l), according to f t (i, j) and f S (k, L) calculates the sparsity correlation matrix C T-＞S ;

[0057] Add the two obtained sparse correlation matrices to obtain the first correlation matrix C;

[0058] C=C ST =C S-＞T +C T-＞S .

[0059] S4. Input the first correlation matrix C into the consistency network to achieve domain consistency and obtain the second correlation matrix C', specifically through a four-dimensional convolutional network including three four-dimensional sparse convolutional layers, where:

[0060] The first four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and has 10 output channels;

[0061] The second four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and has 10 output channels;

[0062] The third four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and an output channel of 1.

[0063] S5. Input the second correlation matrix C' into the regression network. The regression network includes the first two-dimensional convolution layer connected in sequence, using an 11*11 convolution kernel and an output channel of 225; the second two-dimensional convolution layer uses a 7*7 convolution kernel and an output channel of 128; the third two-dimensional convolution layer uses a 5*5 convolution kernel and an output channel of 64; the fourth activation function layer is activated by the relu function; each two-dimensional convolution layer is connected to a regularization layer, and the output is a two-dimensional affine matrix θ ts ; Two-dimensional affine matrix θ ts Contains 6 set transformation parameters a1, a2, a3, a4, a5, a6, expressed as:

[0064]

[0065] S6. Reference Figure 4 , the two-dimensional affine matrix θ tsThe source image S is input into the spatial transformer network STN (Spatial transformer network), and the optimized ternary loss function is realized by calculating the parameters of the ternary loss function through the similarity network. The similarity network includes the advanced feature description network HardNet and the similarity NCC function, which converts the two-dimensional affine matrix θ ts The feature descriptor des(S′) and the similarity coefficient NCC are calculated in sequence through the advanced feature description network HardNet and the similarity NCC function, and the geometrically changed image S′ is obtained;

[0066] S7, Reference Figure 5 , input the geometrically transformed image S′, the target image T and the source image S into the similarity network, and calculate the parameters of the ternary loss function:

[0067] The geometrically transformed image S′ is input into the high-level feature description network HardNet to obtain the feature descriptor des(S′) of this image;

[0068] Input the target image T into the high-level feature description network HardNet to obtain the feature descriptor des(T) of this image;

[0069] Input the source image S into the high-level feature description network HardNet to obtain the feature descriptor des(S) of this image;

[0070] Sim(S′, T) is calculated based on des(S′), des(T) and the specific similarity metric NCC value. The formula is as follows:

[0071]

[0072] Sim(S′,T)=NCC(des(S′),des(T))

[0073] Sim(S, T) is calculated based on des(S), des(T) and the specific similarity metric NCC value. The formula is as follows:

[0074]

[0075] Sim(S,T)=NCC(des(S),des(T))

[0076] S8. Based on the above calculation, the ternary loss function is continuously optimized by iteratively performing image input in an end-to-end manner. The ternary loss function is specifically:

[0077] L=max(Sim(S′,T)-Sim(S,T)+margin(δ),0)

[0078] margin(δ)=ψ1+δ*β

[0079] Wherein, δ refers to the number of times the data set is trained, ψ1 is the initial value of margin, and β represents the growth rate. In this embodiment, the initial value of ψ1 is 0.1, and the initial value of β is 0.01.

[0080] S9, repeat the above steps S2 to S8, and stop the iteration when the ternary loss function stops decreasing in the data set, and obtain the optimized ternary loss function and the final two-dimensional affine matrix θ ts , get the optimized registration algorithm network;

[0081] S10: Input the unregistered image into the registration algorithm network, output the final registered image pair, and complete the image registration.

[0082] On the other hand, the present invention also discloses a UAV aerial image registration system based on unsupervised learning, the system comprising:

[0083] The dataset module is used to obtain continuous frame images taken by drones and establish a dataset. Two adjacent frames of continuous frame images are combined into a group of image pairs. Each image pair includes a source image S and a target image T.

[0084] Registration algorithm network module: used to optimize the registration algorithm network, including: inputting the image into the feature extraction network to extract features to obtain a feature map; generating a first correlation matrix C from the feature map; inputting the first correlation matrix C into the consistency network to obtain a second correlation matrix C′; inputting the second correlation matrix into the regression network to obtain a two-dimensional affine matrix θ ts ; The source image S and the two-dimensional affine matrix θ ts Input into the spatial transformation network to obtain the geometrically transformed image S′;

[0085] The geometrically transformed image S', the source image S and the target image T are input into the similarity network to optimize the ternary loss function. If the value of the ternary loss function stops decreasing, the iteration is stopped and the registration algorithm network optimization is completed.

[0086] Image registration module: used to input the image pair to be registered into the optimized registration algorithm network to obtain the registered image.

[0087] In a third aspect, the present invention also discloses a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for unmanned aerial vehicle (UAV) aerial image registration based on unsupervised learning is implemented.

[0088] In a fourth aspect, the present invention further discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements a drone aerial image registration method based on unsupervised learning.

[0089] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A UAV aerial image registration method based on unsupervised learning, characterized in that: The following steps are involved: Establish a data set: obtain continuous frame images taken by drones and establish a data set. The two adjacent frames of the continuous frame images are combined into a group of image pairs. Each image pair includes a source image S and a target image T. Optimizing the registration algorithm network: the registration algorithm network includes a feature extraction network, a consistency network, a regression network, and a spatial transformation network; inputting the image pair into the feature extraction network to extract features to obtain a feature map; generating a first correlation matrix C based on the feature map; inputting the first correlation matrix C into the consistency network to obtain a second correlation matrix C'; Input the second correlation matrix C' into the regression network to obtain the two-dimensional affine matrix θ ts ; The source image S and the two-dimensional affine matrix θ ts Input into the spatial transformation network to obtain the geometrically transformed image S'; The geometrically changed image S', the source image S and the target image T are input into the similarity network to optimize the ternary loss function. If the value of the ternary loss function stops decreasing, the iteration is stopped and the registration algorithm network optimization is completed. The optimized ternary loss function is realized by calculating the parameters of the ternary loss function through the similarity network. The similarity network includes the advanced feature description network HardNet and the similarity NCC function. The two-dimensional affine matrix θ ts The feature descriptor des(S') and the similarity coefficient NCC are calculated by the advanced feature description network HardNet and the similarity NCC function in sequence. The steps for calculating the ternary loss function include: Input the image S', source image S and target image T into the advanced feature description network HardNet to obtain feature descriptors des(S'), des(S) and des(T); Calculate Sim(S',T) based on des(S') and des(T) using the similarity NCC function, and calculate Sim(S,T) based on des(S) and des(T) using the similarity NCC function; The ternary loss function is calculated based on Sim(S',T) and Sim(S,T), and its expression is: L=max(Sim(S′,T)-Sim(S,T)+margin(δ),0), Among them, δ refers to the number of times the data set is trained; Complete image registration: Input the image pair to be registered into the optimized registration algorithm network to obtain the registered image.

2. The unsupervised learning-based drone aerial image registration method according to claim 1, characterized in that: The specific steps of generating the first correlation matrix C from the feature graph include: Input the source image S and target image T into the feature extraction network for feature extraction to obtain the source feature map f s and the target feature map f t ; For source feature map f s Each point f s (u,j), in the target feature map f t Take out the n points with the highest cosine similarity with point (i, j) and record them as f t (k,l); According to f s (i,j) and f t (k,l) calculates the sparsity correlation matrix The source feature map f s and target feature map f t The order of taking points is exchanged, and the target feature map f t Every point f t (i,j), in the source feature map f s Take out the n points with the highest cosine similarity of point (i, j) and record them as f S (k,l), according to f t (i,j) and f S (k,l) calculates the sparsity correlation matrix C T-＞S ; Add the two obtained sparse correlation matrices to obtain the first correlation matrix C; C=C ST =C S-＞T +C T-＞S 。 3. The unsupervised learning-based drone aerial image registration method according to claim 1, characterized in that: The second correlation matrix C' is specifically obtained by achieving domain consistency through a four-dimensional convolutional network.

4. The unsupervised learning-based UAV aerial image registration method according to claim 3, characterized in that: The four-dimensional convolutional network includes three four-dimensional sparse convolutional layers, wherein: The first four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and has 10 output channels; The second four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and has 10 output channels; The third four-dimensional sparse convolution layer uses a 3*3*3*3 convolution kernel and an output channel of 1.

5. The unsupervised learning-based UAV aerial image registration method according to claim 1, characterized in that: The regression network includes a first two-dimensional convolution layer connected in sequence, using an 11*11 convolution kernel and an output channel of 225; a second two-dimensional convolution layer using a 7*7 convolution kernel and an output channel of 128; a third two-dimensional convolution layer using a 5*5 convolution kernel and an output channel of 64; and a fourth activation function layer activated by a relu function; wherein, each two-dimensional convolution layer is connected to a regularization layer.

6. The unsupervised learning-based UAV aerial image registration method according to claim 1, characterized in that: The two-dimensional affine matrix θ ts Contains 6 set transformation parameters a1, a2, a3, a4, a5, a6, expressed as:

7. A UAV aerial image registration system based on unsupervised learning, characterized in that: The system includes; The dataset module is used to obtain continuous frame images taken by drones and establish a dataset. Two adjacent frames of continuous frame images are combined into a group of image pairs. Each image pair includes a source image S and a target image T. Registration algorithm network module: used to optimize the registration algorithm network, including: inputting the image into the feature extraction network to extract features to obtain a feature map; generating a first correlation matrix C from the feature map; inputting the first correlation matrix C into the consistency network to obtain a second correlation matrix C'; inputting the second correlation matrix into the regression network to obtain a two-dimensional affine matrix θ ts ; The source image S and the two-dimensional affine matrix θ ts Input into the spatial transformation network to obtain the geometrically transformed image S'; The geometrically changed image S', the source image S and the target image T are input into the similarity network to optimize the ternary loss function. If the value of the ternary loss function stops decreasing, the iteration is stopped and the registration algorithm network optimization is completed. The optimized ternary loss function is realized by calculating the parameters of the ternary loss function through the similarity network. The similarity network includes the advanced feature description network HardNet and the similarity NCC function. The two-dimensional affine matrix θ ts The feature descriptor des(S') and the similarity coefficient NCC are calculated by the advanced feature description network HardNet and the similarity NCC function in sequence. The steps for calculating the ternary loss function include: Input the image S', source image S and target image T into the advanced feature description network HardNet to obtain feature descriptors des(S'), des(S) and des(T); Calculate Sim(S',T) based on des(S') and des(T) using the similarity NCC function, and calculate Sim(S,T) based on des(S) and des(T) using the similarity NCC function; The ternary loss function is calculated based on Sim(S',T) and Sim(S,T), and its expression is: L=max(Sim(S′,T)-Sim(S,T)+margin(δ),0), Among them, δ refers to the number of times the data set is trained; Image registration module: used to input the image pair to be registered into the optimized registration algorithm network to obtain the registered image.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images