Aerial image matching method, electronic equipment and medium

Abstract

The invention discloses an aerial image matching method, electronic equipment and a medium, and the method comprises the following steps: S1, collecting an aerial image data set, forming an image pair composed of a source image and a target image, and processing the image pair through an efficient Transform module; s2, inputting the processed data pair into a feature extraction network fused by a residual network and dynamic convolution, and extracting effective feature points in the image; s3, the feature matching network screens out matching point pairs which are highly consistent and have no repetition through bidirectional matching and Pearson's correlation evaluation, and a corresponding relation between the two images is established; and S4, calculating and outputting geometric transformation parameters between the two aerial images through a parameter regression network. Compared with the prior art, the aerial images in different time periods are used for comparison, feature extraction of local information is enriched while structure and context information in the aerial images is captured, a good effect is achieved in disaster monitoring, and disaster differences are visually reflected.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to an aerial image matching method, electronic equipment and medium. Background Art

[0002] Aerial image matching involves geometrically aligning images from different conditions (such as time, perspective, and sensor differences), and is a key technology in areas such as assessing environmental changes, disaster monitoring, and urban planning. This process can use traditional classical algorithms or advanced end-to-end deep learning algorithms to achieve efficient matching. In short, aerial image matching technology uses different algorithms to ensure accurate correspondence between images to support research and applications in multiple fields.

[0003] Traditional convolutional neural networks (such as ResNet101) usually rely on local receptive fields for feature extraction. Although deep networks can gradually increase the receptive field by stacking multiple convolutional layers, their ability to capture global contextual information is still limited when dealing with high-dimensional and complex scenes. Compared with traditional convolutional neural networks, the Transformer model can capture global contextual information more effectively through the self-attention mechanism. The self-attention mechanism allows the model to dynamically focus on information at different positions when processing the input sequence, thereby achieving a comprehensive understanding of global information.

[0004] They pointed out that traditional matching methods often only focus on the one-way mapping from source image to target image. This one-way processing method limits the accuracy and effect of matching. In view of the weak robustness and low matching accuracy of aerial images in disaster monitoring, a feature extraction framework combining improved Transformer and dynamic convolution is proposed. In the aerial image feature matching stage, the Pearson bidirectional correlation matching algorithm is designed to match the source image and target image of the aerial image pair. Summary of the invention

[0005] The purpose of the present invention is to provide an aerial image matching method, electronic device and medium to solve the problems raised in the above background technology.

[0006] To achieve the above-mentioned object of the invention, one aspect of the present invention provides an aerial image matching method, comprising the following steps:

[0007] Step S1, collect aerial image datasets, which consist of multiple pairs of images. Each pair of images shows the disaster changes in the same area at different time points, that is, each pair contains a source image and a target image, which represent the aerial scenes of the area at earlier and later time points, respectively, to reflect the disaster changes over a long period of time. First, enter the improved efficient Transformer module. The efficient Transformer is an improvement on the core structure of the Transformer, the multi-headed self-attention, and maintains the diversity capability by introducing the efficient multi-headed self-attention (EMSA).

[0008] Step S2, inputting the processed data pair into a feature extraction network fused by a residual network and a dynamic convolution, and improving the feature expression capability by aggregating multiple parallel convolution kernels into one convolution kernel to extract effective feature points in the image;

[0009] Step S3, the feature extraction network extracts key features from the source image and the target image, and the feature matching network selects highly consistent and non-repetitive matching point pairs through bidirectional matching and Pearson correlation evaluation to establish a corresponding relationship between the two images;

[0010] Step S4, after double screening by bidirectional matching and adaptive neighborhood consensus network, we obtain dense image correspondence maps. Subsequently, these correspondence maps are passed to the parameter regression network, which directly calculates and outputs the geometric transformation parameters between the two aerial images, respectively for the direction from source image to target image and from target image to source image. In order to obtain more robust transformation parameters, we further perform weighted fusion on the parameters in these two directions. Using this weighted synthesis parameter, we perform affine transformation on the source image to achieve an accurate match with the target image.

[0011] Furthermore, in step S1, a random affine transformation is performed on the image pairs to generate a training set. The training set constructs a large collection of 18,000 image pairs, which are generated by carefully designed random affine transformations to ensure the diversity and representativeness of the data. At the same time, in order to ensure the effectiveness of the training process, we also set up a validation set containing 800 pairs of images to regularly evaluate the performance of the model during the training process and perform necessary tuning and optimization accordingly. This has successfully laid a solid foundation for the training of the aerial image semantic matching model and promoted the model's accurate matching capabilities in a variety of scenarios.

[0012] Furthermore, the efficient Transformer module is generated by the following steps:

[0013] Step S101, a set of projections is used to obtain the query Q. In order to compress memory, the 2D input The input is reshaped into 3D along the spatial dimension, represented as The height and width dimensions are reduced by a factor of s through depthwise convolution. s is adaptively set by the number of EMSAv2 heads k. The kernel size, stride, and padding are s+1, s, and s / 2, respectively.

[0014] Step S102, is reshaped into a two-dimensional map and represented as Where n′=h / s×w / s; Input two sets of projections to get the key K and value V;

[0015] Step S103, calculating the attention function on the query Q, key K and value V, the improved Transformer module can be defined as the following formula:

[0016]

[0017] Among them, Conv is a standard 1×1 convolution, K and Q are the attention dependencies of the head, and IN (Softmax) represents the instance normalized exponential function;

[0018] Step S104, concatenating and linearly projecting the output values ​​of each head to form a final output.

[0019] Furthermore, in step S2, the dynamic convolution does not use a single convolution kernel in each layer, but aggregates multiple convolution kernels into a dynamic kernel through attention. By adjusting the parameters of each weight, the appropriate parameters can be selected to extract image features. First, a general dynamic perceptron is defined, and then applied to convolution. In the architecture of Resnet101, the traditional 3x3 convolution layer is replaced by a dynamic convolution layer. This transformation aims to optimize the balance between the structure and computational efficiency of the network, while enhancing the expressive power of the model. By introducing dynamic convolution, Resnet101 can achieve more refined and adaptable extraction of image features without expanding the scale of the network.

[0020] Furthermore, the dynamic perceptron is based on the attention of the linear function π k Dynamically aggregate them, the linear function is expressed as follows:

[0021]

[0022] Then aggregate the linear function To define the dynamic perceptron, it is expressed as the following formula:

[0023]

[0024] in g represents weight, bias and activation function respectively; π k Represents the attention weight, which is not fixed but changes with the input; π k The calculation of attention weights and dynamic weights is integrated, and O(·) represents the amount of calculation of the perceptron. Similar to the dynamic perceptron, the dynamic convolution also has K convolution kernels. The dynamic convolution is followed by the BN layer and the ReLU layer. K kernels with the same scale and number of channels are set for one of the layers, and the respective attention weights are integrated to obtain the convolution kernel parameters of the layer. At the same time, global average pooling is performed in the attention layer to obtain the global spatial features, which are then mapped to K dimensions through two fully connected layers. Finally, Softmax normalization is performed, and the obtained attention weights can be allocated to the K kernels of the layer. The original fixed convolution kernel can now be dynamically selected according to the input, which effectively improves the feature expression capability.

[0025] Furthermore, a bidirectional semantic matching strategy is adopted in step S3, which significantly enhances the robustness and accuracy of semantic matching. In the process of bidirectional matching, the dense feature map between the source image and the target image becomes the key bridge. In short, by introducing the bidirectional semantic matching mechanism and accurately calculating the dense feature map between images, a more accurate and stable matching effect is achieved. Through the bidirectional semantic matching network, the corresponding parameters of the dense feature points are obtained and transferred through the parameter regression network, which directly estimates the geometric transformation parameters between the two aerial images. The parameter regression network takes into account the needs of feature extraction and parameter estimation. It consists of two convolutional layers with convolution kernel sizes of 7×7 and 5×5 respectively. These two convolutional layers are responsible for extracting deep features from the dense correspondence map. Subsequently, three fully connected layers further process and integrate these features and finally output the geometric transformation parameters. The parameter regression network structure not only ensures the accuracy of parameter estimation, but also ensures the efficiency of calculation.

[0026] A second aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned aerial image matching method when executing the computer program.

[0027] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aerial image matching method as described above.

[0028] Compared with the prior art, the present system and method have the following advantages:

[0029] 1. The present invention can compare aerial images taken at different time periods to monitor disaster changes. The proposed innovation can not only capture the structural and contextual information in the aerial images, but also enrich the feature extraction of local information, thus playing a good role in disaster monitoring and intuitively reflecting the differences in disasters.

[0030] 2. The present invention replaces the 3×3 convolution in the original Resnet101 residual block with dynamic convolution and combines it with other pre-trained convolution layers for feature extraction. By aggregating multiple parallel convolution kernels into one convolution kernel, the feature expression capability is improved, the calculation of additional parameters is reduced, and deep features can be extracted, avoiding the incomplete expression of single feature information.

[0031] 3. The present invention will improve the feature extraction framework of Transformer and dynamic convolution. In the aerial image feature matching stage, the Pearson bidirectional cross-correlation matching algorithm is designed to match the source image and target image of the aerial image pair. The improved efficient Transformer module can capture local information related to long-distance dependencies and more comprehensive spatial information, and learn the contextual information of aerial images. Dynamic convolution replaces the convolution in the original Resnet101 residual block and is combined with other pre-trained convolution layers for feature extraction, which can extract deep and refined features. The features extracted from the aerial source image and the target image through the feature extraction network are input into the feature matching network for matching, a bidirectional matching network is designed, and the Pearson cross-correlation algorithm is used to establish the correspondence between the aerial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flow chart of an aerial image matching method.

[0033] Figure 2 The figure is a framework diagram of the algorithm of the method of the present invention.

[0034] Figure 3 This is the network structure diagram of the efficient Transformer module proposed in this invention.

[0035] Figure 4 This is a qualitative comparison experiment result diagram of the comparative experiment of the present invention on the Aerial Image Dataset.

[0036] Figure 5 The following is a qualitative comparison of the experimental results on the Google Earth dataset. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 The flowchart of the method of the present invention is shown. The embodiment of the present invention provides an aerial image matching method based on disaster monitoring. In order to obtain image pairs before and after the disaster from the Google Earth dataset, we first uniformly adjust the images to 240x240 pixels. Next, a feature extraction network is designed, which combines the advantages of the initial structure of ResNet101 and Transformer, retains the initial convolution layer, pooling layer and the first three residual layers of ResNet101 (the number of channels is 64, 128, and 256, respectively), to effectively extract the basic and deep features of the image. Subsequently, the remaining residual blocks and fully connected layers are removed to simplify the network and focus on capturing the key features of disaster changes. Finally, the efficient Transformer module and the advantages of ResNet101 and dynamic convolution are combined to construct a feature extraction network optimized for disaster image pairs. The source image and the target image of the aerial image are input into the feature extraction network composed of the efficient Transformer module and dynamic convolution to extract the effective feature points of the two images. Figure 2 This is the algorithm framework diagram of this method.

[0039] The specific steps are as follows:

[0040] Step S1, a dataset of 9,000 pairs of aerial images was collected from the Google Earth platform. We used random affine transformation techniques to augment the data and generate source and target image pairs suitable for training aerial image matching models. Before training, a preprocessing step was performed on each pair of images, including cropping them to match the specific input size required by the feature extraction network. Subsequently, these pairs of images were used as samples and fed into the feature extraction network for processing in batches of 4 pairs of images. The entire training process was based on a large training set containing 18,000 pairs of images, and a validation set containing 625 pairs of images was set up to ensure effective validation and tuning of the model during training.

[0041] First, we will move on to the improved efficient Transformer module. The efficient Transformer improves the core structure of the Transformer, the multi-head self-attention. While extracting the structural and contextual information in the aerial images, it can also enrich the feature extraction of local information.

[0042] like Figure 3The network structure diagram of the efficient Transformer module of the present invention is shown in FIG. Similar to the standard Transformer (MSA), EMSA first uses a set of projections to obtain the query Q. In order to compress memory, the 2D input The input is reshaped into 3D along the spatial dimension (i.e. ), the height and width dimensions are reduced by a factor of s through depthwise convolution. s is adaptively set by the number of EMSAv2 heads k. The kernel size, stride, and padding are s+1, s, and s / 2, respectively.

[0043] is reshaped into a two-dimensional map, namely n′=h / s×w / s. Then Two sets of projections are fed in to obtain the key K and value V. After that, the attention function on the query Q, key K and value V is calculated. The EMSAV2 module can be defined as Equation 1:

[0044]

[0045] Conv is a standard 1×1 convolution, K and Q are the attention dependencies of the head. Instance normalization (i.e., IN) is added after Softmax. Finally, the output values ​​of each head are concatenated and linearly projected to form the final output.

[0046] Step S2, then input into the feature extraction network composed of residual network and dynamic convolution, and aggregate multiple parallel convolution kernels into one convolution kernel to improve the feature expression ability and extract effective feature points of the two images. First, define a general dynamic perceptron. The dynamic perceptron is based on the attention of multiple linear functions (π k ) dynamically aggregate them, these linear functions depend on the input, by aggregating multiple k linear functions To define a dynamic perceptron, where: g represents weight, bias and activation function respectively; π k Represents the attention weight, which is not fixed but changes with the input. k The calculation of attention weights and dynamic weights is integrated. Similar to the dynamic perceptron, dynamic convolution also has K convolution kernels. The dynamic convolution is followed by a BN layer and a ReLU layer. K kernels with the same scale and number of channels are set for one of the layers, and the respective attention weights are integrated to obtain the convolution kernel parameters of the layer. At the same time, global average pooling is performed in the attention layer to obtain the global spatial features, which are then mapped to K dimensions through two fully connected layers. Finally, Softmax normalization is performed, and the obtained attention weights can be allocated to the K kernels of the layer. The original fixed convolution kernel can now be dynamically selected according to the input, which effectively improves the feature expression capability.

[0047] Step S3, the output feature tensor (1×1024×3×3) is input into the feature matching network as the input for image matching. After the feature extraction network completes its task, it will output feature tensors, which are then sent to the feature matching network to achieve accurate matching of valid feature points in the two aerial images. In the matching process, we originally used the cross-correlation algorithm to evaluate the similarity between the two images, but this method only implements a one-way matching from the source image to the target image, and its effect is not optimal. In order to improve this situation, we designed and implemented a matching network that not only performs matching from the source image to the target image, but also performs the reverse operation, that is, matching from the target image to the source image. Through the two-way matching strategy, combined with the Pearson cross-correlation algorithm, we can more comprehensively establish the correspondence between aerial images, thereby improving the accuracy and efficiency of matching.

[0048] Park et al. highlighted a key problem faced by aerial images when performing semantic matching: asymmetry. They pointed out that traditional matching methods often focus only on the one-way mapping from the source image to the target image. This one-way processing method limits the accuracy and effect of the matching. To overcome this limitation, a bidirectional semantic matching strategy was adopted, which significantly enhanced the robustness and accuracy of semantic matching. In the process of bidirectional matching, the dense feature map between the source image and the target image becomes the key bridge. In short, by introducing a bidirectional matching mechanism and accurately calculating the dense feature map between images, a more accurate and stable matching effect is achieved.

[0049] In step S4, the image is passed to a parameter regression network, which calculates and outputs the geometric transformation parameters between the two aerial images for the directions from the source image to the target image and from the target image to the source image, and performs weighted fusion on the parameters of the two directions.

[0050] Using the bidirectional semantic matching mechanism, we are able to obtain the correspondence parameters between dense feature points, which are then passed and processed through a carefully designed parameter regression network. This network focuses on extracting and estimating the precise geometric transformation parameters between the two aerial images from these correspondences. The structure of this regression network takes into account both the depth of feature extraction and the accuracy of parameter prediction. It consists of two key convolutional layers. The first layer uses a 7x7 convolution kernel, and the second layer uses a 5x5 convolution kernel. These two layers are responsible for deeply mining and abstracting high-level features in the dense correspondence map. Next, three fully connected layers work in relay to carefully analyze and synthesize the extracted features, and finally directly output the required geometric transformation parameters. This network architecture not only ensures high accuracy of parameter estimation, but also optimizes the calculation process through its efficient design.

[0051] Qualitative and quantitative comparison experiments on semantic alignment were conducted using three datasets: Aerial Image Dataset, Google Earth, and MTRSRD. The algorithm in this paper was compared with the traditional algorithm ORB and the deep learning algorithms A2net and Two-stream. 500 pairs of real remote sensing images were selected as test sets. These images cover a variety of complex scenes such as rivers, land, coasts, mountains, cities, etc., which can test the performance of the model in different environments. Python 3.8 and Pytorch deep learning framework were used, and the hardware environment was configured with RTX 3080 graphics card, Intel core i5 (2.3GHz) processor and 10GB memory.

[0052] We selected three sets of images to be semantically aligned from the Aerial Image Dataset for algorithm performance comparison, such as Figure 4 As shown in the figure, the alignment results of different algorithms for multi-angle docks, multi-temporal seaside and multi-source urban roads are shown respectively. In order to evaluate the local alignment effect of different algorithm images in detail, the results of each algorithm are shown in a checkerboard diagram. The target image and the aligned image are cut into 77 small blocks, and these small blocks are alternately spliced ​​to form a new image. By observing the connection and alignment between adjacent small blocks, it can be seen that this method has the best alignment effect. Figure 5 The three sets of images are from the Google Earth dataset, showing the alignment results of different algorithms on the narrow gaps on the grassland, the coastline on the sea, and the mountains. By comparing the results of the chessboard graph, it can be seen that the alignment effect of this method is the best.

[0053] In order to evaluate the performance of this method, the four quantitative indicators mentioned above, namely PCK, MSE, MAE and RMSE, were used. Among them, 300 pairs of images were randomly selected from each of the three datasets mentioned above, and a total of 900 pairs of remote sensing images were used as test sets for PCK indicator evaluation, which were used to calculate the correct key point semantic alignment ratios of four different methods. By comparing these ratios, the accuracy of these four methods in semantic alignment is inferred. When the PCK value is 0.1, the performance of this method is improved by more than 30% compared with the traditional method. Under the condition that the PCK value is 0.1, the PCK value of this method is as high as 95.3%, surpassing other end-to-end deep learning methods in recent years. It proves the effectiveness of this method in improving the ratio of correct semantic alignment points.

[0054] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for matching aerial images, characterized in that: The following steps are involved: Step S1, collecting an aerial image dataset to form an image pair consisting of a source image and a target image, and processing the image pair through an efficient Transformer module, wherein the efficient Transformer module introduces efficient multi-head self-attention to improve the multi-head self-attention of the Transformer; Step S2, inputting the processed data pair into a feature extraction network fused by a residual network and a dynamic convolution, and improving the feature expression capability by aggregating multiple parallel convolution kernels into one convolution kernel to extract effective feature points in the image; Step S3, through bidirectional matching and Pearson correlation evaluation, highly consistent and non-repetitive matching point pairs are screened out to establish a corresponding relationship between the two images; Step S4, through the parameter regression network, respectively for the direction from the source image to the target image and the direction from the target image to the source image, calculate and output the geometric transformation parameters between the two aerial images, and perform weighted fusion on the parameters of the two directions.

2. The aerial image matching method according to claim 1, characterized in that: A training set is generated by performing a random affine transformation on the image pairs to enhance the robustness and adaptability of the aerial image semantic matching model; and a validation set is set to regularly evaluate the performance of the model during the training process, and to adjust and optimize it accordingly.

3. The aerial image matching method based on disaster monitoring according to claim 1 is characterized in that: The efficient Transformer module is generated by the following steps: Step S101, the input is expressed as After depthwise convolution, the height and width dimensions are reduced by a factor of s; s is adaptively set by the number of EMSAv2 heads k; the kernel size, stride, and padding are s+1, s, and s / 2, respectively; Step S102, is reshaped into a two-dimensional map and represented as Where n′=h / s×w / s; Through different projection methods, meaningful information is extracted from the data and expressed in the form of keys and values ​​to obtain keys K and values ​​V. Step S103, calculating the attention function on the query Q, key K and value V, the improved Transformer module can be defined as the following formula: Among them, Conv is a standard 1×1 volume, product, K and Q are the attention dependencies of the head, IN (Softmax) function represents the instance normalized exponential function, d k Represents the dimension of the feature, and Up(V) means upsampling the value vector V to align with other feature maps; Step S104, concatenating and linearly projecting the output values ​​of each head to form a final output.

4. The aerial image matching method according to claim 1, characterized in that: In step S2, dynamic convolution aggregates multiple convolution kernels into a dynamic kernel through attention, and selects appropriate parameters to extract image features in a targeted manner by adjusting the parameters of each weight. The implementation method is to first define a general dynamic perceptron and then perform convolution on it.

5. The aerial image matching method according to claim 4, characterized in that: The dynamic perceptron is based on the attention of the linear function π k Dynamically aggregate them, the linear function is expressed as follows: Then aggregate the linear function To define the dynamic perceptron, it is expressed as the following formula: in g represents weight, bias and activation function respectively; π k Represents the attention weight, which changes with the input; π k The calculation of attention weights and dynamic weights is integrated, and O(·) represents the amount of calculation of the perceptron.

6. The aerial image matching method according to claim 1, characterized in that: In step S3, a bidirectional semantic matching strategy is adopted, and dense feature maps between images are accurately calculated to enhance the robustness and accuracy of semantic matching.

7. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, an aerial image matching method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, an aerial image matching method as described in any one of claims 1 to 6 is implemented.

Images