Lightweight cross-view image matching and positioning method and system based on multi-scale feature aggregation

Abstract

The invention relates to the technical field of geographic positioning, in particular to a lightweight cross-view image matching and positioning method and system based on multi-scale feature aggregation, which are used for performing matching and positioning on a ground panoramic image without geographic position information. The method comprises the following steps: firstly, collecting a satellite image with geographic position information in a target area, performing slicing and polar coordinate transformation on the satellite image, and constructing a reference image database; and then extracting global feature descriptors of the ground view query image and the satellite image by using a pre-trained cross-view image matching network, and matching the feature descriptors of the query image with the feature descriptors of the satellite image so as to determine the geographic position of the ground view query image. Wherein the cross-view image matching network model is constructed by adopting a twin neural network architecture, and the network architecture comprises a feature extraction network used for extracting image local features and a multi-scale feature aggregation network used for aggregating the image features into global descriptors from multiple scales. According to the invention, while high-precision cross-view-angle image matching positioning is maintained, the parameter quantity and the calculation quantity of the model are reduced, and the hardware requirement of model deployment is reduced.

Description

Technical Field

[0001] The present invention relates to the field of geographic positioning technology, and in particular to a lightweight cross-view image matching positioning method and system based on multi-scale feature aggregation. Background Art

[0002] With the rapid advancement of science and technology, the Global Navigation Satellite System (GNSS) has profoundly changed the daily life of human beings and played an indispensable role in all fields of society. However, the GNSS signal is easily interfered by the environment and electromagnetic signals, which makes it unable to fully meet the needs of positioning, navigation and other services. Therefore, it is of great significance to study efficient geolocation technology in GNSS denial or interference environment. In this context, image-based geolocation technology has been continuously developed as an effective solution. In recent years, due to the wide coverage and easy acquisition of satellite images and aerial images, geolocation technology based on cross-view image matching has gradually attracted people's attention. Geolocation technology based on cross-view image matching refers to the technology of matching and identifying the reference image most similar to the query image (drone, satellite, ground image, etc.) in a pre-built reference aerial image database with geographic information through a certain algorithm framework, so as to obtain the geographic location of the query image. This paper mainly studies the cross-view image matching and geolocation technology for ground panoramic images and satellite images.

[0003] Due to the different shooting angles of cross-view images, there is a huge domain gap between satellite images and ground panoramic images, which is manifested in the differences in the appearance and spatial layout of objects. These factors make it difficult for traditional matching algorithms based on local image feature descriptors to successfully match. In recent years, with the continuous breakthroughs in deep learning technology in the field of computer vision, new solutions have been provided for geolocation technology based on cross-view image matching.

[0004] At present, most cross-view image matching research focuses on perspective conversion and metric learning methods, ignoring the differences in features in images and the resource consumption of models. Specifically, these problems include but are not limited to the following three aspects: (1) Failure to fully consider the differences in the scale of ground object features: Different types of objects have significantly different ranges of context information required in the feature extraction process. The extraction of some object features may only require reference to local context information, while others may require a wider context to reveal their full picture. This difference requires that when we design feature extraction algorithms, we must consider the requirements of different objects for feature extraction ranges to achieve dynamic and accurate feature information extraction; (2) Ignoring the context dependence of image spatial layout information: The spatial position information between objects is also a key factor that cannot be ignored in cross-view image matching. The extraction of this information also depends on context information of different ranges. For example, the relative position relationship of some objects may only be meaningful within their immediate neighborhood, while others may require a wider field of view to be accurately determined. Therefore, building a model that can adapt to different context ranges to optimize the extraction of spatial position information is a key way to improve the accuracy of cross-view image matching. (3) Computational complexity and resource consumption: Existing high-performance matching models are often accompanied by a large parameter scale and heavy computational burden, which places strict requirements on the computing power of the execution platform and limits the deployment scope of the model in practical applications. Summary of the invention

[0005] To this end, the present invention provides a lightweight cross-view image matching and positioning method and system based on multi-scale feature aggregation, which solves the problems that the existing cross-view image matching and positioning model does not fully consider the scale differences of ground feature features, resulting in affected matching and positioning accuracy and large model hardware resource consumption.

[0006] According to the design scheme provided by the present invention, on the one hand, a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation is provided, comprising:

[0007] Obtain satellite images with geographic location information and ground panoramic images to be located in the target area, perform polar coordinate transformation on the satellite images, use the polar coordinate transformed satellite images as aerial view reference images, and use the ground panoramic images as ground view query images;

[0008] The ground-view query image is matched with the aerial-view reference image using a pre-trained cross-view image matching network model, and the geographic location of the ground-view query image is determined by the location information of the matched aerial-view image.

[0009] Among them, the cross-view image matching network model is constructed using a twin neural network architecture, which includes a feature extraction network for extracting local features of the image and a multi-scale feature aggregation network for aggregating image features from multiple scales into global descriptors.

[0010] As a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation of the present invention, further, polar coordinate transformation is performed on the satellite image, including:

[0011] The center of the satellite image is taken as the origin of polar coordinates, and the due north direction is taken as the zero degree angle in polar coordinates;

[0012] Polar coordinate transformation is performed on pixel coordinates in the satellite image using polar coordinate transformation rules, and the polar coordinate transformation rules are set according to the size of the satellite image before and after the transformation.

[0013] As a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation of the present invention, further, the feature extraction network adopts a selective large convolution kernel network LskNet as the backbone network for feature extraction, and the large convolution kernel network LskNet includes a large kernel selection sub-block for dynamically adjusting the network receptive field and a feedforward network sub-block for channel mixing and feature refinement.

[0014] As a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation of the present invention, further, a feature extraction network is used to extract local features of the image, including: four feature extraction stages, in each feature extraction stage, the input image is first downsampled, and then the features of the input image are extracted using the feature extraction network; the feature map extracted in the last stage is used as output, so that the output is used as the input feature of the multi-scale feature aggregation network.

[0015] As a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation of the present invention, further, the multi-scale feature aggregation network is constructed based on the SPE module, and the convolution kernel in the SPE module is decomposed into two consecutive layer-by-layer convolutions to utilize two consecutive layer-by-layer convolutions to extract different scale features of the image.

[0016] As a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation of the present invention, further, a multi-scale feature aggregation network is used to aggregate image features from multiple scales into a global descriptor, including:

[0017] Two consecutive layer-by-layer convolutions are used to extract features of different scales of the input image, wherein the convolution kernel sizes and dilation rates of the two consecutive layer-by-layer convolutions are different;

[0018] The features extracted by convolution layer by layer are mixed using a convolution kernel of size 1×1, while compressing the number of feature channels;

[0019] The spliced ​​image features are averaged and maximum pooled in turn along the channel direction, and the pooling results are projected using two fully connected layers to obtain a position encoding map of the image features;

[0020] The position coding map and the feature map on each channel of the image are used to perform matrix inner product calculation, and the calculation results are concatenated to obtain the feature descriptor of the image.

[0021] As a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation of the present invention, further, the training process of the cross-view image matching network model includes:

[0022] Acquire positive sample data and negative sample data for model training, wherein the positive sample data includes street view images and satellite images corresponding to the street view images, and the negative sample data includes street view images and satellite images with mismatched regional locations;

[0023] The model training loss function is constructed based on the cosine similarity between the street view image and the satellite image in the positive sample data and the negative sample data;

[0024] Based on the model training loss function, the cross-view image matching network model is trained and learned using positive sample data and negative sample data, so that the trained cross-view image matching network model can learn the feature representation space between the query image and the reference image.

[0025] On the other hand, the present invention also provides a lightweight cross-view image matching and positioning system based on multi-scale feature aggregation, comprising:

[0026] Aerial image database module, image feature extraction module and cross-view image matching module, among which,

[0027] The aerial image database module is used to collect satellite images with geographic location information in the target area, slice and transform the satellite images into polar coordinates, and build a reference image database;

[0028] An image feature extraction module, which is used to extract global feature descriptors of ground-view query images and satellite images using a pre-trained cross-view image matching network;

[0029] A cross-view image matching module is used to match the feature descriptors of the query image with the feature descriptors of the satellite image to determine the geographic location of the ground-view query image;

[0030] Among them, the cross-view image matching network model is constructed using a twin neural network architecture, which includes a feature extraction network for extracting local features of the image and a multi-scale feature aggregation network for aggregating image features from multiple scales into global descriptors.

[0031] Beneficial effects of the present invention:

[0032] The present invention performs polar coordinate transformation on satellite images, reduces the geometric layout differences between images at different viewing angles, uses a feature extraction network to extract local features in images, uses a multi-scale feature aggregation network to aggregate image features from multiple scales into a global descriptor, and encodes the spatial layout information of the image into the global feature descriptor, so as to realize multi-scale feature aggregation lightweight cross-view image matching and positioning for ground panoramic images and satellite images; and reduces the hardware resource consumption of the model and the difficulty of lightweight deployment by introducing LskNet and multi-scale feature aggregation; in order to enhance the flexibility and generalization ability of the model and avoid over-reliance on data enhancement technology, an adaptive sharpness perceived minimization (ASAM) strategy is used, and by adjusting the adaptive sharpness of the loss function, the problem that the loss function is prone to fall into the local optimum during training is effectively avoided, thereby improving the generalization ability of the model. Further experimental results show that the recall rate of the top 1 images of this solution on the VIGOR and CVACT datasets reached 79.00% and 91.43%, respectively, which was 1.14% and 0.62% higher than Sample4Geo, and the accuracy on the CVUSA dataset reached 98.64%, which is almost the same as Sample4Geo. The number of parameters of this solution on the VIGOR dataset dropped to 30.09M, and the amount of calculation was 16.05GFLOPs, which was 57.48M parameters and 51.66GFLOPs lower than Sample4Geo. Compared with the existing classic cross-view image matching methods, this solution significantly reduces the number of parameters and calculations while maintaining high accuracy, thereby reducing the hardware requirements for model deployment and meeting the application scenarios that require lightweight model deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of a lightweight cross-view image matching and positioning process based on multi-scale feature aggregation in an embodiment;

[0034] Figure 2 This is a schematic diagram of the model training algorithm principle in the embodiment;

[0035] Figure 3 This is a schematic diagram of polar coordinate transformation in the embodiment;

[0036] Figure 4 This is a schematic diagram of the LskNet network structure in the embodiment;

[0037] Figure 5 Schematic diagram of the multi-scale feature aggregation module structure in the embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0039] Cross-view image matching and positioning refers to the technology of determining the geographic location of a ground-view query image by matching it with an aerial-view reference image with a geographic tag. Current cross-view image matching and positioning technologies mainly use CNNs with fixed receptive fields or Transformers with global modeling capabilities as feature extraction backbone networks. However, these methods fail to fully consider the scale differences between different features in the image, and due to the high number of network parameters and computational complexity, lightweight deployment faces significant challenges. To this end, the embodiments of the present invention, see Figure 1 As shown, a lightweight cross-view image matching and positioning method based on multi-scale feature aggregation is provided, comprising:

[0040] S101, obtaining a satellite image with geographic location information and a ground panoramic image to be located in the target area, performing polar coordinate transformation on the satellite image, using the satellite image after polar coordinate transformation as an aerial view reference image, and using the ground panoramic image as a ground view query image;

[0041] S102, using a pre-trained cross-view image matching network model to match the ground view query image with the air view reference image, and determining the geographical location of the ground view query image through the location information of the matched air view image;

[0042] Among them, the cross-view image matching network model is constructed using a twin neural network architecture, which includes a feature extraction network for extracting local features of the image and a multi-scale feature aggregation network for aggregating image features from multiple scales into global descriptors.

[0043] Specifically, the polar coordinate transformation of satellite images can be designed to include:

[0044] The center of the satellite image is taken as the origin of polar coordinates, and the due north direction is taken as the zero degree angle in polar coordinates;

[0045] Polar coordinate transformation is performed on pixel coordinates in the satellite image using polar coordinate transformation rules, wherein the polar coordinate transformation rules are set according to the size of the satellite image before and after the transformation.

[0046] When the image centers are aligned (the shooting position of the panoramic image is located at the center of the satellite image), there is a specific geometric correspondence between the satellite image and the ground panoramic image. In this embodiment, the satellite image is transformed into polar coordinates. Specifically, the center of the satellite image is used as the origin of the polar coordinates, and the due north direction is used as the 0° angle in the polar coordinates. When the size of the original satellite image is A a ×A a, the size of the transformed satellite image is W g× H g When , the formula for polar coordinate transformation can be defined as:

[0047]

[0048] in represents the pixel coordinates of satellite images, Represents the pixel coordinates of the satellite image after polar coordinate transformation.

[0049] like Figure 3 As shown in the figure, the comparison results of satellite images and panoramic images before and after polar coordinate transformation. After polar coordinate transformation, the geometric correspondence between satellite images and ground panoramic images is clearer, which simplifies the model's learning of the geometric correspondence between images, thereby greatly promoting the convergence of the model.

[0050] Specifically, the feature extraction network uses a selective large convolution kernel network LskNet as the backbone network for feature extraction, and the large convolution kernel network LskNet includes a large kernel selection sub-block for dynamically adjusting the network receptive field and a feedforward network sub-block for channel mixing and feature refinement. The multi-scale feature aggregation network is constructed based on the SPE module, and the convolution kernel in the SPE module is decomposed into two consecutive layer-by-layer convolutions to extract different scale features of the image using two consecutive layer-by-layer convolutions.

[0051] like Figure 2 As shown in the figure, the satellite image is first transformed into polar coordinates to reduce the geometric layout differences between images at different viewing angles. Then, the large selective convolutional network LskNet is used to extract local features in the image. Finally, the multi-scale feature aggregation module is used to aggregate image features from multiple scales into global descriptors. In the feature matching and loss calculation stage, the weighted soft margin ternary loss function proposed in CVM-Net is used to calculate the model loss.

[0052] Among them, the feature extraction network is used to extract local features of the image, which can be designed to include: four feature extraction stages, in each feature extraction stage, the input image is first downsampled, and then the features of the input image are extracted using the feature extraction network; the feature map extracted in the last stage is used as the output, so that the output is used as the input feature of the multi-scale feature aggregation network.

[0053] like Figure 4The structure of the LskNet module shown is a core repetitive block in LskNet, which consists of two residual sub-blocks: a large kernel selection (LK Selection) sub-block and a feedforward network (FFN) sub-block. The large kernel selection sub-block can dynamically adjust the receptive field of the network as needed, and the feedforward network sub-block is used for channel mixing and feature refinement, and consists of full connection, layer-by-layer convolution, GELU activation function and a second full connection. The LskNet network has a total of four stages. At the beginning of each stage, the input image is downsampled once, and then a series of LskNet blocks composed of large kernel selection sub-blocks and feedforward network sub-blocks are used to extract the features of the input image, and finally the feature maps of the last three stages are output. In this embodiment of the case, the feature map output by the last stage of LskNet is used as the input feature of the multi-scale feature aggregation module.

[0054] Among them, a multi-scale feature aggregation network is used to aggregate image features from multiple scales into a global descriptor, which can be designed to include:

[0055] Two consecutive layer-by-layer convolutions are used to extract features of different scales of the input image, wherein the convolution kernel sizes and dilation rates of the two consecutive layer-by-layer convolutions are different;

[0056] The features extracted by convolution layer by layer are mixed using a convolution kernel of size 1×1, while compressing the number of feature channels;

[0057] The spliced ​​image features are averaged and maximum pooled in turn along the channel direction, and the pooling results are projected using two fully connected layers to obtain a position encoding map of the image features;

[0058] The position coding map and the feature map on each channel of the image are used to perform matrix inner product calculation, and the calculation results are concatenated to obtain the feature descriptor of the image.

[0059] Since the features contained in the image have different scales and the context information required for extracting different features is also different, blindly using the same size convolution kernel in the feature aggregation stage cannot effectively utilize features of different scales and spatial layout information of different ranges. Therefore, designing a feature aggregation module that can effectively utilize features of different scales and aggregate spatial layout information into a global descriptor is of vital importance for cross-view image matching.

[0060] In this embodiment, based on the SPE module of SAFA, a large-size convolution kernel is decomposed into two consecutive relatively small layer-by-layer convolutions, and a multi-scale feature aggregation network is designed, such as Figure 5As shown. Specifically, the network can decompose a large convolution kernel of size 23×23 into two layer-by-layer convolutions of size 5×5 and 7×7, with expansion rates of 1 and 3, respectively, to extract features of different scales of the image. Compared with using only one large convolution kernel, decomposing a large convolution kernel into two relatively small layer-by-layer convolutions not only reduces the amount of computation of the model, but also enables the model to better adapt to features of different scales. Table 1 shows the number of parameters and computation of the convolution kernels in the two modes when calculating features of size 320×320 pixels and number of channels 64, where the convolution kernel parameter (23,1) represents a layer-by-layer convolution of size 23 and expansion rate 1. Compared with using only one large convolution kernel, after decomposing it into two relatively small convolution kernels, the amount of computation of the convolution kernel is reduced from 3.473GFLOPs to 0.498GFLOPs, and the number of parameters is reduced from 33.92K to 4.864K, which greatly reduces the consumption of computing resources.

[0061] Table 1 Parameters and calculations before and after convolution kernel decomposition

[0062]

[0063] After using layer-by-layer convolution to extract image features, 1×1 convolution is used to mix features of different scales, compress the number of channels, and concatenate the features extracted by two layer-by-layer convolutions. The process can be expressed as follows:

[0064]

[0065] Among them, X represents the image features extracted by LskNet, and U, U1, and U2 represent intermediate features. It represents a convolutional layer with d input channels, d output channels, a kernel size of 5, and a dilation rate of 1. [] represents the concatenation operation of the feature vector along the channel direction.

[0066] Then, the features are averaged and maximum pooled along the channel direction to screen the significant features in the image. Then, two fully connected layers are used to process the pooling results, and the pooling results are projected to obtain the position encoding map of the features. The process can be expressed as follows:

[0067] SA avg =P avg (U),SA max =P max (U) (4)

[0068]

[0069] P=FC(SA) (6) Where P avg () and P max() represents mean pooling and maximum pooling operations, SA avg and SA max It represents the spatial features obtained by mean pooling and maximum pooling, SA is the spatial attention map, FC represents full connection, and P is the obtained position encoding map.

[0070] Finally, the inner product of the matrix is ​​calculated using the position encoding map and the feature map on each channel of the image, and the calculation results are concatenated to finally obtain the feature descriptor of the image. Given the input feature X∈R H×W , feature descriptor K = {k c}, c=1,2,…, the calculation formula of C is as follows:

[0071] k c = <X c ,P> F (7)

[0072] This process effectively eliminates redundant information and ensures that the retained features are key features that are useful for matching tasks, thereby achieving feature screening and aggregation.

[0073] Specifically, the training process of the cross-view image matching network model can be designed to include:

[0074] Acquire positive sample data and negative sample data for model training, wherein the positive sample data includes street view images and satellite images corresponding to the street view images, and the negative sample data includes street view images and satellite images with mismatched regional locations;

[0075] The model training loss function is constructed based on the cosine similarity between the street view image and the satellite image in the positive sample data and the negative sample data;

[0076] Based on the model training loss function, the cross-view image matching network model is trained and learned using positive sample data and negative sample data, so that the trained cross-view image matching network model can learn the feature representation space between the query image and the reference image.

[0077] A weighted soft-margin ternary loss function is used to train feature extraction networks for ground and satellite images. Given a set of query images and reference images, the goal of model training is to learn a feature representation space in which the features of each street view query image are close to the features of its corresponding satellite image. In the dataset, each street view image and its corresponding satellite image are considered positive pairs, and other pairs are considered negative pairs. If there are multiple satellite images covering the same street view image area, such as the VIGOR dataset, the satellite image closest to the location where the street view image was taken can be used as a positive sample, while avoiding sampling other adjacent aerial images in the same batch to prevent blurred supervision. The loss function of the model can be expressed as:

[0078]

[0079] where d pos and d neg represents the cosine similarity between positive and negative pairs, α is a parameter used to adjust the loss gradient, which can control the convergence speed. Specifically, α can be set to 20. In a batch with N pairs of street view images and satellite images, 2N×(N-1) triplets can be sampled.

[0080] Furthermore, based on the above method, an embodiment of the present invention also provides a lightweight cross-view image matching and positioning system based on multi-scale feature aggregation, comprising:

[0081] Aerial image database module, image feature extraction module and cross-view image matching module, among which,

[0082] The aerial image database module is used to collect satellite images with geographic location information in the target area, slice and transform the satellite images into polar coordinates, and build a reference image database;

[0083] An image feature extraction module, which is used to extract global feature descriptors of ground-view query images and satellite images using a pre-trained cross-view image matching network;

[0084] A cross-view image matching module is used to match the feature descriptors of the query image with the feature descriptors of the satellite image to determine the geographic location of the ground-view query image;

[0085] Among them, the cross-view image matching network model is constructed using a twin neural network architecture, which includes a feature extraction network for extracting local features of the image and a multi-scale feature aggregation network for aggregating image features from multiple scales into global descriptors.

[0086] In order to verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0087] Comparative Algorithm: Sample4Geo uses a comparative learning method based on InfoNCE loss to train the model, while introducing GPS sampling and dynamic similarity sampling strategies to mine hard samples. The algorithm has shown excellent performance on several public cross-view matching datasets. However, the high accuracy of Sample4Geo relies on its huge number of parameters and computational complexity. Even if the weights are shared in the network of satellite images and panoramic images, the number of parameters required by the model is still large, which not only increases the time cost of model training, but also puts higher requirements on hardware resources.

[0088] Experimental environment: All programs in the experiment were run on the Ubuntu 22.04 system. The model was implemented based on the PyTorch 2.1.0 framework, the Python version was 3.9.18, the CPU was Intel Core i9-13900K, and the GPU was NVIDIA GeForceRTX4090. For the VIGOR dataset, this paper adjusted the size of the panoramic image and satellite image to 320×640 pixels and 320×320 pixels respectively, and the batch size was set to 24. For the CVUSA and CVACT datasets, the satellite images were polar transformed in the experiment. The size of the transformed satellite image and panoramic image was 128×704 pixels, the batch size was set to 48, and the global hard sample mining strategy was added when reading the data. LskNet-S was used as the baseline network of the model, and the LskNet pre-trained weights were used for initialization. During the training process, the initial learning rate was set to 0.0008, and the cosine decay strategy was used to adjust the learning rate.

[0089] The recall rates of the top 1 images on the VIGOR and CVACT datasets reached 79.00% and 91.43%, respectively, which were 1.14% and 0.62% higher than Sample4Geo, respectively. The accuracy on the CVUSA dataset reached 98.64%, which is almost the same as Sample4Geo. The parameters of this solution on the VIGOR dataset reached 30.09M, and the computational complexity reached 16.05GFLOPs, which were 57.48M parameters and 51.66GFLOPs lower than Sample4Geo. Compared with the existing classic cross-view image matching methods, this solution significantly reduces the number of parameters and computational complexity while maintaining high accuracy, thereby reducing the hardware requirements for model deployment.

[0090] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0091] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0092] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0093] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0094] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A lightweight cross-view image matching and positioning method based on multi-scale feature aggregation, characterized in that: Include: Obtain satellite images with geographic location information and ground panoramic images to be located in the target area, perform polar coordinate transformation on the satellite images, use the polar coordinate transformed satellite images as aerial view reference images, and use the ground panoramic images as ground view query images; The ground-view query image is matched with the aerial-view reference image using a pre-trained cross-view image matching network model, and the geographic location of the ground-view query image is determined by the location information of the matched aerial-view image. Among them, the cross-view image matching network model is constructed using a twin neural network architecture, which includes a feature extraction network for extracting local features of the image and a multi-scale feature aggregation network for aggregating image features from multiple scales into global descriptors.

2. The lightweight cross-view image matching and positioning method based on multi-scale feature aggregation according to claim 1 is characterized in that: Polar coordinate transformation of satellite images, including: The center of the satellite image is taken as the origin of polar coordinates, and the due north direction is taken as the zero degree angle in polar coordinates; Polar coordinate transformation is performed on pixel coordinates in the satellite image using polar coordinate transformation rules, wherein the polar coordinate transformation rules are set according to the size of the satellite image before and after the transformation.

3. The lightweight cross-view image matching and positioning method based on multi-scale feature aggregation according to claim 1 is characterized in that: The feature extraction network adopts a selective large convolution kernel network LskNet as a backbone network for feature extraction. The large convolution kernel network LskNet includes a large kernel selection sub-block for dynamically adjusting the network receptive field and a feedforward network sub-block for channel mixing and feature refinement.

4. The lightweight cross-view image matching and positioning method based on multi-scale feature aggregation according to claim 1 or 3, characterized in that: The local features of the image are extracted using a feature extraction network, which includes four feature extraction stages. In each feature extraction stage, the input image is first downsampled, and then the features of the input image are extracted using the feature extraction network; the feature map extracted in the last stage is used as the output, so that the output is used as the input feature of the multi-scale feature aggregation network.

5. The lightweight cross-view image matching and positioning method based on multi-scale feature aggregation according to claim 1 is characterized in that: The multi-scale feature aggregation network is constructed based on the SPE module, and the convolution kernel in the SPE module is decomposed into two consecutive layer-by-layer convolutions to extract different scale features of the image by using two consecutive layer-by-layer convolutions.

6. The lightweight cross-view image matching and positioning method based on multi-scale feature aggregation according to claim 1 or 5, characterized in that: A multi-scale feature aggregation network is used to aggregate image features from multiple scales into a global descriptor, including: Two consecutive layer-by-layer convolutions are used to extract features of different scales of the input image, wherein the convolution kernel sizes and dilation rates of the two consecutive layer-by-layer convolutions are different; The features extracted by convolution layer by layer are mixed using a convolution kernel of size 1×1, while compressing the number of feature channels; The spliced ​​image features are averaged and maximum pooled in turn along the channel direction, and the pooling results are projected using two fully connected layers to obtain a position encoding map of the image features; The position coding map and the feature map on each channel of the image are used to perform matrix inner product calculation, and the calculation results are concatenated to obtain the feature descriptor of the image.

7. The lightweight cross-view image matching and positioning method based on multi-scale feature aggregation according to claim 1 is characterized in that: The training process of the cross-view image matching network model includes: Acquire positive sample data and negative sample data for model training, wherein the positive sample data includes street view images and satellite images corresponding to the street view images, and the negative sample data includes street view images and satellite images with mismatched regional locations; The model training loss function is constructed based on the cosine similarity between the street view image and the satellite image in the positive sample data and the negative sample data; Based on the model training loss function, the cross-view image matching network model is trained and learned using positive sample data and negative sample data, so that the trained cross-view image matching network model can learn the feature representation space between the query image and the reference image.

8. A lightweight cross-view image matching and positioning system based on multi-scale feature aggregation, characterized in that: It includes: air view image database module, image feature extraction module and cross-view image matching module, among which: The aerial image database module is used to collect satellite images with geographic location information in the target area, slice and transform the satellite images into polar coordinates, and build a reference image database; An image feature extraction module, which is used to extract global feature descriptors of ground-view query images and satellite images using a pre-trained cross-view image matching network; A cross-view image matching module is used to match the feature descriptors of the query image with the feature descriptors of the satellite image to determine the geographic location of the ground-view query image; Among them, the cross-view image matching network model is constructed using a twin neural network architecture, which includes a feature extraction network for extracting local features of the image and a multi-scale feature aggregation network for aggregating image features from multiple scales into global descriptors.

Images