Unmanned aerial vehicle visual positioning method based on twin network

Abstract

The invention belongs to the technical field of unmanned aerial vehicle visual positioning. The invention provides an unmanned aerial vehicle visual positioning method based on a twin network. According to the embodiment of the invention, the dual-branch convolutional neural network sharing the weight is constructed, the features of the real-time aerial image and the digital image map are extracted, the response graph is generated by using the cross-correlation operation, and the actual geographic position of the unmanned aerial vehicle is determined according to the response value. And the positioning precision and robustness in the GNSS denial state are obviously improved. Through cutting and blocking processing of high-resolution map data, the calculation pressure of full-width map matching is reduced, and efficient resource allocation is realized. Aiming at the matching difficulty of aerial images and off-line image maps under the conditions of illumination variation, scale difference and different visual angles, the adaptive capacity of the model to a complex environment is improved by combining data enhancement and a dynamic response graph screening strategy. And calculating latitude and longitude information of the unmanned aerial vehicle by combining an optimal matching area in the response graph with a coordinate file of the digital image map and utilizing affine transformation.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of unmanned aerial vehicle visual positioning, and in particular to a unmanned aerial vehicle visual positioning method based on a twin network. Background Art

[0002] A single navigation method is difficult to meet the requirements of drones for navigation system accuracy and reliability, so combined navigation technology has gradually become the main direction of research on aircraft navigation technology. The current combined navigation method is basically a navigation method based on the inertial navigation system and supplemented by the global navigation satellite system. However, when the drone is in a complex environment or a special mission scenario, the GNSS on which it relies for navigation has poor autonomy and weak anti-interference ability, and enters the GNSS denial state. At this time, the drone navigation can only rely on its own inertial navigation system. The inertial navigation system uses airborne accelerometers, gyroscopes and other equipment to integrate the speed and angular velocity offsets to obtain the navigation information of the drone, such as the position, attitude, and speed. It has the advantages of high short-term accuracy, fast update frequency, and is not easily affected by external interference, but its measurement error will accumulate over time during the calculation and update process. At this time, the drone cannot complete the established mission and may even find it difficult to return.

[0003] With the widespread use of drones, their autonomous flight capabilities have put forward higher requirements on the accuracy and reliability of navigation systems. In this context, visual navigation technology has become one of the important means to solve the above problems. Through visual navigation, drones use their onboard visual sensors to obtain environmental information, and combine high-resolution offline image maps for image matching and location information solution, thereby achieving high-precision positioning. Traditional visual navigation algorithms face many challenges, such as lighting changes, scale differences, and different perspectives between aerial images and offline image maps, resulting in insufficient matching accuracy; at the same time, the computing resources of embedded devices are limited, and how to strike a balance between real-time performance and accuracy is also a major technical difficulty.

[0004] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0005] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the invention

[0006] The purpose of the embodiments of the present disclosure is to provide a UAV visual positioning method based on a twin network, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0007] According to an embodiment of the present disclosure, a method for visual positioning of a UAV based on a twin network is provided, the method comprising: Acquire a real-time aerial image of the drone and a reference digital image map, and pre-process the real-time aerial image and the reference digital image map to obtain an aerial image of the drone, a plurality of reference digital image map sub-blocks, and an index table corresponding to the reference digital image map sub-blocks; Inputting the drone aerial image and all the reference digital image map sub-blocks into the twin network for feature extraction to obtain a small feature map and a large feature map respectively; wherein the twin network includes a first convolutional neural network and a second convolutional neural network with the same structure and shared weights; Inputting the small feature map and the large feature map into a cross-correlation module, performing a cross-correlation operation, and obtaining a response atlas; Screening out the best matching response graph with the highest response value score in the response graph set, and searching for the reference digital image map sub-block corresponding to the best matching response graph in combination with the index table; The actual geographic coordinates of the UAV are obtained according to the response value of the best matching response map and the corresponding reference digital image map sub-block.

[0008] Furthermore, the step of acquiring a real-time aerial image of a drone and a reference digital image map, and preprocessing the real-time aerial image and the reference digital image map to obtain the drone aerial image and a plurality of reference digital image map sub-blocks includes: A Gaussian filtering method is used to remove random noise in the real-time aerial image; wherein the kernel size is 3x3 and the standard deviation σ=1; Using camera internal parameters and distortion parameters, the real-time aerial image is subjected to distortion correction to eliminate radial and tangential distortions; Adopting an adaptive histogram equalization method to enhance the local contrast of the real-time aerial image and highlight the features of the ground objects, so as to obtain the drone aerial image; According to the preset latitude and longitude range, the reference digital image map is divided into blocks by a sliding window method according to the preset grid size and grid overlap rate to obtain a plurality of reference digital image map sub-blocks; wherein the reference digital image map sub-blocks contain their corresponding geographic coordinate ranges.

[0009] Furthermore, the first convolutional neural network includes a backbone network, an SE module and a multi-scale fusion module connected in sequence; wherein the backbone network includes a 7×7 convolutional layer, a MaxPool layer, a Layer1 module, a Layer2 module, a Layer3 module and a Layer4 module connected in sequence, and the Layer1 module, the Layer2 module, the Layer3 module and the Layer4 module are all composed of multiple residual blocks.

[0010] Furthermore, the steps of inputting the drone aerial image and all the reference digital image map sub-blocks into the twin network for feature extraction to obtain a small feature map and a large feature map respectively include: The drone aerial image with a size of H×W Sequentially input into the 7×7 convolutional layer and the MaxPool layer in the first convolutional neural network, and output an initial feature map with a size of H / 4×W / 4 and 64 channels; The first feature map is sequentially input into the Layer1 module, the Layer2 module, the Layer3 module and the Layer4 module, and a first feature map with a size of H / 4×W / 4 and a number of channels of 256, a second feature map with a size of H / 8×W / 8 and a number of channels of 512, a third feature map with a size of H / 16×W / 16 and a number of channels of 1024, and a fourth feature map with a size of H / 32×W / 32 and a number of channels of 2048 are output respectively; The SE module is used to calculate the channel attention weight of the fourth feature map, and the fourth feature map is globally averaged pooled in the spatial dimension through a Squeeze operation to generate a global description vector of [1, 1, C]; The generated global description vector is input into the fully connected layer through Excitation to learn the weight of each channel; Recalibrating the channels of the fourth feature map using the weight of each channel, and enhancing or suppressing the information of a specific channel by element-by-element multiplication, so that the network can effectively capture key features; Using the multi-scale fusion module, the second feature map and the third feature map are adjusted to the same size as the fourth feature map through bilinear interpolation, and are spliced ​​in the channel dimension to form an initial fusion feature of 3584 channels; The initial fusion feature is reduced to 1024 channels through a 1×1 convolution, and the output is the small feature map with a size of H / 32×W / 32 and a channel number of 1024 channels. ; The reference digital image map sub-block Input into the second convolutional neural network to perform the same operation, and output the large feature map with a size of H / 32×W / 32 and a channel number of 1024 channels .

[0011] Furthermore, the method further comprises: During the training of the twin network, according to the small feature map And the large feature map , calculate the cosine similarity; wherein the expression of the cosine similarity is:

[0012] Among them, the range of cosine similarity is [-1,1]. The closer the cosine similarity is to 1, the more similar the small feature map and the large feature map are; The loss function is defined as:

[0013] The average loss of positive sample pairs and negative sample pairs is calculated according to the cosine similarity loss. The loss value is calculated by back-propagation of the gradient of each convolutional layer in the network, and the optimizer Adam is used to update the weight parameters of the feature extraction network.

[0014] Furthermore, the step of inputting the small feature map and the large feature map into a cross-correlation module, performing a cross-correlation operation, and obtaining a response atlas comprises: The small feature map As the convolution kernel acts on the large feature map On the other hand, the small feature map is used to perform sliding convolution on the large feature map to generate the response map; All the large feature maps are traversed to obtain the response atlas; wherein the response value represents the degree of matching between the drone aerial image and each position in the reference digital image map.

[0015] Furthermore, the expression of the matching degree is:

[0016] in, is the position in the response graph The value of indicates the similarity between the drone aerial image and the reference digital image map at that location; For small feature maps in the channel and location The value of For large feature maps in channel c and position The value of .

[0017] Furthermore, the step of screening out the best matching response graph with the highest response value score in the response graph set, and searching out the reference digital image map sub-block corresponding to the best matching response graph in combination with the index table includes: In the response graph set, the response graphs are sorted according to the response values, and the response graph with the highest response value is selected as the best matching response graph; The reference digital image map sub-block corresponding to the best matching response map is searched in the index table.

[0018] Furthermore, the step of obtaining the actual geographic coordinates of the drone according to the response value of the best matching response graph and the corresponding reference digital image map sub-block includes: According to the affine transformation parameters of the reference digital image map, the pixel coordinates in the response map are mapped to the actual geographic coordinates, and the formula is as follows:

[0019] Among them, A is the pixel resolution in the x direction, E is the pixel resolution in the y direction, D is the rotation coefficient in the x direction, B is the rotation coefficient in the y direction, C is the longitude of the upper left pixel, and F is the latitude of the upper left pixel.

[0020] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: In the embodiments of the present disclosure, through the above-mentioned UAV visual positioning method based on the twin network, on the one hand, a weight-sharing dual-branch convolutional neural network, i.e., a twin network, is constructed to extract the features of real-time aerial images and digital image maps, generate a response map using cross-correlation operations, and determine the actual geographic location of the UAV based on the response value. The positioning accuracy and robustness in the GNSS denial state are significantly improved. At the same time, by cropping and blocking high-resolution map data, the computational pressure of full-width map matching is reduced, and efficient resource allocation is achieved. On the other hand, in view of the difficulty in matching aerial images and offline image maps under different lighting changes, scale differences, and different viewing angles, data enhancement and dynamic response map screening strategies are combined to improve the model's adaptability to complex environments. By combining the optimal matching area in the response map with the coordinate file of the digital image map, the longitude and latitude information of the UAV is calculated using affine transformation, and the accuracy and reliability of positioning are further enhanced, providing a solid guarantee for autonomous navigation of the UAV. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 A step diagram showing a method for visual positioning of a UAV based on a twin network in an exemplary embodiment of the present disclosure is shown; Figure 2 A schematic diagram showing the overall structure of a twin network in an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic diagram showing the structure of a first convolutional neural network or a second convolutional neural network in an exemplary embodiment of the present disclosure is shown; Figure 4 A diagram showing a positioning result of exemplary embodiment 1 of the present disclosure is shown; Figure 5 A diagram showing positioning results of exemplary embodiment 2 of the present disclosure is shown; Figure 6 A diagram showing a positioning result of exemplary embodiment 3 of the present disclosure is shown; Figure 7 A diagram showing positioning results of exemplary embodiment 4 of the present disclosure is shown; Figure 8 A program output diagram of exemplary embodiment 1 of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0025] This example implementation provides a UAV visual positioning method based on a twin network. Figure 1 As shown in , the UAV visual positioning method based on the twin network may include: step S101 to step S105.

[0026] Step S101: obtaining a real-time aerial image of a drone and a reference digital image map, and preprocessing the real-time aerial image and the reference digital image map to obtain an aerial image of the drone, a plurality of reference digital image map sub-blocks, and an index table corresponding to the reference digital image map sub-blocks; Step S102: inputting the drone aerial image and all the reference digital image map sub-blocks into the twin network for feature extraction to obtain a small feature map and a large feature map respectively; wherein the twin network includes a first convolutional neural network and a second convolutional neural network with the same structure and shared weights; Step S103: inputting the small feature map and the large feature map into a cross-correlation module, performing a cross-correlation operation, and obtaining a response atlas; Step S104: Filter out the best matching response graph with the highest response value score in the response graph set, and query the reference digital image map sub-block corresponding to the best matching response graph in combination with the index table; Step S105: obtaining the actual geographic coordinates of the UAV according to the response value of the best matching response map and the corresponding reference digital image map sub-block.

[0027] Through the above-mentioned UAV visual positioning method based on the twin network, on the one hand, a shared-weight dual-branch convolutional neural network, namely the twin network, is constructed to extract the features of real-time aerial images and digital image maps, generate a response map using cross-correlation operations, and determine the actual geographic location of the UAV based on the response value. The positioning accuracy and robustness in the GNSS denial state are significantly improved. At the same time, by cropping and blocking high-resolution map data, the computational pressure of full-width map matching is reduced, and efficient resource allocation is achieved. On the other hand, in view of the difficulty in matching aerial images and offline image maps under different lighting changes, scale differences, and different perspectives, data enhancement and dynamic response map screening strategies are combined to improve the model's adaptability to complex environments. By combining the optimal matching area in the response map with the coordinate file of the digital image map, the longitude and latitude information of the UAV is calculated using affine transformation, which further enhances the accuracy and reliability of positioning, providing a solid guarantee for the autonomous navigation of the UAV.

[0028] Next, we will refer to Figures 1 to 7 Each step of the above-mentioned twin network-based drone visual positioning method in this example implementation is described in more detail.

[0029] In step S101, the UAV visual payload is used to collect real-time aerial images, obtain high-resolution satellite images, and pre-process them, including: using noise removal, geometric correction, image enhancement, occlusion removal and other methods to process the UAV aerial images, so as to provide high-quality input for subsequent image matching; at the same time, due to the large amount of data and wide coverage of high-resolution satellite images, it is difficult to process them directly, so map segmentation, image cropping and other methods are used to reduce storage and computing pressure and concentrate computing resources to process related areas.

[0030] More specifically, the image map data volume is huge, and it is very difficult to directly process and match the entire map. In addition, drone aerial images are easily affected by environmental factors such as lighting and wind speed, which may cause noise, distortion and other problems. The present invention pre-processes the map and drone aerial images respectively through the following method.

[0031] Map cropping and block processing: Based on the drone’s aerial photography area, a specific algorithm is used to preliminarily locate the map area related to the aerial image. Block processing is used to divide the map into several sub-blocks to improve the efficiency of the matching algorithm.

[0032] UAV aerial image preprocessing can solve the error caused by noise, geometric distortion and non-ground object occlusion.

[0033] In one embodiment, the drone collects aerial images in real time through the onboard visual sensor during the mission. To ensure image quality, the camera equipment should meet the following parameter requirements: the resolution should be at least 1920x1080 pixels to ensure that detailed information is fully captured; the frame rate should be no less than 30 frames per second to meet real-time requirements; the field of view should be above 60° to cover a larger range of objects.

[0034] A high-resolution digital image map covering the possible navigation area of ​​the UAV is obtained on a certain map, with a spatial resolution of 0.5m / pixel to ensure accurate matching of details; the WGS84 geographic coordinate system is uniformly adopted to facilitate subsequent location information solution; the GeoTIFF format is uniformly adopted to include geographic reference information.

[0035] Since drone aerial images may be affected by noise, lighting changes, geometric distortion, and occlusion, they need to be preprocessed to improve the accuracy of subsequent feature extraction and matching. The specific processing steps are as follows: The Gaussian filtering method is used to remove random noise, with a kernel size of 3x3 and a standard deviation of σ=1. The camera intrinsic parameters and distortion parameters are used to correct the image distortion and eliminate radial and tangential distortion. The adaptive histogram equalization method is used to enhance the local contrast of the image and highlight the features of the ground objects.

[0036] Since the amount of satellite image map data is huge, directly processing the full map will bring huge storage and computing pressure, so the satellite image map needs to be cropped and divided into blocks. The specific steps are as follows: According to the preset latitude and longitude range, the full map is divided into blocks with a grid size of 1000x1000 and a grid overlap rate of 0.8 using a sliding window method. After block division, each sub-block contains its corresponding geographic coordinate range.

[0037] In step S102, a convolutional neural network based on ResNet-50 (i.e., a twin network) is constructed and trained, specifically including: constructing a shared feature extraction network, and lightweight optimization of ResNet-50 to adapt to the computing power limitations of airborne embedded devices, and using the UAV_VisLoc dataset for training. Figure 2 As shown in Figure 1, it is a schematic diagram of the overall structure of the twin network. Figure 3 , which is a schematic diagram of the structure of the first convolutional neural network or the second convolutional neural network.

[0038] In order to realize the feature extraction of aerial images and satellite image maps, a pseudo-twin network structure is adopted, that is, two convolutional neural network branches with shared weights. The network is based on ResNet-50 model, and the computational complexity is reduced by optimizing the lightweight structure to adapt to the computing power limitation of the embedded platform. In the training stage, the paired data sets of the collected aerial images and the corresponding satellite images are used, combined with data enhancement techniques (such as rotation, scaling, brightness adjustment, etc.) for training. During the training process, the cross entropy loss function is used to improve the adaptability of the network to changes in illumination, scale and perspective.

[0039] More specifically, a convolutional neural network based on ResNet-50 is constructed and trained. In order to realize the feature extraction of drone aerial images and digital image maps, a two-branch convolutional neural network with shared weights, namely a pseudo-twin network, is adopted. The network structure is based on the ResNet-50 model and is lightweight and optimized. The specific structure is as follows: the two branches are designed based on ResNet-50, the last fully connected layer is removed, and only the convolutional layer part is retained. Each module contains two 3x3 convolutional layers, which are connected to the ReLU activation function and batch normalization respectively. Skip connections are added to retain the underlying features and alleviate the gradient disappearance problem. The two branches use convolutional neural networks with the same weights to ensure the consistency of feature extraction, so that the feature maps have high similarity in the same semantic space. The feature extraction network consists of a backbone network based on ResNet50 and additional modules (Squeeze-and-Excitation module and multi-scale fusion module) to extract multi-level features from the input image. First, the input image of size H×W undergoes a 7×7 convolution (Conv1) and a maximum pooling operation (BN1+ReLU+MaxPool), and the feature map size becomes 1 / 4 of the original, with 64 channels. Subsequently, the feature map passes through the Layer1, Layer2, Layer3, and Layer4 modules of ResNet in sequence, where each module consists of multiple residual blocks. The feature map size is reduced to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original, respectively, and the number of channels gradually increases to 256, 512, 1024, and 2048. After the output of Layer4, the SE module is enabled to calculate the channel attention weights through the Squeeze-and-Excitation mechanism. The network obtains the global information of each channel through a global average pooling operation, and first reduces the dimension and then increases the dimension through two fully connected layers to calculate the channel-level attention weights. These weights are used to dynamically weight the channels of the feature map, allowing the network to strengthen key features and suppress irrelevant features. Next, the multi-scale fusion module is enabled, and the feature maps of Layer2 and Layer3 are adjusted to the same size as Layer4 through bilinear interpolation, and concatenated in the channel dimension to form a fusion feature of 3584 channels. Then, a 1×1 convolution is used to reduce the dimension to 1024 channels, and the final feature map is output with a size of H / 32×W / 32. The overall network realizes the comprehensive extraction of low-level, medium-level and high-level features of the image through residual learning and multi-scale fusion. During the training process, the UAV_VisLoc dataset is used to input the positive sample pairs of aerial images and the corresponding satellite map areas. To the two branches of the twin network, after feature extraction, the output feature map is obtained ,calculate The cosine similarity of:

[0040] The cosine similarity value range is [-1, 1]. The closer the value is to 1, the more similar the feature maps are.

[0041] The loss function is defined as:

[0042] The average loss of positive and negative sample pairs is calculated based on the cosine similarity loss. The loss value is calculated through back propagation of the gradient of each convolutional layer in the network, and the weight parameters of the feature extraction network are updated using the optimizer Adam.

[0043] In step S103, the obtained UAV aerial image feature map and digital image map feature map are input into the cross-correlation layer as input, and a weighted cross-correlation operation is performed to finally generate a response atlas.

[0044] Specifically, the feature maps of the drone aerial images and digital image maps after feature extraction are input into the cross-correlation layer, and the cross-correlation operation is performed to obtain the response atlas. The trained pseudo-twin network is used to process the drone aerial images and offline image maps in real time. First, the input image is standardized and resized to adapt it to the input requirements of the network; then the features of the two images are extracted respectively through the dual-branch network to generate a deep feature map. Next, the two feature maps are input into the cross-correlation layer, the point-by-point similarity is calculated, and the response map is generated. Each value in the response map represents the degree of match between a certain area of ​​the aerial image and a certain area of ​​the digital image map.

[0045] More specifically, a response map is generated by cross-correlation calculation to calibrate the matching area between the aerial image and the digital image map. As a convolution kernel, it acts on the large feature map output by the digital image map branch The small feature map is used to perform sliding convolution on the large feature map to generate a response map. The response value at each position represents the degree of match between the drone aerial image and the specific area of ​​the map. The formula is as follows:

[0046] in: : Position in the response graph The value of indicates the similarity between the drone aerial image and the digital image map at that location; :Small feature map in channel and location The value of : Large feature map in channel c and position The value of .

[0047] In step S104, the obtained response atlas is analyzed to determine the area with the highest response value, find the digital image map sub-block, and further locate the sub-block area.

[0048] Specifically, the obtained response atlas is filtered out according to the response value to select the response map with the highest score, and the corresponding large digital image map sub-block is queried. By analyzing the generated response map, the area with the highest response value is filtered out, indicating the best matching position between the drone aerial image and the digital image map. Since the digital image map is processed by block, each response map corresponds to a map sub-block. After filtering the response value, the map sub-block corresponding to the response map is queried, and the specific location of the matching area in the sub-block is further determined.

[0049] More specifically, in the response atlas, the response values ​​are sorted, and the response atlas with the highest response value is selected as the best matching response atlas, and the corresponding digital image map sub-block is quickly found in the index table generated in the preprocessing stage.

[0050] In step S105, the coordinates of the highest response position are obtained, the pixel coordinates are converted to geographic coordinates, and then the accuracy is optimized and calibrated, and finally the longitude and latitude information is output.

[0051] Specifically, according to the pixel coordinates of the matching area in the response map, combined with the coordinate file of the digital image map, the pixel coordinates are converted into actual geographic coordinates. Specifically, the affine transformation parameters of the image map (including pixel resolution, rotation coefficient and reference point coordinates) are used to transform the coordinates of the matching points and calculate the longitude and latitude position of the drone.

[0052] More specifically, the affine transformation parameters are used to convert the response map coordinates into geographic coordinates, and finally the latitude and longitude information of the drone is output. Specifically: According to the affine transformation parameters of the digital image map, the pixel coordinates in the response map are mapped to actual geographic coordinates. The formula is as follows:

[0053] The geographic information stored in satellite images is generally expressed as six affine coefficients, namely A and E represent the pixel resolution in the x and y directions, D and B represent the rotation coefficients in the x and y directions, and C and F represent the longitude and latitude of the upper left corner pixel.

[0054] In a specific embodiment, Figures 4 to 7 The figures shown are the positioning result diagrams of Example 1, Example 2, Example 3 and Example 4 respectively.

[0055] by Figure 4 For example, Figure 8 yes Figure 4The program output graph. The maximum response value coordinates in the response graph are [297,871]. Combined with the coordinate file of the digital image map, the affine transformation parameters [108.971165, 2.682201e-06, 0.0,34.383362,0.0,-2.682195e-06] respectively represent the longitude of the upper left corner pixel, the pixel resolution in the x direction, the rotation coefficient in the x direction, the dimension of the upper left corner pixel, the pixel resolution in the y direction, and the rotation coefficient in the y direction. According to the above formula, the pixel coordinates in the response graph are mapped to the actual geographic coordinates [108.97196,34.38336].

[0056] Through the above-mentioned UAV visual positioning method based on the twin network, on the one hand, a shared-weight dual-branch convolutional neural network, namely the twin network, is constructed to extract the features of real-time aerial images and digital image maps, generate a response map using cross-correlation operations, and determine the actual geographic location of the UAV based on the response value. The positioning accuracy and robustness in the GNSS denial state are significantly improved. At the same time, by cropping and blocking high-resolution map data, the computational pressure of full-width map matching is reduced, and efficient resource allocation is achieved. On the other hand, in view of the difficulty in matching aerial images and offline image maps under different lighting changes, scale differences, and different perspectives, data enhancement and dynamic response map screening strategies are combined to improve the model's adaptability to complex environments. By combining the optimal matching area in the response map with the coordinate file of the digital image map, the longitude and latitude information of the UAV is calculated using affine transformation, which further enhances the accuracy and reliability of positioning, providing a solid guarantee for the autonomous navigation of the UAV.

[0057] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0058] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A UAV visual positioning method based on twin network, characterized in that: The method includes: Acquire a real-time aerial image of the drone and a reference digital image map, and pre-process the real-time aerial image and the reference digital image map to obtain an aerial image of the drone, a plurality of reference digital image map sub-blocks, and an index table corresponding to the reference digital image map sub-blocks; Inputting the drone aerial image and all the reference digital image map sub-blocks into the twin network for feature extraction to obtain a small feature map and a large feature map respectively; wherein the twin network includes a first convolutional neural network and a second convolutional neural network with the same structure and shared weights; Inputting the small feature map and the large feature map into a cross-correlation module, performing a cross-correlation operation, and obtaining a response atlas; Screening out the best matching response graph with the highest response value score in the response graph set, and searching for the reference digital image map sub-block corresponding to the best matching response graph in combination with the index table; The actual geographic coordinates of the UAV are obtained according to the response value of the best matching response map and the corresponding reference digital image map sub-block.

2. The UAV visual positioning method based on twin network according to claim 1 is characterized in that: The step of acquiring a real-time aerial image of a drone and a reference digital image map, and preprocessing the real-time aerial image and the reference digital image map to obtain the drone aerial image and a plurality of reference digital image map sub-blocks includes: A Gaussian filtering method is used to remove random noise in the real-time aerial image; wherein the kernel size is 3x3 and the standard deviation σ=1; Using camera internal parameters and distortion parameters, the real-time aerial image is subjected to distortion correction to eliminate radial and tangential distortions; Adopting an adaptive histogram equalization method to enhance the local contrast of the real-time aerial image and highlight the features of the ground objects, so as to obtain the drone aerial image; According to the preset latitude and longitude range, the reference digital image map is divided into blocks by a sliding window method according to the preset grid size and grid overlap rate to obtain a plurality of reference digital image map sub-blocks; wherein the reference digital image map sub-blocks contain their corresponding geographic coordinate ranges.

3. The unmanned aerial vehicle visual positioning method based on twin network according to claim 1 is characterized in that: The first convolutional neural network includes a backbone network, an SE module and a multi-scale fusion module connected in sequence; wherein the backbone network includes a 7×7 convolutional layer, a MaxPool layer, a Layer1 module, a Layer2 module, a Layer3 module and a Layer4 module connected in sequence, and the Layer1 module, the Layer2 module, the Layer3 module and the Layer4 module are all composed of multiple residual blocks.

4. The UAV visual positioning method based on twin network according to claim 3 is characterized in that: The step of inputting the drone aerial image and all the reference digital image map sub-blocks into the twin network for feature extraction to obtain a small feature map and a large feature map respectively includes: The drone aerial image with a size of H×W Sequentially input into the 7×7 convolutional layer and the MaxPool layer in the first convolutional neural network, and output an initial feature map with a size of H / 4×W / 4 and 64 channels; The first feature map is sequentially input into the Layer1 module, the Layer2 module, the Layer3 module and the Layer4 module, and a first feature map with a size of H / 4×W / 4 and a number of channels of 256, a second feature map with a size of H / 8×W / 8 and a number of channels of 512, a third feature map with a size of H / 16×W / 16 and a number of channels of 1024, and a fourth feature map with a size of H / 32×W / 32 and a number of channels of 2048 are output respectively; The SE module is used to calculate the channel attention weight of the fourth feature map, and the fourth feature map is globally averaged pooled in the spatial dimension through a Squeeze operation to generate a global description vector of [1, 1, C]; The generated global description vector is input into the fully connected layer through Excitation to learn the weight of each channel; Recalibrating the channels of the fourth feature map using the weight of each channel, and enhancing or suppressing the information of a specific channel by element-by-element multiplication, so that the network can effectively capture key features; Using the multi-scale fusion module, the second feature map and the third feature map are adjusted to the same size as the fourth feature map through bilinear interpolation, and are spliced ​​in the channel dimension to form an initial fusion feature of 3584 channels; The initial fusion feature is reduced to 1024 channels through a 1×1 convolution, and the output is the small feature map with a size of H / 32×W / 32 and a channel number of 1024 channels. ; The reference digital image map sub-block Input into the second convolutional neural network to perform the same operation, and output the large feature map with a size of H / 32×W / 32 and a channel number of 1024 channels .

5. The unmanned aerial vehicle visual positioning method based on twin network according to claim 4 is characterized in that: The method further includes: During the training of the twin network, according to the small feature map And the large feature map , calculate the cosine similarity; wherein the expression of the cosine similarity is: Among them, the range of cosine similarity is [-1,1]. The closer the cosine similarity is to 1, the more similar the small feature map and the large feature map are; The loss function is defined as: The average loss of positive sample pairs and negative sample pairs is calculated according to the cosine similarity loss. The loss value is calculated by back-propagation of the gradient of each convolutional layer in the network, and the optimizer Adam is used to update the weight parameters of the feature extraction network.

6. The unmanned aerial vehicle visual positioning method based on twin network according to claim 5 is characterized in that: The step of inputting the small feature map and the large feature map into a cross-correlation module, performing a cross-correlation operation, and obtaining a response atlas comprises: The small feature map As the convolution kernel acts on the large feature map On the other hand, the small feature map is used to perform sliding convolution on the large feature map to generate the response map; All the large feature maps are traversed to obtain the response atlas; wherein the response value represents the degree of matching between the drone aerial image and each position in the reference digital image map.

7. The UAV visual positioning method based on twin network according to claim 6 is characterized in that: The expression of the matching degree is: in, The position in the response graph The value of indicates the similarity between the drone aerial image and the reference digital image map at that location; For small feature maps in the channel and location The value of For large feature maps in channel c and position The value of .

8. The unmanned aerial vehicle visual positioning method based on twin network according to claim 1 is characterized in that: The step of selecting the best matching response graph with the highest response value score in the response graph set, and searching the reference digital image map sub-block corresponding to the best matching response graph in combination with the index table, comprises: In the response graph set, the response graphs are sorted according to the response values, and the response graph with the highest response value is selected as the best matching response graph; The reference digital image map sub-block corresponding to the best matching response map is searched in the index table.

9. The unmanned aerial vehicle visual positioning method based on twin network according to claim 1 is characterized in that: The step of obtaining the actual geographic coordinates of the drone according to the response value of the best matching response graph and the corresponding reference digital image map sub-block includes: According to the affine transformation parameters of the reference digital image map, the pixel coordinates in the response map are mapped to the actual geographic coordinates, and the formula is as follows: Among them, A is the pixel resolution in the x direction, E is the pixel resolution in the y direction, D is the rotation coefficient in the x direction, B is the rotation coefficient in the y direction, C is the longitude of the upper left pixel, and F is the latitude of the upper left pixel.

Images