A method and system for visual positioning of aerial images by a drone

Abstract

The application relates to a UAV aerial image visual positioning method and system, and belongs to the technical field of image processing.The method comprises the following steps: performing feature extraction on an aerial image and a natural language text instruction corresponding to the aerial image based on a double-flow feature extraction module; performing alignment on multi-scale visual features and text features based on a cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features; decoding the cross-modal visual features and the aligned text features based on a semantic-guided target query decoder; fusing a predicted bounding box and target query features based on a spatial relationship adaptive module to generate a comprehensive relationship graph; and reasoning the comprehensive relationship graph based on the aligned text features to obtain a final target bounding box of the UAV aerial image, thereby improving the accuracy of UAV aerial image visual positioning.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a visual positioning method and system for aerial images taken by unmanned aerial vehicles. Background Technology

[0002] Visual positioning, as one of the core technologies of the UAV's embodied intelligent perception system, is an important method to solve the problem of intelligent systems accurately identifying and locking onto targets based on natural language instructions in complex open environments. Visual positioning utilizes cross-modal learning algorithms to eliminate semantic differences between different modalities and establish fine-grained mapping relationships at the bounding box or pixel level between unstructured text descriptions and specific entities in images.

[0003] The main challenge of cross-modal visual localization is how to achieve efficient alignment and accurate spatial reasoning between visual representations and natural language semantics. Most existing mainstream visual localization methods have made significant progress in natural image scenes with conventional perspectives, but drone aerial photography scenes face objective challenges such as drastic scale changes caused by flight altitude and high-density clustering of similar targets under large field of view. Existing cross-modal fusion methods generally rely too heavily on global flattening mapping, roughly flattening and interacting multi-scale features directly. This can easily lead to the dilution of fine-grained features of extremely small targets by high-order abstract semantics in deep networks, causing semantic shifts and alignment deviations. In addition, existing multimodal models mostly rely solely on data-driven implicit appearance feature matching, lacking explicit modeling of the spatial topology between target entities. This makes it easy to cause instance-level localization ambiguity and matching failure when facing dense target clusters with highly similar visual appearances, resulting in low accuracy of visual localization in drone aerial images. Summary of the Invention

[0004] In view of this, it is necessary to provide a visual positioning method and system for UAV aerial images to solve the technical problem of low accuracy in visual positioning of UAV aerial images.

[0005] To address the aforementioned issues, in a first aspect, the present invention provides a visual positioning method for UAV aerial images, which is used to perform visual positioning of aerial images to be positioned based on a constructed aerial image visual positioning model. The aerial image visual positioning model includes a dual-stream feature extraction module, a cross-scale bidirectional collaborative feature alignment module, a semantically guided target query decoder, and a spatial relationship adaptive module. The visual positioning method for drone aerial images includes: Based on the dual-stream feature extraction module, feature extraction is performed on the aerial image and the natural language text command corresponding to the aerial image to obtain multi-scale visual features and text features; The multi-scale visual features and text features are aligned based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features. The semantically guided target query decoder decodes the cross-modal visual features and aligned text features to obtain the predicted bounding boxes of candidate targets and the target query features corresponding to the predicted bounding boxes. The spatial relationship adaptive module fuses the predicted bounding box and target query features to generate a comprehensive relationship graph. Based on the aligned text features, the comprehensive relationship graph is inferred to obtain the final target bounding box of the UAV aerial image.

[0006] In one possible implementation, the dual-stream feature extraction module includes a visual backbone network and a text backbone network. The step of extracting features from the aerial image and the corresponding natural language text instructions based on the dual-stream feature extraction module to obtain multi-scale visual and text features includes: Based on the visual backbone network, the aerial image is segmented and hierarchical features are extracted to obtain multi-scale visual features. The natural language text instructions are serialized and encoded based on the text backbone network to obtain text features.

[0007] In one possible implementation, the alignment of the multi-scale visual features and text features based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features includes: The multi-scale visual features are dynamically segmented to obtain multiple scale feature sequences; Multiple visual query vectors are obtained by mapping the multiple scale feature sequences based on a fully connected layer; The text features are mapped based on a fully connected layer to obtain the key vector and value vector of the text features. Cross-attention calculation is performed on the multiple visual query vectors, the key vector and value vector of the text features to generate multiple enhanced visual sequences. The multiple enhanced visual sequences are concatenated and combined to obtain preliminary cross-modal aligned visual features. Text query vectors are obtained by mapping text features using fully connected layers; The multi-scale visual features are mapped using a fully connected layer to obtain visual key vectors and visual value vectors. The text query vectors and visual key vectors are then processed using a multi-head attention mechanism to obtain an initial global image-text similarity matrix. The initial global image-text similarity matrix is ​​dynamically segmented and flattened to obtain multiple two-dimensional hierarchical similarity matrices; The multiple two-dimensional hierarchical similarity matrices are dynamically updated, and the updated multiple two-dimensional hierarchical similarity matrices are concatenated and combined to obtain a global collaborative update similarity matrix; The global collaborative update similarity matrix is ​​processed by negative infinity masking using a visual mask matrix. The global collaborative update similarity matrix after negative infinity masking is then normalized to obtain the cross-scale image-text attention weight distribution. The visual value vector is then weighted and summed based on the cross-scale image-text attention weight distribution to obtain the aligned text features. Based on multilayer perceptron, the global context features of text features are reduced in dimensionality and nonlinearly mapped to generate modulation vectors; The visual features of the initial cross-modal alignment are normalized to obtain the query vector; Multi-head cross-attention computation is performed on the modulation vector and the query vector to obtain enhanced visual features that fuse global text semantics. The enhanced visual features that fuse global text semantics are then added to the visual features that are initially aligned across modalities by residual addition to obtain aligned cross-modal visual features.

[0008] In one possible implementation, the semantically guided target query decoder decodes the cross-modal visual features and aligned text features to obtain predicted bounding boxes of candidate targets and target query features corresponding to the predicted bounding boxes, including: Based on the aligned cross-modal visual features and aligned text features, a dot product similarity matrix is ​​determined, and a max pooling operation is performed on the dot product similarity matrix to obtain a global semantic response score. The candidate anchor set and the spatial location of each candidate anchor in the candidate anchor set are determined based on the global semantic response score. The initial predicted bounding box of the candidate target is determined based on the spatial location of each candidate anchor point and the width and height of the preset initial anchor point. The initial target query features are determined based on the spatial location of each candidate anchor point and the aligned cross-modal visual features. The initial predicted bounding box and the initial target query features are interacted to generate the initial target query vector; The initial target query vector is sequentially cross-attention calculated with the aligned cross-modal visual features and the aligned text features to obtain the target query features of the candidate target and the bounding box residual prediction. The predicted bounding box of the candidate target is determined based on the target query features of the candidate target and the predicted bounding box residual.

[0009] In one possible implementation, the step of fusing the predicted bounding box and target query features based on the spatial relationship adaptive module to generate a comprehensive relationship graph includes: Construct a spatial relationship adjacency matrix based on the predicted bounding boxes of the candidate targets; Construct a semantic relation adjacency matrix based on the target query features of the candidate targets; The semantic relationship adjacency matrix and the spatial relationship adjacency matrix are fused to generate a comprehensive relationship graph.

[0010] In one possible implementation, constructing a spatial relation adjacency matrix based on the predicted bounding boxes of the candidate targets includes: Adaptive cross-scale normalization and logarithmic nonlinear transformation are applied to the predicted bounding boxes of the candidate targets to obtain the relative spatial geometric features between the candidate targets. The relative spatial geometric features are mapped using a high-frequency Fourier position coding function, and the mapped relative spatial geometric features are nonlinearly mapped and dimension aligned based on a multilayer perceptron to obtain a spatial relationship adjacency matrix.

[0011] In one possible implementation, constructing a semantic relation adjacency matrix based on the target query features of the candidate target includes: The feature cosine similarity of the target query features of the candidate targets is calculated to construct a semantic relationship adjacency matrix.

[0012] In one possible implementation, the fusion of the semantic relation adjacency matrix and the spatial relation adjacency matrix to generate a comprehensive relation graph includes: The semantic relationship adjacency matrix and the spatial relationship adjacency matrix are concatenated and combined along the feature channel dimension. The concatenated and combined matrix is ​​then input into an adaptive gating network to generate dynamic gating routing weights. Based on the dynamic gated routing weights, the semantic relationship adjacency matrix and the spatial relationship adjacency matrix are adaptively weighted and fused to generate a comprehensive relationship graph.

[0013] In one possible implementation, the step of reasoning about the comprehensive relationship graph based on the aligned text features to obtain the final target bounding box of the UAV aerial image includes: Using the comprehensive relationship graph as the query vector and the aligned text features as the key vector and value vector, cross-attention calculation is performed on the query vector, key vector, and value vector to obtain the comprehensive relationship graph of the aggregated and aligned text features; Self-attention calculation is performed on the comprehensive relation graph of the aggregated and aligned text features to obtain the updated relation graph features; The updated relation graph features are pooled and aggregated along different matrix dimensions to obtain feature residual increments; The feature residual increment is added to the residual of the target query feature of the candidate target to obtain the calibrated target query feature; The calibrated target query features are input into the linear projection layer to calculate the matching score of the candidate targets. Based on the matching score, the candidate targets are filtered from their predicted bounding boxes to obtain the final target bounding boxes of the UAV aerial images.

[0014] Secondly, the present invention also provides a visual positioning system for UAV aerial images, which is used to perform visual positioning of aerial images to be positioned based on a constructed aerial image visual positioning model. The aerial image visual positioning model includes a dual-stream feature extraction module, a cross-scale bidirectional collaborative feature alignment module, a semantically guided target query decoder, and a spatial relationship adaptive module. The UAV aerial image visual positioning system includes: The feature extraction module is used to extract features from the aerial image and the natural language text command corresponding to the aerial image based on the dual-stream feature extraction module, so as to obtain multi-scale visual features and text features; The feature alignment module is used to align the multi-scale visual features and text features based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features. The feature decoding module is used to decode the cross-modal visual features and aligned text features based on the semantically guided target query decoder to obtain the predicted bounding box of the candidate target and the target query feature corresponding to the predicted bounding box. The target bounding box determination module is used to fuse the predicted bounding box and target query features based on the spatial relationship adaptive module to generate a comprehensive relationship graph, and to infer the comprehensive relationship graph based on the aligned text features to obtain the final target bounding box of the UAV aerial image.

[0015] The beneficial effects of this invention are as follows: A dual-stream feature extraction module extracts features from aerial images and corresponding natural language text commands to obtain multi-scale visual and text features; a cross-scale bidirectional collaborative feature alignment module aligns the multi-scale visual and text features to obtain aligned cross-modal visual and text features; a semantically guided target query decoder decodes the cross-modal visual and aligned text features to obtain predicted bounding boxes of candidate targets and target query features corresponding to the predicted bounding boxes; a spatial relationship adaptive module fuses the predicted bounding boxes and target query features to generate a comprehensive relationship graph; and the aligned text features are used to infer the comprehensive relationship graph to obtain the final target bounding box of the UAV aerial image. The bidirectional collaborative feature alignment module aligns multi-scale visual and textual features. Utilizing a decoupled multi-scale layer-by-layer interaction and bidirectional update mechanism, it achieves high-precision, fine-grained alignment of targets at any scale with textual instructions, providing a more reliable multi-dimensional judgment basis for target recognition in dense scenes. Through the inference of the spatial relationship adaptive module, which extracts the spatial geometric relationship between candidate targets based on the predicted bounding box and calculates the semantic similarity relationship by combining the target query features, the predicted bounding box and target query features are fused to generate a comprehensive relationship graph. Guided by the aligned textual features, graph inference is performed. By matching the confidence of the graph with the textual intent, exclusive disambiguation is achieved. Among a large number of similar interference items, the unique target is exclusively and accurately locked, improving the accuracy of visual positioning of UAV aerial images. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of an embodiment of the visual positioning method for drone aerial images provided by the present invention; Figure 2 This is a schematic diagram of the structure of the aerial image visual positioning model of the UAV aerial image visual positioning method provided by the present invention. Figure 3 This is a schematic diagram of the cross-scale bidirectional collaborative feature alignment module of the UAV aerial image visual localization method provided by the present invention. Figure 4 A schematic diagram of the spatial relationship adaptive module of the UAV aerial image visual positioning method provided by the present invention; Figure 5This is a schematic diagram of an embodiment of the UAV aerial image visual positioning system provided by the present invention. Detailed Implementation

[0018] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0019] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] A specific embodiment of the present invention discloses a visual positioning method for aerial images captured by a drone, such as... Figure 1 As shown, the UAV aerial image visual localization method is used to visually locate the aerial image to be located based on the constructed aerial image visual localization model. For a schematic diagram of the aerial image visual localization model, please refer to [link / reference needed]. Figure 2 ,like Figure 2 As shown, the aerial image visual localization model includes a two-stream feature extraction module, a cross-scale bidirectional collaborative feature alignment module, a semantically guided target query decoder, and a spatial relationship adaptation module; the UAV aerial image visual localization method includes: S101. Based on the dual-stream feature extraction module, feature extraction is performed on aerial images and corresponding natural language text commands to obtain multi-scale visual features and text features.

[0021] It should be noted that natural language text commands use spatial relationship constraints to filter out targets that meet the conditions from the massive number of pixels in aerial images. For example, if a drone aerial image is an orthophoto of a city street, and the image contains vehicles and street signs, the natural language text command could be to identify a white car located in the zebra crossing area of ​​the image.

[0022] S102. Align multi-scale visual features and text features based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features.

[0023] S103. The semantically guided target query decoder decodes the cross-modal visual features and the aligned text features to obtain the predicted bounding boxes of the candidate targets and the target query features corresponding to the predicted bounding boxes.

[0024] S104. The spatial relationship adaptive module fuses the predicted bounding box and target query features to generate a comprehensive relationship graph. Based on the aligned text features, the comprehensive relationship graph is inferred to obtain the final target bounding box of the UAV aerial image.

[0025] In some embodiments, the collected drone aerial images and natural language text commands are input to a PC. The aerial image visual positioning model on the PC performs feature extraction, feature alignment, decoding, and inference on the drone aerial images and natural language text commands, and outputs the final target bounding box of the drone aerial image.

[0026] In some embodiments, in step S101, feature extraction is performed on the aerial images and the corresponding natural language text commands based on the dual-stream feature extraction module to obtain multi-scale visual features and text features. The aerial images to be located and the corresponding natural language text commands are acquired. Drone aerial images and their corresponding natural language text commands are collected, and a cross-modal visual positioning dataset for the drone's perspective is constructed. This cross-modal visual positioning dataset is then input into the dual-stream feature extraction module, which includes a visual backbone network (Swin-Transformer) and a text backbone network (BERT). Based on the visual backbone network, the aerial images to be located are segmented and hierarchically feature-extracted to obtain multi-scale visual features. Based on a text backbone network, natural language text instructions are serialized and encoded to obtain text features. The dual-stream feature extraction module uses a visual backbone network to segment and extract hierarchical features from the input image, constructing a multi-resolution visual feature pyramid as multi-scale visual features. At the same time, it uses a text backbone network to serialize and encode natural language instructions, extracting text features.

[0027] In some embodiments, in step S102, multi-scale visual features and text features are aligned based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features. The cross-scale bidirectional collaborative feature alignment module uses a hierarchical cross-attention mechanism of cross-modal interaction and bidirectional sampling of multi-scale independent visual features to align multi-scale visual features and text features, and macro-modulates the visual feature pool by introducing global context features extracted from the first and second parts of the text feature sequence to obtain the feature-aligned cross-modal visual features. and text features ; For a schematic diagram of its cross-scale bidirectional collaborative feature alignment module, please refer to [link / reference]. Figure 3 ,like Figure 3As shown, the multi-scale independent visual feature cross-modal interaction process involves dynamically segmenting multi-scale visual features to obtain multiple scale feature sequences. A bidirectional sampling hierarchical cross-attention mechanism is used to calculate cross-attention, which includes a fully connected layer. Based on the fully connected layer, multiple scale feature sequences are mapped to obtain multiple visual query vectors. Based on the fully connected layer, text features are mapped to obtain key and value vectors. Cross-attention is then calculated on the multiple visual query vectors, text feature key and value vectors to generate multiple enhanced visual sequences. These enhanced visual sequences are then concatenated to obtain preliminary cross-modal aligned visual features. Finally, the multi-scale visual features are dynamically segmented along the sequence dimension according to the spatial resolution size at each level, decoupling and restoring them to their original state. A set of mutually independent scale feature sequences, denoted as ,in For each independent scale feature sequence A fully connected layer maps the scale feature sequence to a query vector. At the same time, text features Each vector is mapped to a key vector through a fully connected layer. AND value vector Then, visual-language cross-attention calculation is performed to generate enhanced visual sequences after independent interactions at each level. Its enhanced visual sequence is: , in, The scaling dimension of the feature channels. This is the transpose of the key vector; Enhance visual sequences at various scales after independent interaction The features are collected into a feature list and then reassembled along the sequence dimension according to the original multi-scale spatial resolution order to obtain visual features that have completed the initial cross-modal alignment. .

[0028] Its bidirectional sampling hierarchical cross-attention mechanism is as follows: First, text features are mapped using a fully connected layer to obtain a text query vector. Second, multi-scale visual features are mapped using a fully connected layer to obtain visual key vectors and visual value vectors. Third, the text query vector and visual key vector are processed using a multi-head attention mechanism to obtain an initial global image-text similarity matrix. Fourth, the initial global image-text similarity matrix is ​​dynamically segmented and flattened to obtain multiple two-dimensional hierarchical similarity matrices. Fifth, these multiple two-dimensional hierarchical similarity matrices are dynamically updated, and the updated matrices are concatenated to obtain a global collaborative update similarity matrix. Sixth, the global collaborative update similarity matrix is ​​masked with a negative infinity mask using a visual mask matrix. The resulting global collaborative update similarity matrix is ​​then normalized to obtain a cross-scale image-text attention weight distribution. Finally, the visual value vectors are weighted and summed based on this cross-scale image-text attention weight distribution to obtain aligned text features. Specifically, the text features are... The query vector is obtained through mapping in the fully connected layer. At the same time, multi-scale visual features Key vectors are obtained through mapping using fully connected layers. AND value vector The initial global image-text similarity matrix is ​​obtained through a multi-head attention mechanism. Its initial global image-text similarity matrix is: , By leveraging the spatial resolution of multi-scale visual features at various levels, the initial global image-text similarity matrix is ​​dynamically segmented and inversely flattened along the visual sequence dimension, reconstructing it into... A set of independent two-dimensional hierarchical similarity matrices Among them, the text sequence dimension is used as a feature channel for the first... Two-dimensional similarity matrix at each level Extract the similarity matrix from the previous level. The top-down residual update amount is calculated by upsampling and aligning to the spatial size of the current level using bilinear interpolation. Its top-down residual update amount is: , in, To transform the low-resolution two-dimensional similarity matrix of the previous level After upsampling, the spatial size is the same as the first... Hierarchical aligned feature matrix; Update control parameters based on learnable similarity; simultaneously extract the next level of similarity matrix. The residual update is calculated by sequentially passing the downsampling convolutional layer and bilinear interpolation to align to the spatial size of the current layer, and then calculating the bottom-up residual update. The bottom-up residual update amount is: , in, To generate a high-resolution two-dimensional similarity matrix for the next level After downsampling, the spatial size is the same as the first... Hierarchical aligned feature matrix Update control parameters for learnable similarity; After averaging the top-down and bottom-up residual updates, the result is compared with the two-dimensional similarity matrix of the current level. By adding element by element, we obtain the dynamically updated hierarchical similarity matrix. The calculation formula is as follows: , All updated two-dimensional hierarchical similarity matrices The data is flattened again and pieced together along the visual sequence dimension to reconstruct a globally collaboratively updated similarity matrix. The visual mask matrix is ​​used to collaboratively update the global similarity matrix. The positions of invalid visual filler words in the image are masked with negative infinity, and then normalized to obtain the final cross-scale image-text attention weight distribution. This weight distribution is then used to apply to the visual value vector. We perform weighted summation calculations and output enhanced text features after cross-scale bidirectional relational collaboration through residual connections. That is, the aligned text features.

[0029] Its macro-modulation involves: performing dimensionality reduction and nonlinear mapping on the global context features of the text features based on a multilayer perceptron to generate a modulation vector; normalizing the visual features obtained from the initial cross-modal alignment to obtain a query vector; performing multi-head cross-attention computation on the modulation vector and the query vector to obtain enhanced visual features that fuse global text semantics; and adding the residuals of the enhanced visual features that fuse global text semantics with the visual features obtained from the initial cross-modal alignment to obtain the aligned cross-modal visual features. Specifically, the feature vector corresponding to the first position of the sequence of original input text features is extracted as the global context feature representing the macro-semantic intent of the text. global context features The input is fed into a multilayer perceptron (MLP) consisting of fully connected layers and activation functions for dimensionality reduction and nonlinear mapping to generate a modulation vector. Its modulation vector is: , The visual features initially aligned across modalities are processed through layer normalization and used as the query vector, along with the modulation vector. Performing multi-head cross-attention computation prompts visual features to adaptively aggregate global text semantics, aligning the output features of the attention mechanism with the visual features initially aligned across modalities. The residuals are summed and a second layer normalization is performed to obtain the final global semantically modulated visual features, i.e., the aligned cross-modal visual features.

[0030] In some embodiments, in step S103, a semantically guided target query decoder decodes cross-modal visual features and aligned text features to obtain predicted bounding boxes of candidate targets and target query features corresponding to the predicted bounding boxes. A dot-product similarity matrix is ​​determined based on the aligned cross-modal visual features and aligned text features. Max-pooling is then performed on the dot-product similarity matrix to obtain a global semantic response score. A set of candidate anchor points and the spatial positions of each candidate anchor point in the set are determined based on the global semantic response score. Finally, candidate targets are determined based on the spatial positions of each candidate anchor point and the width and height of a preset initial anchor point. The process involves: generating initial predicted bounding boxes; determining initial target query features based on the spatial locations of each candidate anchor point and aligned cross-modal visual features; performing feature interaction on the initial predicted bounding boxes and initial target query features to generate initial target query vectors; sequentially performing cross-attention calculations on the initial target query vectors with aligned cross-modal visual features and aligned text features to obtain target query features of candidate targets and bounding box residual predictions; determining the predicted bounding boxes of candidate targets based on the target query features and bounding box residual predictions, and the semantically guided target query decoder outputting the predicted bounding boxes of the Top-K candidate targets. and the target query features corresponding to the predicted bounding box The process involves aligning cross-modal visual features. Text features After linear transformation, the image is projected into the same feature dimension space. Then, matrix multiplication is used to calculate the dot product similarity matrix between each local region in the visual space and the intent of the input text. Its dot product similarity matrix is: , in, and These are the linear projection weight matrices for visual and textual data, respectively. To transpose the matrix, use the dot product similarity matrix. Max pooling is performed along the text sequence dimension to extract each spatial location. Global semantic response score Its global semantic response score is: , in, The length of the text sequence. For the first The visual spatial position and the first Semantic similarity between text terms; Global semantic response score for all spatial locations Sort the local feature points in descending order and select the top K local feature points with the highest scores as the candidate anchor point set. Extract the normalized spatial center coordinates corresponding to each candidate anchor point. And by combining the inverse Sigmoid function, the normalized spatial center coordinates corresponding to each candidate anchor point are initialized to the first... Initial predicted bounding boxes for candidate targets Its initial predicted bounding box is: , , in, This is the inverse sigmoid function, used to map normalized coordinates to an unbounded real space to facilitate subsequent regression optimization. and These are the preset initial anchor point width and height parameters; Based on the spatial location of the candidate anchor points From cross-modal visual features The high-dimensional visual features corresponding to the index are extracted, and then the high-dimensional visual features are added element-wise with the two-dimensional spatial location codes corresponding to the spatial locations to generate the initial target query features at layer 0, which contain both spatial and semantic priors. Its initial target query features are: , in, For sine and cosine spatial position encoding functions; The initial predicted bounding box and the initial target query features are input into a cascaded decoder containing M layers. In the m-th layer of the decoder, the target query features, after undergoing feature interaction via a self-attention mechanism, are used as the query vector to sequentially perform cross-attention calculations with the aligned cross-modal visual features and the aligned text features, outputting the updated features of the current layer. With bounding box residual prediction The calculation formula is as follows: , , After M rounds of iterative updates, the refined target query features are output. Predicting quantities using bounding box residuals The bounding box is iteratively updated, and its calculation formula is: , The predicted bounding box of the candidate target is: .

[0031] In some embodiments, in step S104, the predicted bounding box and target query features are fused based on the spatial relationship adaptive module to generate a comprehensive relationship graph; a spatial relationship adjacency matrix is ​​constructed based on the predicted bounding boxes of candidate targets; a semantic relationship adjacency matrix is ​​constructed based on the target query features of candidate targets; the semantic relationship adjacency matrix and the spatial relationship adjacency matrix are fused to generate a comprehensive relationship graph; and reasoning is performed on the comprehensive relationship graph based on the aligned text features to obtain the final target bounding box of the UAV aerial image; for a schematic diagram of the spatial relationship adaptive module, please refer to [link to schematic diagram]. Figure 4 ,like Figure 4 As shown, the spatial relationship adaptive module extracts the spatial geometric relationships between candidate targets based on the predicted bounding boxes, and calculates the semantic similarity relationship by combining the target query features. The two are then fused to construct a comprehensive relationship graph. Guided by the aligned text features, graph inference is performed to output the final target bounding boxes. Specifically, the spatial relationship adaptive module constructs a spatial relationship adjacency matrix using the candidate target predicted bounding boxes and a semantic relationship adjacency matrix using the target query features. Through a dual-domain adaptive gating routing mechanism, the contribution of space and semantics is dynamically weighed and fused to generate a comprehensive relationship graph. Finally, by constructing a text-guided graph inference mechanism, the final matching confidence of each graph node with the text intent is calculated to complete the exclusion disambiguation and output the final target bounding boxes. The spatial adjacency matrix is ​​constructed as follows: Adaptive cross-scale normalization and logarithmic nonlinear transformation are applied to the predicted bounding boxes of candidate targets to obtain the relative spatial geometric features between them. A high-frequency Fourier position coding function is used to map these relative spatial geometric features, and a multilayer perceptron is used to perform dimensionality reduction mapping on the mapped relative spatial geometric features to obtain the spatial adjacency matrix. Specifically, adaptive cross-scale normalization and logarithmic nonlinear transformation are applied to the predicted bounding boxes of the Top-K candidate targets to extract the relative spatial geometric features between them. Its relative spatial geometric characteristics are: , , in, For the first The center coordinates, width, and height of the predicted bounding box. For the first The center coordinates, width, and height of the predicted bounding box; The low-dimensional continuous relative spatial geometric features are mapped to a high-dimensional spatial representation using a high-frequency Fourier position coding function. The mapped relative spatial geometric features are then input into a multilayer perceptron for dimensionality reduction mapping, and the spatial relation adjacency matrix is ​​output. Its spatial adjacency matrix is: , in, This is a high-frequency Fourier position coding function.

[0032] The semantic relation adjacency matrix is ​​constructed by calculating the feature cosine similarity of the target query features of candidate nodes. Specifically, for the Top-K target query features, the feature cosine similarity between each candidate node pair is calculated, thereby constructing the semantic relation adjacency matrix. Its semantic relation adjacency matrix is: , A comprehensive relation graph is generated through a dual-domain adaptive gating routing mechanism. The semantic relation adjacency matrix and the spatial relation adjacency matrix are concatenated along the feature channel dimension. The concatenated matrix is ​​then input into an adaptive gating network to generate dynamic gating routing weights. Based on these dynamic gating routing weights, the semantic relation adjacency matrix and the spatial relation adjacency matrix are adaptively weighted and fused to generate the comprehensive relation graph. Specifically, the spatial relation adjacency matrix and the semantic relation adjacency matrix are concatenated along the feature channel dimension and input into the adaptive gating network to generate a graph constrained to... Dynamic gating routing weights within the interval Its dynamic gating routing weight is: , in, It is the Sigmoid activation function. For the weight matrix of the gated network, For feature splicing operations; By using dynamic gated routing weights to adaptively weight and fuse the two sets of heterogeneous matrices, a comprehensive relational adjacency matrix of the full-dimensional heterogeneous collaborative graph is generated. This comprehensive relational adjacency matrix is ​​the comprehensive relational graph, which is as follows: , Based on the aligned text features, inference is performed on the comprehensive relationship graph to obtain the final target bounding box of the UAV aerial image. Using the comprehensive relationship graph as the query vector and the aligned text features as the key and value vectors, cross-attention is performed on the query vector, key vector, and value vector to obtain the comprehensive relationship graph aggregating the aligned text features. Self-attention is then performed on the comprehensive relationship graph aggregating the aligned text features to obtain updated relationship graph features. The updated relationship graph features are then pooled along different matrix dimensions to obtain feature residual increments. These feature residual increments are added to the residuals of the candidate target query features to obtain calibrated target query features. The calibrated target query features are then input into a linear projection layer to calculate the matching score of the candidate targets. Based on the matching score, the candidate targets are filtered from their predicted bounding boxes to obtain the final target bounding box of the UAV aerial image. Specifically, the comprehensive relationship graph is used as the query vector, and the aligned text features are used as the key and value vectors. Using these as key and value vectors respectively, cross-attention computation is performed to achieve final semantic alignment of the relation graph under the guidance of global textual intent. Then, self-attention computation is performed to complete the exclusive contextual feature interaction between nodes within the graph, outputting the updated relation matrix. The updated relation matrix Feature residual increments are generated through matrix pooling aggregation. Increment the characteristic residual Target query features injected into candidate targets Obtain calibration features This calibration feature After linear projection mapping, the final matching confidence score between each node and the text intent is output. Extract the index of the node with the highest score. The predicted bounding box corresponding to the node index with the highest score is the final target bounding box after disambiguation, which is the coordinate of the final target bounding box of the UAV aerial image.

[0033] During the training of the aerial image visual localization model, sample data from the cross-modal visual localization dataset is input into the model for multiple rounds of iterative training. The total loss function value is calculated, and the network weights are optimized through backpropagation to obtain the trained aerial image visual localization model. The model's total loss function... Including the loss function used for bounding box regression Loss function used for object classification and the relationship loss function used to predict interactions between objects. The total loss function consists of three parts: , Among them, the bounding box regression loss function use Distance loss And generalized intersection and comparison of losses The bounding box regression loss function is calculated by combining the following factors: , in, and They are respectively The weighting coefficients for the loss and the generalized intersection-union loss. Distance loss for: , in, This represents the total number of positive sample pairs. and The first The generalized intersection and union loss is calculated by comparing the ground truth bounding box coordinates with the predicted bounding box coordinates of a positive sample. for: , in, and These represent the intersection area and the union area of ​​the true bounding box and the predicted bounding box, respectively. The minimum convex closure area that can simultaneously and completely contain both the true bounding box and the predicted bounding box; Object classification loss function The classification task is optimized by calculating the cross-modal alignment contrast loss between the predicted visual features of objects and the text features of language tags. The object classification loss function is as follows: , in, This represents the semantic alignment similarity score between visual features and textual features. The Sigmoid activation function is used to map the score to... interval; Indicates the first The predicted object and the first The label indicating whether a language lexical unit actually matches the tag. and These represent the total number of predicted objects and language words, respectively. Relationship loss function A loss function in the form of cross-entropy is used to maximize the score corresponding to the true relationship position in the relationship matrix, in order to model the interaction between object pairs. First, the Hungarian algorithm is used for label assignment to determine the corresponding target object. With auxiliary objects Its relational loss function for: , in, for The first in the relation matrix The object and the first The predicted log odds of the relationship between two objects represent the actual relationship between them. This is used to exponentially sum the relation scores of potential objects in the relation matrix, which is then used for global normalization.

[0034] In summary, the UAV aerial image visual localization method provided by this invention performs visual localization of the aerial image to be localized based on a constructed aerial image visual localization model. The aerial image visual localization model includes a dual-stream feature extraction module, a cross-scale bidirectional collaborative feature alignment module, a semantically guided target query decoder, and a spatial relationship adaptation module. The dual-stream feature extraction module extracts features from the aerial image and the corresponding natural language text commands to obtain multi-scale visual and text features. The cross-scale bidirectional collaborative feature alignment module aligns the multi-scale visual and text features to obtain aligned cross-modal visual and text features. The semantically guided target query decoder decodes the cross-modal visual and aligned text features to obtain predicted bounding boxes of candidate targets and target query features corresponding to the predicted bounding boxes. The spatial relationship adaptation module fuses the predicted bounding boxes and target query features to generate a comprehensive relationship graph. Based on the aligned text features, the comprehensive relationship graph is inferred to obtain the final target bounding box of the UAV aerial image, thus improving the accuracy of UAV aerial image visual localization.

[0035] To better implement the UAV aerial image visual positioning method in this embodiment of the invention, based on the UAV aerial image visual positioning method, correspondingly, as follows: Figure 5 As shown, this embodiment of the invention also provides a UAV aerial image visual positioning system 500. The UAV aerial image visual positioning system 500 is used to perform visual positioning of the aerial image to be positioned based on a constructed aerial image visual positioning model. The aerial image visual positioning model includes a dual-stream feature extraction module, a cross-scale bidirectional collaborative feature alignment module, a semantically guided target query decoder, and a spatial relationship adaptive module. The UAV aerial image visual positioning system 500 includes: The feature extraction module 501 is used to extract features from aerial images and natural language text commands corresponding to aerial images based on the dual-stream feature extraction module, and obtain multi-scale visual features and text features. The feature alignment module 502 is used to align multi-scale visual features and text features based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features. The feature decoding module 503 is used to decode cross-modal visual features and aligned text features based on the semantically guided target query decoder to obtain the predicted bounding box of the candidate target and the target query feature corresponding to the predicted bounding box. The target bounding box determination module 504 is used to fuse the predicted bounding box and target query features based on the spatial relationship adaptive module to generate a comprehensive relationship graph. Based on the aligned text features, the comprehensive relationship graph is inferred to obtain the final target bounding box of the UAV aerial image.

[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual positioning method for aerial images captured by a drone, characterized in that, This is used to perform visual localization of aerial images to be localized based on a constructed aerial image visual localization model. The aerial image visual localization model includes a dual-stream feature extraction module, a cross-scale bidirectional collaborative feature alignment module, a semantically guided target query decoder, and a spatial relationship adaptive module. The visual positioning method for drone aerial images includes: Based on the dual-stream feature extraction module, feature extraction is performed on the aerial image and the natural language text command corresponding to the aerial image to obtain multi-scale visual features and text features; The multi-scale visual features and text features are aligned based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features. The semantically guided target query decoder decodes the cross-modal visual features and aligned text features to obtain the predicted bounding boxes of candidate targets and the target query features corresponding to the predicted bounding boxes. The spatial relationship adaptive module fuses the predicted bounding box and target query features to generate a comprehensive relationship graph. Based on the aligned text features, the comprehensive relationship graph is inferred to obtain the final target bounding box of the UAV aerial image.

2. The UAV aerial image visual positioning method according to claim 1, characterized in that, The dual-stream feature extraction module includes a visual backbone network and a text backbone network. The feature extraction is performed on the aerial image and the corresponding natural language text instructions based on the dual-stream feature extraction module to obtain multi-scale visual and text features, including: Based on the visual backbone network, the aerial image is segmented and hierarchical features are extracted to obtain multi-scale visual features. The natural language text instructions are serialized and encoded based on the text backbone network to obtain text features.

3. The UAV aerial image visual positioning method according to claim 2, characterized in that, The method of aligning the multi-scale visual features and text features based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features includes: The multi-scale visual features are dynamically segmented to obtain multiple scale feature sequences; Multiple visual query vectors are obtained by mapping the multiple scale feature sequences based on a fully connected layer; The text features are mapped based on a fully connected layer to obtain the key vector and value vector of the text features. Cross-attention calculation is performed on the multiple visual query vectors, the key vector and value vector of the text features to generate multiple enhanced visual sequences. The multiple enhanced visual sequences are concatenated and combined to obtain preliminary cross-modal aligned visual features. Text query vectors are obtained by mapping text features using fully connected layers; The multi-scale visual features are mapped using a fully connected layer to obtain visual key vectors and visual value vectors. The text query vectors and visual key vectors are then processed using a multi-head attention mechanism to obtain an initial global image-text similarity matrix. The initial global image-text similarity matrix is ​​dynamically segmented and flattened to obtain multiple two-dimensional hierarchical similarity matrices; The multiple two-dimensional hierarchical similarity matrices are dynamically updated, and the updated multiple two-dimensional hierarchical similarity matrices are concatenated and combined to obtain a global collaborative update similarity matrix; The global collaborative update similarity matrix is ​​processed by negative infinity masking using a visual mask matrix. The global collaborative update similarity matrix after negative infinity masking is then normalized to obtain the cross-scale image-text attention weight distribution. The visual value vector is then weighted and summed based on the cross-scale image-text attention weight distribution to obtain the aligned text features. Based on multilayer perceptron, the global context features of text features are reduced in dimensionality and nonlinearly mapped to generate modulation vectors; The visual features of the initial cross-modal alignment are normalized to obtain the query vector; Multi-head cross-attention computation is performed on the modulation vector and the query vector to obtain enhanced visual features that fuse global text semantics. The enhanced visual features that fuse global text semantics are then added to the visual features that are initially aligned across modalities by residual addition to obtain aligned cross-modal visual features.

4. The UAV aerial image visual positioning method according to claim 3, characterized in that, The semantically guided target query decoder decodes the cross-modal visual features and aligned text features to obtain the predicted bounding boxes of candidate targets and the target query features corresponding to the predicted bounding boxes, including: Based on the aligned cross-modal visual features and aligned text features, a dot product similarity matrix is ​​determined, and a max pooling operation is performed on the dot product similarity matrix to obtain a global semantic response score. The candidate anchor set and the spatial location of each candidate anchor in the candidate anchor set are determined based on the global semantic response score. The initial predicted bounding box of the candidate target is determined based on the spatial location of each candidate anchor point and the width and height of the preset initial anchor point. The initial target query features are determined based on the spatial location of each candidate anchor point and the aligned cross-modal visual features. The initial predicted bounding box and the initial target query features are interacted to generate the initial target query vector; The initial target query vector is sequentially cross-attention calculated with the aligned cross-modal visual features and the aligned text features to obtain the target query features of the candidate target and the bounding box residual prediction. The predicted bounding box of the candidate target is determined based on the target query features of the candidate target and the predicted bounding box residual.

5. The UAV aerial image visual positioning method according to claim 4, characterized in that, The step of fusing the predicted bounding box and target query features based on the spatial relationship adaptive module to generate a comprehensive relationship graph includes: Construct a spatial relationship adjacency matrix based on the predicted bounding boxes of the candidate targets; Construct a semantic relation adjacency matrix based on the target query features of the candidate targets; The semantic relationship adjacency matrix and the spatial relationship adjacency matrix are fused to generate a comprehensive relationship graph.

6. The UAV aerial image visual positioning method according to claim 5, characterized in that, The construction of the spatial relationship adjacency matrix based on the predicted bounding boxes of the candidate targets includes: Adaptive cross-scale normalization and logarithmic nonlinear transformation are applied to the predicted bounding boxes of the candidate targets to obtain the relative spatial geometric features between the candidate targets. The relative spatial geometric features are mapped using a high-frequency Fourier position coding function, and the mapped relative spatial geometric features are nonlinearly mapped and dimension aligned based on a multilayer perceptron to obtain a spatial relationship adjacency matrix.

7. The UAV aerial image visual positioning method according to claim 5, characterized in that, The construction of a semantic relation adjacency matrix based on the target query features of the candidate targets includes: The feature cosine similarity of the target query features of the candidate targets is calculated to construct a semantic relationship adjacency matrix.

8. The UAV aerial image visual positioning method according to claim 5, characterized in that, The process of fusing the semantic relationship adjacency matrix and the spatial relationship adjacency matrix to generate a comprehensive relationship graph includes: The semantic relationship adjacency matrix and the spatial relationship adjacency matrix are concatenated and combined along the feature channel dimension. The concatenated and combined matrix is ​​then input into an adaptive gating network to generate dynamic gating routing weights. Based on the dynamic gated routing weights, the semantic relationship adjacency matrix and the spatial relationship adjacency matrix are adaptively weighted and fused to generate a comprehensive relationship graph.

9. The UAV aerial image visual positioning method according to claim 5, characterized in that, The step of reasoning about the comprehensive relationship graph based on the aligned text features to obtain the final target bounding box of the UAV aerial image includes: Using the comprehensive relationship graph as the query vector and the aligned text features as the key vector and value vector, cross-attention calculation is performed on the query vector, key vector, and value vector to obtain the comprehensive relationship graph of the aggregated and aligned text features; Self-attention calculation is performed on the comprehensive relation graph of the aggregated and aligned text features to obtain the updated relation graph features; The updated relation graph features are pooled and aggregated along different matrix dimensions to obtain feature residual increments; The feature residual increment is added to the residual of the target query feature of the candidate target to obtain the calibrated target query feature; The calibrated target query features are input into the linear projection layer to calculate the matching score of the candidate targets. Based on the matching score, the candidate targets are filtered from their predicted bounding boxes to obtain the final target bounding boxes of the UAV aerial images.

10. A visual positioning system for drone aerial images, characterized in that, This is used to perform visual localization of aerial images to be localized based on a constructed aerial image visual localization model. The aerial image visual localization model includes a dual-stream feature extraction module, a cross-scale bidirectional collaborative feature alignment module, a semantically guided target query decoder, and a spatial relationship adaptive module. The UAV aerial image visual positioning system includes: The feature extraction module is used to extract features from the aerial image and the natural language text command corresponding to the aerial image based on the dual-stream feature extraction module, so as to obtain multi-scale visual features and text features; The feature alignment module is used to align the multi-scale visual features and text features based on the cross-scale bidirectional collaborative feature alignment module to obtain aligned cross-modal visual features and aligned text features. The feature decoding module is used to decode the cross-modal visual features and aligned text features based on the semantically guided target query decoder to obtain the predicted bounding box of the candidate target and the target query feature corresponding to the predicted bounding box. The target bounding box determination module is used to fuse the predicted bounding box and target query features based on the spatial relationship adaptive module to generate a comprehensive relationship graph, and to infer the comprehensive relationship graph based on the aligned text features to obtain the final target bounding box of the UAV aerial image.

Images