Air-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching

Abstract

The present application relates to the field of image recognition, and discloses a kind of air-ground collaborative unified coarse-to-fine visual positioning method with task adaptive switching, comprising: step S1, obtain the global view image of unmanned aerial vehicle and natural language instruction, and determine the coarse-grained target candidate frame in the global view image of unmanned aerial vehicle;Step S2, determine local detail view image according to the current visual angle task switching state through task input switching module, and extract the detail scaling and positioning reference parameters of target area based on coarse-grained target candidate frame, in air-ground collaborative unified coarse-to-fine visual positioning, use two-stage cascade architecture to cooperate task input switching switch, after generating normalized rough candidate frame in the first stage, according to fine adjustment or collaborative requirement, adaptively supply local cutting image source or space-time alignment ground image source, and the second stage network shares parameters between task modes, driven by transient geometric feature Feature calibration, avoid feature space repulsion in multi-task learning.

Description

Technical Field

[0001] This invention relates to a unified coarse-to-fine visual positioning method for air-to-ground collaborative positioning with task adaptive switching, belonging to the field of image recognition technology. Background Technology

[0002] Currently, localization technology that locks spatial coordinates based on language commands is a key link in image semantic understanding. It generally adopts a cascaded network architecture, which firstly uses large field-of-view images to screen out candidate regions and capture high-resolution detailed views to improve regression accuracy. However, existing technologies assume that the command end and the acquisition end are at the same viewpoint. When facing air-ground collaborative sensing, the aerial top-down slice image has obvious geometric distortion, while the ground-side planar image is mixed with high-density background interference. The spatial distribution and morphology of the same target in heterogeneous image sources are completely different, which leads to feature manifold mismatch when the system processes heterogeneous detailed features under the same backbone network. The traditional approach leads the entire sensing system to a redundant path of stacking independent branch networks for different image sources.

[0003] To address the feature space separation caused by perspective differences, the industry has attempted to introduce mapping matrices and fully connected transformation networks to forcibly map heterogeneous features to a unified space. However, such processing logic does not deeply consider the physical transformation mechanism of cross-view image features. This not only disrupts the topological relationships of high-dimensional details but also causes network weight oscillations and introduces weight repulsion during dynamic mode switching. Furthermore, not only are there room for improvement in the physical constraints of the image acquisition platform hardware camera payload structure and deployment, but there are also fundamental flaws in software control methods and cross-view feature alignment. For example, the paper with publication number CN122024177A... A Chinese invention patent application discloses a pedestrian alignment method, system, electronic device, and storage medium. It completes cross-viewpoint representation alignment in a pure representation space through a mask prediction network. The underlying logic implicitly relies on the assumption of contextual continuity of target identity features in a relatively static field of view. When faced with extreme conditions such as large-span asymmetric perspective distortion, dynamic self-occlusion of 3D entities, and transient switching of multiple tasks in air-ground collaboration, the pure representation space mapping, which is detached from the underlying topological geometric mechanism, lacks in-situ spatial metric constraints. This causes the high-dimensional detail feature manifold to diverge in nonlinear spatial transformation, making it impossible to achieve sub-pixel-level fine coordinate regression and failing to meet the high-precision positioning convergence requirements.

[0004] Therefore, the technical problem to be solved by this invention is how to perform in-situ manifold affine transformation on the detailed visual features under heterogeneous perspectives based on the transient geometric parameters of the normalized coarse target candidate box output by the previous stage, cooperate with the topological nonlinear recombination of the channel dimension to offset perspective mismatch, and use the semantic prototype sequence containing global context and local identity clues to construct the dot product retrieval of cross-task flow, so as to guide the back-end shared prediction branch to achieve accurate coordinate convergence under multi-task adaptive switching. Summary of the Invention

[0005] To address the problems in the background art, the technical solution of the present invention is as follows: A unified coarse-to-fine visual localization method for air-to-ground collaborative localization with task adaptive switching, comprising the following steps: Step S1: Obtain the global view image of the UAV and natural language commands, and determine coarse-grained target candidate boxes in the global view image of the UAV; Step S2: Based on the current viewpoint task switching state, determine the local detail view image through the task input switching module, and extract the detail scaling and positioning reference parameters of the target area based on the coarse-grained target candidate box; Step S3: Extract detailed visual features of the local detail view image using a parameter-sharing visual encoder; when the current view task is switched to a single-view UAV visual positioning task, perform feature space geometric correction on the detailed visual features based on detail scaling and positioning reference parameters; when the current view task is switched to a cross-view UAV visual positioning task, perform cross-view feature transformation and alignment on the detailed visual features. Step S4: Perform multi-scale processing on the visual representation enhanced by text conditions in the first stage, extract region of interest features based on coarse-grained target candidate boxes, generate target candidate box decoding features, convert the target candidate box decoding features into semantic prototype sequences, input the semantic prototype sequences as key-value vectors into the fine-grained decoder, and retrieve and output the target fine-grained localization results from the transformed detailed visual features.

[0006] Preferably, step S2 includes the following sub-steps: Step S21, when switching to a single-view UAV visual positioning task, the UAV partial cropping image is determined as a local detail view image; when switching to an air-ground collaborative visual positioning task, the spatiotemporally aligned ground view image is determined as a local detail view image; Step S22, multi-scale feature sampling is performed on the local detail view image to extract the detail representation of the local area.

[0007] Preferably, step S3, which involves performing an affine transformation of the detail visual features in the feature space, includes the following sub-steps: Step S31, when the current viewpoint task switching state is a single-view UAV visual positioning task, a spatial geometric correction operator is determined in the first feature calibration operator module based on the detail scaling and positioning reference parameters; Step S32, the affine matrix multiplication transformation of the detail visual features is performed using the spatial geometric correction operator to obtain the feature space visual features.

[0008] Preferably, step S3, which involves performing cross-view feature transformation and alignment on the detailed visual features, includes the following sub-steps: Step S33, when the current view task switching state is air-ground collaborative visual positioning task, the cross-view feature transformation operator is determined in the second feature calibration operator module based on the detailed visual features; Step S34, the cross-view feature transformation operator is used to perform matrix transformation on the detailed visual features to achieve cross-view feature alignment.

[0009] Preferably, step S4, which converts the target candidate box decoding features into a semantic prototype sequence, includes the following sub-steps: Step S41, performing global average pooling on the target candidate box decoding features to extract global context features; Step S42, performing region of interest alignment and cropping on the target candidate box decoding features to extract target local identity features; Step S43, concatenating the global context features and target local identity features to generate a complete semantic prototype sequence.

[0010] Preferably, step S4, which uses the semantic prototype sequence as a key-value vector to input the refined decoder, includes the following sub-steps: step S44, mapping the semantic prototype sequence to the key vector and value vector in the cross-attention mechanism; step S45, performing a linear mapping transformation on the transformed detail visual features to obtain the query vector, using the query vector and key vector to perform matrix dot product similarity calculation, and weighted retrieval of the value vector.

[0011] Preferably, after outputting the refined target localization result in step S4, the following steps are also included: Step S5, introducing cross-view contrast alignment constraints in the two-stage prediction loop, and jointly supervising the text features of the natural language instructions and the visual features extracted in the two stages by contrast loss.

[0012] Preferably, step S5 includes the following sub-steps: Step S51, extracting the text physical parsing vector of the natural language instruction using a text encoder; Step S52, performing pooling and projection on the visual features enhanced by text conditions in the first stage and the detailed visual features in the second stage respectively to obtain global view embedding and detailed view embedding, taking the two-stage visual embeddings corresponding to the same target as positive sample pairs, taking the embeddings of other targets in the batch as negative sample pairs, calculating the symmetric InfoNCE contrastive loss value, and updating the network parameters of the trainable adapter, alignment module and task prediction module.

[0013] Preferably, step S1 includes the following sub-steps: step S11, extracting multi-scale visual feature maps of the UAV global view image using an image-text dual-stream network, and extracting text feature sequences of natural language commands; step S12, calculating a multi-head cross-modal attention weight matrix between the multi-scale visual feature maps and the text feature sequences, generating a cross-modal candidate probability map, and determining coarse-grained target candidate boxes by bounding box regression calculation in the cross-modal candidate probability map.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. In the unified coarse-to-fine visual localization of air and ground collaboration, a two-stage cascaded architecture is used in conjunction with a task input switching switch. After the normalized coarse candidate box is generated in the first stage, the local cropped image source or spatiotemporally aligned ground image source is adaptively supplied according to the fine-tuning or collaborative requirements. In the second stage, the network shares parameters between task modes and the feature calibration is driven by transient geometric features. This avoids feature space exclusion in multi-task learning and enables the top view topology layout and frontal high-resolution appearance details to be continuously integrated within the same architecture. This eliminates the localization ambiguity caused by viewpoint mismatch and thus achieves lossless switching between cross-view multi-tasks.

[0015] 2. This invention sets up a lexical generation module that integrates global average pooling and region of interest alignment operations to convert decoded features into a complete semantic prototype sequence containing global context and local identity clues. This sequence serves as the key-value vector for the cross-attention mechanism in the second stage, providing semantically consistent priors and guiding the refined decoder to establish focused retrieval within the detailed view. This maintains strong semantic coherence of the target in the data flow between different stages, suppresses the dynamic interference of densely arranged similar targets on the weights, and ensures the engineering steady state of cross-view target association. Attached Figure Description

[0016] Figure 1 This is a flowchart of the air-to-ground collaborative coarse-to-fine visual positioning method of the present invention; Figure 2 This is a schematic diagram of the refined decoder network architecture of the present invention.

[0017] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0019] A unified coarse-to-fine visual localization method for air-to-ground collaborative localization with task adaptive switching includes the following steps: Step S1: Obtain the global view image of the UAV and natural language commands, and determine coarse-grained target candidate boxes in the global view image of the UAV; Step S2: Based on the current viewpoint task switching state, determine the local detail view image through the task input switching module, and extract the detail scaling and positioning reference parameters of the target area based on the coarse-grained target candidate box; Step S3: Extract detailed visual features of the local detail view image using a parameter-sharing visual encoder; when the current view task is switched to a single-view UAV visual positioning task, perform feature space geometric correction on the detailed visual features based on detail scaling and positioning reference parameters; when the current view task is switched to a cross-view UAV visual positioning task, perform cross-view feature transformation and alignment on the detailed visual features. Step S4: Perform multi-scale processing on the visual representation enhanced by text conditions in the first stage, extract region of interest features based on coarse-grained target candidate boxes, generate target candidate box decoding features, convert the target candidate box decoding features into semantic prototype sequences, input the semantic prototype sequences as key-value vectors into the fine-grained decoder, and retrieve and output the target fine-grained localization results from the transformed detailed visual features.

[0020] Preferably, step S2 includes the following sub-steps: Step S21, when switching to a single-view UAV visual positioning task, the UAV partial cropping image is determined as a local detail view image; when switching to an air-ground collaborative visual positioning task, the spatiotemporally aligned ground view image is determined as a local detail view image; Step S22, multi-scale feature sampling is performed on the local detail view image to extract the detail representation of the local area.

[0021] Preferably, step S3, which involves performing an affine transformation of the detail visual features in the feature space, includes the following sub-steps: Step S31, when the current viewpoint task switching state is a single-view UAV visual positioning task, a spatial geometric correction operator is determined in the first feature calibration operator module based on the detail scaling and positioning reference parameters; Step S32, the affine matrix multiplication transformation of the detail visual features is performed using the spatial geometric correction operator to obtain the feature space visual features.

[0022] Preferably, step S3, which involves performing cross-view feature transformation and alignment on the detailed visual features, includes the following sub-steps: Step S33, when the current view task switching state is air-ground collaborative visual positioning task, the cross-view feature transformation operator is determined in the second feature calibration operator module based on the detailed visual features; Step S34, the cross-view feature transformation operator is used to perform matrix transformation on the detailed visual features to achieve cross-view feature alignment.

[0023] Preferably, step S4, which converts the target candidate box decoding features into a semantic prototype sequence, includes the following sub-steps: Step S41, performing global average pooling on the target candidate box decoding features to extract global context features; Step S42, performing region of interest alignment and cropping on the target candidate box decoding features to extract target local identity features; Step S43, concatenating the global context features and target local identity features to generate a complete semantic prototype sequence.

[0024] Preferably, step S4, which uses the semantic prototype sequence as a key-value vector to input the refined decoder, includes the following sub-steps: step S44, mapping the semantic prototype sequence to the key vector and value vector in the cross-attention mechanism; step S45, performing a linear mapping transformation on the transformed detail visual features to obtain the query vector, using the query vector and key vector to perform matrix dot product similarity calculation, and weighted retrieval of the value vector.

[0025] Preferably, after outputting the refined target localization result in step S4, the following steps are also included: Step S5, introducing cross-view contrast alignment constraints in the two-stage prediction loop, and jointly supervising the text features of the natural language instructions and the visual features extracted in the two stages by contrast loss.

[0026] Preferably, step S5 includes the following sub-steps: Step S51, extracting the text physical parsing vector of the natural language instruction using a text encoder; Step S52, performing pooling and projection on the visual features enhanced by text conditions in the first stage and the detailed visual features in the second stage respectively to obtain global view embedding and detailed view embedding, taking the two-stage visual embeddings corresponding to the same target as positive sample pairs, taking the embeddings of other targets in the batch as negative sample pairs, calculating the symmetric InfoNCE contrastive loss value, and updating the network parameters of the trainable adapter, alignment module and task prediction module.

[0027] Preferably, step S1 includes the following sub-steps: step S11, extracting multi-scale visual feature maps of the UAV global view image using an image-text dual-stream network, and extracting text feature sequences of natural language commands; step S12, calculating a multi-head cross-modal attention weight matrix between the multi-scale visual feature maps and the text feature sequences, generating a cross-modal candidate probability map, and determining coarse-grained target candidate boxes by bounding box regression calculation in the cross-modal candidate probability map.

[0028] Example 1: This example combines Figures 1 to 2 This paper describes a unified coarse-to-fine visual localization method for air-to-ground collaborative localization with task adaptive switching, as follows: Figure 1As shown, the method includes steps S1, S2, S3, and S4. Step S1 involves acquiring a global view image of the UAV and natural language commands, and determining coarse-grained target candidate boxes in the global view image. Step S2 involves determining a local detail view image through a task input switching module based on the current view task switching state, and extracting detail scaling and positioning reference parameters for the target region based on the coarse-grained target candidate boxes. Step S3 involves extracting detailed visual features of the local detail view image using a parameter-sharing visual encoder. When the current view task switching state is a single-view UAV visual positioning task, feature space geometric correction is performed on the detailed visual features based on the detail scaling and positioning reference parameters. When the current view task switching state is an air-ground collaborative visual positioning task, cross-view orthogonal projection and spatial topology reconstruction are performed based on the detailed visual features. Step S4 involves multi-scale decoding of the coarse-grained target candidate boxes to generate target candidate box decoding features, converting the target candidate box decoding features into a semantic prototype sequence, inputting the semantic prototype sequence as a key-value vector into a fine-grained decoder, and retrieving and outputting the target fine-grained positioning result from the transformed detailed visual features.

[0029] like Figure 2 As shown, the refined decoder uses the transformed detailed visual features as the query vector and the semantic prototype sequence as the key vector and value vector, and completes the refined localization correction through a multi-layer cross-attention mechanism.

[0030] Example 2: In a collaborative security patrol scenario within a large industrial park, the system needs to visually locate specific cyclists within the park. When a high-altitude patrol drone acquires a global view image and receives a natural language command to locate a specific cyclist, the fine-grained visual features of the cyclist from a high-altitude top-down perspective are easily visually obscured by shadows or buildings. Based on the actual working conditions, the system adaptively switches between single-view drone visual positioning and air-to-ground collaborative visual positioning tasks. This eliminates cross-view feature space mismatch and target identity perception ambiguity caused by differences in the perspectives of heterogeneous platforms. The system uses a lightweight dense alignment adapter to encode the global view image acquired by the high-altitude patrol drone to generate a global view visual query term sequence. Simultaneously, a pre-trained text encoder converts the natural language command into corresponding text embedding features. Then, a multimodal decoder performs cross-modal decoding interaction between the text embedding features and the global view visual query term sequence, outputting a text-enhanced visual representation. The prediction head then regresses and calculates normalized coarse-grained target candidate boxes in the global view image. , The calculation expression is as follows: ,in, For coarse-grained target candidate boxes, the subscripts are... Indicates a drone platform; The x-coordinate of the top-left corner of the coarse-grained target candidate box. The ordinate of the top-left corner of the coarse-grained target candidate box. The width of the coarse-grained target candidate box. The height of the coarse-grained target candidate box is given. In the above processing, the lightweight dense alignment adapter consists of two 2D convolutional layers that do not change the feature map size and one normalization layer. It is used to uniformly scale the global view image collected by the UAV to a resolution of 448×448 pixels and convert it into sequence data containing 1024 visual words. The multimodal decoder adopts a standard dual-stream cross-attention network, which contains four cascaded multimodal interaction modules. By performing a dot product operation between the 512-dimensional text embedding features and the visual word sequence in the attention matrix, the multimodal association weight is calculated, thereby completing the conditional enhancement guidance of text on visual features.

[0031] In generating coarse-grained target candidate boxes Then, the system uses coarse-grained target candidate boxes. width With height The system extracts the adaptive clipping size of the local detail view in situ. and the cropping center coordinates The calculation formula is as follows: , In the formula These are the preset local search coefficients. To prevent the introduction of excessive redundant background context, a maximum clipping size threshold is set. When the current viewpoint task is determined to be a single-view UAV visual localization task, the system will use coarse-grained target candidate boxes. The cropped image of the drone, magnified and cropped to the center, is identified as a local detail view image. This cropped area is then scaled to a fixed resolution of 448×448 pixels. A parameter-sharing visual encoder is used to extract the detailed visual features of this local detail view image. The initial visual features are then processed in the first feature calibration operator module based on detail scaling and positioning reference parameters. The feature space geometric correction is used to obtain the corrected detailed visual features. In the physical implementation of the network architecture, the first feature calibration operator module supports selecting either the online dynamic regression mode or the offline calibration matrix library retrieval mode of the neural network according to the computing platform resources. In this embodiment, the online dynamic regression mode of the neural network is preferred. In terms of physical construction, it is mapped and organized into an adaptive weight generation module. The system inputs image data into the neural network composed of this adaptive weight generation module, and regresses the spatial correction operator as the spatial transformation parameter tensor online. The spatial correction operator is then used to correct the detailed visual features. The system performs a feature space geometric transformation to obtain feature space visual features. In depth feature manifold processing, this transformation involves using a spatial transformation parameter tensor to perform spatial network resampling on the feature data, thereby dynamically correcting the geometric perspective distortion caused by the high-altitude overhead view. This maps the perspective distortion from the high-altitude view back to a unified spatial metric space. When the current viewpoint task state is switched to an air-ground collaborative visual positioning task, the system determines the spatiotemporally aligned ground view image as a local detail view image. It then uses a parameter-shared visual encoder to perform multi-scale feature sampling on the local detail view image, extracting detail representations of local regions as detail visual features, and using a second feature calibration operator. The module performs cross-view orthogonal projection and spatial topology reconstruction on detailed visual features. In the deep forward network topology, the second feature calibration operator module is constructed as a whole and is academically interpreted as a cross-view mapping network. It accurately aligns the UAV view features to the ground view feature space by performing channel-level linear projection on the feature data through the cross-view mapping network in the forward direction. This completes the online dynamic activation and generation of the parameters of the cross-view orthogonal projection operator. The cross-view orthogonal projection operator is used to perform matrix orthogonal projection transformation on the detailed visual features, reconstructing the spatial topology of the detailed visual features, thereby constraining the high-dimensional embedding vectors under the top view layout and ground appearance image to a unified spatial metric.

[0032] After completing the feature space transformation, the system enhances the visual representation with textual conditions. Perform global average pooling operations to extract global contextual features. and the visual representation Perform region of interest alignment and cropping to extract local identity features of the target. A complete semantic prototype sequence is generated by concatenating and combining channels. , The semantic combination relationships are as follows: ,in, For a complete semantic prototype sequence, where the subscripts are... Represents semantic elements; For global context feature units, where the subscripts are... Indicates the global context; This is a sequence of local region identity feature lexical units, where the subscripts... Representing a local region; the system will use semantic prototype sequences The key and value vectors in the cross-attention mechanism are mapped and input into the refined decoder. This allows the refined decoder to perform in-depth attention interaction computation between the cross-viewpoint aligned feature tensor and the pre-stored semantic prototype vectors constructed from semantic prototype sequence mappings. The refined decoder then processes the transformed detailed visual features. The system performs a linear mapping transformation to obtain a query vector. The query vector and key vector are then used to calculate the matrix dot product similarity and weighted to retrieve the value vector. A shared corner prediction head predicts the continuous coordinate probability distributions of the top-left and bottom-right corners. Soft-Argmax integral regression is introduced to generate continuously differentiable coordinate predictions, thus maintaining semantic coherence of the target during data flow and outputting high-precision bounding box coordinates as refined localization results. In this embodiment, the refined decoder adopts a 3-layer standard transformer decoder stacked architecture. Each layer contains a multi-head cross-attention mechanism sublayer and a feedforward neural network sublayer, with its feature channel dimension constantly calibrated to 256 dimensions. The shared corner prediction head consists of 4 two-dimensional convolutional layers and one linear layer. The first 3 two-dimensional convolutional layers have a kernel size of 3 rows and 3 columns with a stride of 1. By changing the number of channels layer by layer, accurate coordinate convergence is achieved. After outputting the refined localization results, the system performs a two-stage... Cross-view contrast alignment constraints are introduced into the prediction loop. The text encoder extracts the text physical resolution vector of the natural language command and constructs a training sample set consisting of positive and negative sample pairs. The normalized contrast loss value between the text physical resolution vector and the visual feature vectors at different positioning stages is calculated. By jointly supervising the text features of the natural language command and the visual features extracted in the two stages through the contrast loss value, the dynamic interference of densely arranged similar targets on the weights is effectively suppressed. This enables the global spatial layout under the high-altitude top view and the fine-grained appearance details under the ground level view to be aligned in a unified feature space, reducing the positioning ambiguity caused by the view mismatch and maintaining the engineering steady state of cross-view target association.

[0033] Example 3: In a validation environment consisting of a large-scale air-to-ground cooperative visual localization benchmark dataset built from real-world scenes (i.e., the RefUAV benchmark dataset covering 59 types of real-world complex scenes, over 110,000 images, and 240,000 natural language descriptive statements) and the RefUAV-Co cooperative subset containing 16,000 spatiotemporally synchronized images, 8,000 air-to-ground image pairs, and 13,000 matching instances), this example is used to verify the adaptive detail scaling based on the target scale proposed in this method. With center positioning reference parameters Semantic prototype sequence and cross-view symmetric contrast alignment loss The effectiveness of adaptive switching and parameter calibration during actual system operation was tested. The entire experiment was deployed on a computing platform equipped with four NVIDIA TITANRTX graphics cards and the PyTorch deep learning framework. DINOv2 ViT-B / 14 or ViT-L was used as the pre-trained visual backbone network, and CLIP text encoder was used to extract language features. The global batch size was set to 32, and the experiment was conducted on the RefUAV dataset with an initial learning rate of [missing information]. Pre-train for 50 rounds and decay to [a certain value] on the 30th round. Subsequently, on the RefUAV-Co dataset with the initial learning rate Fine-tune over 30 rounds and decay to [a certain value] in the 20th round. .

[0034] The system continuously receives global view images collected by high-altitude inspection drones and calculates coarse-grained target candidate boxes. Under different target span and shape variations, calculate the corresponding normalized width output. With normalization height The coarse-grained target candidate box is represented as a normalized quadruple. , The normalized coordinates of the top-left corner of the candidate box; this method An adaptive clipping strategy with clear mathematical and physical meaning is established based on the geometric parameters of coarse-grained target candidate boxes. Compared with empirical derivations lacking clear physical relationships, this strategy directly maps the clipping size and center position to the candidate box width and height, achieving interpretable acquisition of detailed views. The system calculates the adaptive clipping size of local detailed views in situ. and the cropping center coordinates The actual mathematical formula is as follows: In the formula, These are the preset local search coefficients. The maximum clipping size threshold is used to prevent the introduction of excessive redundant background context; specifically, during operation, when the system is in single-view UAV visual positioning mission mode, the system uses the calculated center coordinates... With cutting size The system determines the cropping region (constructs a cropping window), performs local adaptive cropping on the UAV's global view, and scales it to a fixed resolution of 448×448 pixels as the local detail view image I. This restores the fine-grained spatial geometry and appearance features lost during high-altitude global downsampling. In the air-to-ground collaborative visual positioning task mode, the system directly uses the temporally and spatially aligned ground view original image as the local detail view image through the task input switching module. The input is fed into the parameter-shared DINOv2 visual encoder to extract detailed visual features. This aligns high-resolution frontal surface details on the ground with aerial perspective features into a unified spatial metric.

[0035] Next, the system uses the lexical generation module to transform the visual representation that has been enhanced by textual conditions in the first stage. Convert to semantic prototype sequence The specific derivation process is as follows: First, for Perform global average pooling (GAP) operations to extract global context terms. Secondly, based on coarse-grained candidate boxes In features Execution RoIAlign operation at grid resolution and flattened into local region word sequences Finally, global context words and local region words are concatenated along the sequence dimension to form a complete semantic prototype sequence. In the second stage of the unified fine-tuning loop, the system will use semantic prototype sequences. As a cross-attention mechanism, the key vector and value vector combine detailed visual features. The query vector is input to the refined decoder, and a shared corner prediction head is used to predict the coordinate heatmaps of the top-left and bottom-right corners through Soft-Argmax integral regression to output refined bounding boxes. In order to explicitly bring the heterogeneous viewpoint manifold closer together during the training phase, the system introduces a cross-viewpoint symmetric InfoNCE contrastive loss between the two-stage features. Joint Phase 1 regression loss Compared with the second-stage regression loss Constructing the total loss function The balancing weights are configured as follows: , , .

[0036] To objectively and rigorously verify the technical effectiveness of this method, this embodiment uses the academically recognized average intersection-union ratio (mIoU) and positioning accuracy Pr@0.5 with a threshold of 0.5, as well as the collaborative success rate index Co-Pr@0.5, specifically constructed for air-to-ground collaborative positioning tasks, as evaluation benchmarks; where Co-Pr@0.5 is mathematically defined as: This indicator strictly requires that, under the same language command, both the drone's aerial view and the ground view can correctly and unambiguously locate the same target; any deviation in positioning from either viewpoint is considered a failure of coordination.

[0037] Data testing results show that: First, in comparative tests on the RefUAV-Co air-to-ground collaborative benchmark subset, mainstream models trained solely on a single viewpoint are limited in cross-view collaborative localization due to viewpoint mismatch and identity ambiguity. TransVG's Co-Pr@0.5 is only 6.13%, VGTR is only 2.97%, EEVG is only 4.60%, SimVG is only 6.61%, Dynamic-MDETR is only 0.34%, and zero-sample GroundingDINO is only 8.12%. Even our method, trained solely on RefUAV single-view data, achieves a Co-Pr@0.5 of only 0.5. The accuracy was only 15.35%; however, after introducing RefUAV-Co for collaborative alignment training, the Co-Pr@0.5 of the parallel baseline architecture of this invention improved to 35.83%, and the Co-Pr@0.5 of the serial visual baseline architecture reached 37.99%. The model using the complete unified architecture of this invention achieved 45.60% mIoU and 44.20% Co-Pr@0.5 from the UAV perspective, and 46.13% mIoU and 45.18% Pr@0.5 from the ground perspective. Finally, it achieved the best collaborative localization performance of Co-Pr@0.5 with a Co-Pr@0.5 of up to 39.09%, which is better than all the comparison benchmarks.

[0038] Second, in the comprehensive evaluation of the RefUAV single-view benchmark dataset, the ViT-B backbone network model of this invention achieved 66.40% mIoU and 75.54% Co-Pr@0.5, while the ViT-L backbone network model achieved 68.43% mIoU and 77.52% Co-Pr@0.5. Compared with the current industry-leading baseline method SimVG, which achieved 63.37% mIoU and 70.93% Co-Pr@0.5, this invention achieved significant technical improvements of 5.06% and 6.59% in mIoU and Co-Pr@0.5, respectively. It significantly outperformed TransVG (54.25% and 64.69%), VGTR (44.11% and 54.35%), EEVG (34.23% and 37.93%), Dynamic-MDETR (59.29% and 68.09%), and zero-sample GroundingDINO (29.26% and 34.36%) in terms of mIOU and Co-Pr@0.5.

[0039] Third, in the cross-domain generalization test on the publicly available AerialVG aerial visual localization dataset (which was not used for training), the ViT-B model of this invention achieved 50.24% mIoU and 54.29% Co-Pr@0.5, while the ViT-L model achieved 55.67% mIoU and 59.37% Co-Pr@0.5. This represents a 9.36% improvement compared to the 50.01% Co-Pr@0.5 published by the original baseline method for AerialVG, and a 30.01% improvement compared to the 29.36% Co-Pr@0.5 of GroundingDINO. This demonstrates extremely excellent zero-shot and cross-domain spatial semantic generalization robustness.

[0040] Fourth, in the semantic prototype lexical granularity ablation experiment conducted on the RefUAV-Co subset, Co-Pr@0.5 was 38.03% when using only global context lexical units, 38.20% when using only 4×4 local lexical units, and 38.28% when using a 1+2×2 configuration. However, Co-Pr@0.5 reached the optimal 39.09% when using the 1+4×4 medium granularity configuration of this invention. When the local grid was further expanded to 1+8×8, Co-Pr@0.5 decreased to 38.16%. The experiment confirmed that the 1+4×4 semantic prototype sequence can achieve the best complementary state between global context and local identity prior while suppressing local noise.

[0041] Fifth, in the cross-view contrast alignment loss ablation experiment, when the contrast loss constraint is removed, Co-Pr@0.5 is only 37.52%, when local RoI contrast alignment is introduced, Co-Pr@0.5 is 38.62%, and when the complete global symmetric contrast loss of this invention is introduced, Co-Pr@0.5 reaches 39.09%. This objectively confirms the key technical contribution of cross-view contrast alignment in constraining viewpoint-invariant characterization and eliminating heterogeneous manifold divergence.

[0042] Example 4: In the scenario of collaborative security inspection in industrial parks, this example provides an offline calibration matrix library retrieval and interpolation implementation method suitable for low-power edge chips or embedded systems. This serves as a parallel alternative to online regression in low-computing-power scenarios, or as an offline baseline initialization matrix for online regression. When the system faces the attitude pitch condition of a high-altitude inspection drone caused by strong wind disturbances, the global view image acquired from the high-altitude top-down perspective suffers nonlinear geometric perspective distortion due to the dynamic deflection of the drone camera's optical axis relative to the ground plane. Simultaneously, due to the presence of high-frequency strong light noise from specular reflections on the ground, the feature matching method experiences manifold distortion and alignment loss in the spatial feature representation within the coarse-grained target candidate box. The inefficiency leads to a decrease in target recognition accuracy during the refined tracking stage. To address the cross-view manifold mismatch problem, the processing unit establishes the functional specifications of the initial enabling environment before receiving the UAV global view image and natural language commands. The processing unit includes an embedded image processing chip and a memory for feature matrix caching, driving the multimodal data stream flow under a preset hardware sampling period. Before operation, the feature space correction loop constructs a multi-dimensional manifold correction benchmark. The first feature calibration operator module calibrates standard geometric pattern images under different optical axis tilt angles. In an exemplary embodiment, the value range of detail scaling and positioning reference parameters can be discretized into multiple calibration sub-intervals, and each calibration sub-interval stores a set of distinct values. The reference spatial transformation array can be directly retrieved from an offline matrix library or used as the initial reference parameters for an online regression neural network. The residual correction is calculated by the online neural network. In the deep feedforward network, the first feature calibration operator module is topologically represented as an adaptive weight generation module, and the reference spatial transformation array is mathematically represented as a spatial transformation parameter tensor. In another exemplary embodiment, the second feature calibration operator module can divide the gradient domain of the detail visual feature space into multiple gradient intervals. Each gradient interval stores a set of calculated cross-view orthogonal projection matrix parameter fine-tuning correction weights. This weight matrix constitutes an offline projection matrix library for table lookup retrieval. In contrast to linear interpolation, the second feature calibration operator module in deep learning networks is constructed as a cross-view mapping network. Specifically, the pre-stored reference space transformation array in the first feature calibration operator module can adopt, for example, a two-dimensional homogeneous coordinate transformation matrix architecture. Its internal matrix parameters are obtained by simulating the imaging calibration of standard calibration patterns by a drone at different tilt angles in a laboratory environment. This is specifically used to eliminate linear shearing and scaling of the image plane in the horizontal and vertical directions. The pre-stored cross-view orthogonal projection matrix in the second feature calibration operator module can adopt, for example, a channel-level linear mapping weight matrix structure. This is used to perform nonlinear orthogonal recombination of high-dimensional feature maps in the channel dimension, thereby achieving topological alignment of the feature space.

[0043] In this baseline environment, the lightweight dense alignment adapter reads the global view image captured by the high-altitude inspection UAV and calculates coarse-grained target candidate boxes. Extract coarse-grained target candidate boxes normalized width With normalization height The calculation unit is based on the normalized width With normalization height The current detail scaling and positioning reference parameters are calculated. When the detail scaling and positioning reference parameters fall within a specific calibration sub-interval, the calculation unit directly retrieves the pre-stored reference spatial transformation array of that sub-interval from the first feature calibration operator module as the spatial geometric correction operator. When the detail scaling and positioning reference parameters are at the boundary of two adjacent calibration sub-intervals, the calculation unit uses a linear weighted interpolation method to perform a weighted summation of the matrix elements at corresponding positions in the two adjacent reference spatial transformation arrays based on the algebraic distance between the current detail scaling and positioning reference parameters and the two endpoints, and outputs the corrected spatial geometric correction operator. The shared visual encoder extracts the detailed visual features of the local detail view image. Load the feature matrix cache and store the detailed visual features. Matrix multiplication is performed with the output spatial geometric correction operator, and spatial network resampling is performed on the feature data using the spatial transformation parameter tensor to transform the detailed visual features. The spatial dimension is determined, perspective distortion is eliminated, and the transformed spatial visual features are output.

[0044] When the ground view task is activated and the ground view image is input into the feature matrix buffer, the computing unit calculates the first-order differences of pixels in the horizontal and vertical directions of the ground view image. The spatial mean of the absolute values ​​of the first-order differences is calculated to output the current detailed visual features. Based on the current detailed visual features, the corresponding gradient interval in the second feature calibration operator module is matched. The system retrieves the pre-stored cross-view orthogonal projection operator by performing channel-level linear projection on the feature data through a cross-view mapping network. Specifically, the detailed visual features of the ground view image represent the high-frequency edge distribution density of the ground scene at the current imaging distance. When the relative pose and perspective projection distance between the UAV and the ground acquisition device change, the sampling rate of the texture details of the ground image in the discrete pixel space will synchronously undergo nonlinear convergence or divergence, causing a regular shift in the spatial mean of the first-order difference. Therefore, the detailed visual features can indirectly map the degree of geometric distortion caused by the difference in perspective depth between air and ground heterogeneous platforms, thus providing a digital feature basis with manifold continuity for matching cross-view orthogonal projection operators with specific transformation scales. The computing unit then uses the detailed visual features... Perform matrix multiplication with the cross-view orthogonal projection operator to transform detailed visual features. The spatial topology is mapped to the feature space visual features under a unified spatial metric space. As a refined decoder, the aligned feature tensor and the pre-stored semantic prototype vectors are used for attention interaction computation and semantic prototype sequences are utilized. To prevent feature data overflow caused by environmental changes or strong light noise, the input parameters for retrieving detailed features are configured with boundary truncation control in the feature processing path. When pose jitter causes the detail scaling and positioning reference parameters to exceed the preset range, or strong light reflection causes the measured detailed visual features to exceed the preset boundary, the computing unit truncates the abnormal input data and converges it to the corresponding interval endpoints. That is, the endpoint values ​​of the preset range are used as independent variables input into the operator matching and weighted interpolation processes, respectively. This parameter limiting control prevents the abnormal state of feature manifold divergence caused by the introduction of unregistered variables in the high-dimensional feature space, thus ensuring that the visual representation enhanced by text conditions remains intact. Semantic prototype sequence Furthermore, the refined positioning results remain stable and convergent under varying operating conditions and noise interference, keeping the corner point deviation of the refined positioning within a preset pixel range, thus maintaining the stability of the cross-view heterogeneous positioning architecture.

[0045] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A unified coarse-to-fine visual localization method for air-to-ground collaborative localization with task adaptive switching, characterized in that, Includes the following steps: Step S1: Obtain the global view image of the UAV and natural language commands, and determine coarse-grained target candidate boxes in the global view image of the UAV; Step S2: Based on the current viewpoint task switching state, determine the local detail view image through the task input switching module, and extract the detail scaling and positioning reference parameters of the target area based on the coarse-grained target candidate box; Step S3: Extract detailed visual features from the local detail view image using a parameter-sharing visual encoder; When the current viewpoint task is switched to a single-view UAV visual positioning task, feature space geometric correction is performed on the detailed visual features based on detail scaling and positioning reference parameters; when the current viewpoint task is switched to a cross-view UAV visual positioning task, cross-view feature transformation and alignment are performed on the detailed visual features. Step S4: Perform multi-scale processing on the visual representation enhanced by text conditions in the first stage, extract region of interest features based on coarse-grained target candidate boxes, generate target candidate box decoding features, convert the target candidate box decoding features into semantic prototype sequences, input the semantic prototype sequences as key-value vectors into the fine-grained decoder, and retrieve and output the target fine-grained localization results from the transformed detailed visual features.

2. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching as described in claim 1, characterized in that, Step S2 includes the following sub-steps: Step S21, when switching to a single-view UAV visual positioning task, the UAV partial cropping image is determined as a local detail view image; when switching to an air-ground cooperative visual positioning task, the spatiotemporally aligned ground view image is determined as a local detail view image; Step S22, multi-scale feature sampling is performed on the local detail view image to extract the detail representation of the local area.

3. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching as described in claim 1, characterized in that, Step S3 involves performing an affine transformation of the detail visual features in the feature space, which includes the following sub-steps: Step S31: When the current viewpoint task switching state is a single-view UAV visual positioning task, determine the spatial geometric correction operator in the first feature calibration operator module based on the detail scaling and positioning reference parameters; Step S32: Perform an affine matrix multiplication transformation on the detail visual features using the spatial geometric correction operator to obtain the feature space visual features.

4. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching as described in claim 1, characterized in that, Step S3, which involves performing cross-view feature transformation and alignment on detailed visual features, includes the following sub-steps: Step S33, when the current view task switching state is air-ground collaborative visual positioning task, the cross-view feature transformation operator is determined in the second feature calibration operator module based on the detailed visual features; Step S34, the cross-view feature transformation operator is used to perform matrix transformation on the detailed visual features to achieve cross-view feature alignment.

5. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching according to claim 1, characterized in that, Step S4, which converts the target candidate box decoding features into a semantic prototype sequence, includes the following sub-steps: Step S41, performing global average pooling operation on the target candidate box decoding features to extract global context features; Step S42: Perform region of interest alignment and cropping on the decoded features of the target candidate box to extract the target local identity features; Step S43: Concatenate the global context features and the target local identity features to generate a complete semantic prototype sequence.

6. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching according to claim 1, characterized in that, Step S4, which uses the semantic prototype sequence as a key-value vector to input the refined decoder, includes the following sub-steps: Step S44, mapping the semantic prototype sequence to the key vector and value vector in the cross-attention mechanism; Step S45, performing a linear mapping transformation on the transformed detail visual features to obtain the query vector, using the query vector and key vector to perform matrix dot product similarity calculation, and weighted retrieval of the value vector.

7. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching as described in claim 1, characterized in that, After outputting the refined target localization result in step S4, the following steps are also included: Step S5, introducing cross-view contrast alignment constraints in the two-stage prediction loop, and jointly supervising the text features of natural language instructions and the visual features extracted in the two stages by contrast loss.

8. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching according to claim 7, characterized in that, Step S5 includes the following sub-steps: Step S51, extract the text physical parsing vector of the natural language instruction using the text encoder; Step S52, perform pooling and projection on the visual features enhanced by text conditions in the first stage and the detailed visual features in the second stage respectively to obtain the global view embedding and the detailed view embedding, take the two-stage visual embeddings corresponding to the same target as positive sample pairs, take the embeddings of other targets in the batch as negative sample pairs, calculate the symmetric InfoNCE contrastive loss value, and update the network parameters of the trainable adapter, alignment module and task prediction module.

9. The air-to-ground cooperative unified coarse-to-fine visual positioning method with task adaptive switching according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S11, using an image-text dual-stream network to extract multi-scale visual feature maps of the UAV global view image and extract text feature sequences of natural language commands; Step S12, calculating a multi-head cross-modal attention weight matrix between the multi-scale visual feature maps and the text feature sequences to generate a cross-modal candidate probability map, and determining coarse-grained target candidate boxes by bounding box regression calculation in the cross-modal candidate probability map.

Images