Unmanned aerial vehicle air navigation visual language motion control method with active dialogue capability

Abstract

The invention provides an unmanned aerial vehicle air navigation visual language motion control method with an active dialogue capability, and belongs to the technical field of unmanned aerial vehicle navigation. The method comprises the following steps: processing dialogue historical data through LLM, extracting image features through a visual encoder, obtaining space and time sequence features in combination with observation historical data, introducing a PNaQ module to realize displacement updating, termination judgment and active question asking, and amplifying a data set by matching with a one-time data enhancement technology, so that the problems of disordered long-range navigation time-space relationship, lack of active question asking and the like are solved; and the navigation accuracy and interactivity are improved.

Description

Technical Field

[0001] This invention belongs to the field of UAV navigation technology, specifically relating to a visual language action control method for UAV aerial navigation with proactive dialogue capabilities. Background Technology

[0002] The goal of Visual Language Navigation (VLN) is to enable intelligent agents to navigate autonomously based on human language instructions and visual observations.

[0003] Since the inherent ambiguity of language descriptions often makes a single command insufficient to accurately locate the target position, visual dialogue navigation (VDN) methods are needed. For example, Thomason et al. pioneered dialogue-aware navigation by integrating human dialogue history as text input, while Fan et al. extended this paradigm to UAV navigation.

[0004] However, current drone navigation systems suffer from the following problems, the main causes of which include: 1. In long-range navigation missions, historical spatiotemporal relationships are chaotic: long-range flight of UAVs leads to significant changes in perspective and scale. Existing methods often stack the observation history in the order of video sequences or only make coarse-grained memories. There is a lack of a unified spatial reference system and explicit spatial structure modeling, which makes it difficult to correspond "what was observed" with "global position". This results in chaotic historical spatiotemporal relationships, difficulty in backtracking and retrieval, and untimely correction after deviation.

[0005] 2. Lack of proactive questioning ability: Existing models mostly rely on pre-given dialogue history or template-based dialogue generation, lacking an explicit evaluation mechanism for navigation uncertainty. They cannot proactively initiate clarification and supplementary questions when instructions are ambiguous or the trajectory deviates, thus making them prone to continuous erroneous exploration in complex environments.

[0006] 3. Coupling of action prediction and language generation leads to mutual interference: The optimization goals and output forms of the two types of tasks are significantly different; if a single fusion feature or a single prediction branch is used to simultaneously handle action and language output, conflicts between representation and gradient are likely to occur, resulting in unstable action decisions or unreasonable timing / content of questions, thereby reducing the overall navigation efficiency and success rate. Summary of the Invention

[0007] This invention proposes a visual language action control method for UAV aerial navigation with proactive dialogue capabilities, which can solve the above-mentioned problems.

[0008] To address the above problems, the present invention proposes the following technical solutions: A visual language action control method for UAV aerial navigation with proactive dialogue capability includes the following steps: S1: The dialogue history data of the drone is processed through the embedding layer of the Large Language Model (LLM) to obtain the dialogue history encoding vector Etext,t at the current time t; the current frame image It of the drone is processed through the visual encoder to extract features to obtain the image features vt of the current frame image, which are then flattened and linearly projected onto the LLM space to obtain the visual encoding vector Evis,t at the current time t. S2: Based on the historical observation data of the UAV, the historical observation data is projected onto the ground using orthophoto reprojection and then stitched together to obtain the stitched image Mt at the current time t; the stitched image Mt at the current time t is processed by taking the minimum bounding rectangle and filling invalid areas with 0 to obtain a map Rt of uniform size at the current time t; the map Rt of uniform size at the current time t is first processed by visual feature extraction, and then by flattening and encoding operations to obtain the spatial features at the current time t; where Ik represents the image of the k-th frame; Tk represents the camera pose in the image of the k-th frame; S3: First, based on the historical image feature sequence v1:t-1 at the current time t, obtain its feature sequence in the time dimension; then, based on the map Rt of uniform size, extract the global map encoding features at the current time; subsequently, adopt a multi-granularity attention mechanism, and through linking, finally obtain the temporal features of historical exploration; S4: Introducing the progress-driven navigation-question selection module PNaQ. PNaQ introduces two special lexical units [act] and [que]. [act] is used for action prediction, and [que] is used for language generation. [act] is followed by four learnable lexical units: , which, combined with Etext,t, Evis,t, and , form the input sequence Dt. The input sequence Dt is input into the LLM, which outputs the behavior of the drone to be jointly managed. , is used to update the drone's horizontal and vertical displacement vectors. , is used to predict the drone's termination probability, and based on a set threshold, it is determined whether the drone should stop navigation. , is used to evaluate the navigation progress, and based on set rules, it is determined whether to issue a question. When a question is issued, the external response module returns an answer and updates the dialogue history data in S1 with the answer. S5: Based on one-time data augmentation, the new “dialogue-action-trajectory triples” generated by LLM predictions are incorporated into the augmented dataset and sampled at a set ratio in each round of training.

[0009] Furthermore, in step S1, the current time... t Dialogue history encoding vector E text,t The formula is:

[0010] In the formula, X text,t Indicates the current timet Dialogue history data; Embed (*) indicates an embedded function operation; E text,t Indicates the current time t The dialogue history encoding vector, L For the length of the marker; D for LLM The dimensions of hidden space; Current moment t Visual encoding vector E vis,t The formula is:

[0011] In the formula, E vis,t Indicates the current frame image I t The visual encoding vector; reshape (*) indicates the flattening operation; v t Indicates the current frame image I t Image features extracted by a visual encoder; W v Represents image features projected onto LLM The weight matrix of the space; b v Represents the weight matrix W v The bias term; N Indicates the number of visual markers; D for LLM The dimensions of hidden space.

[0012] Further, step S2 includes the following steps: S2.1: The historical observation data is projected onto the ground using orthophoto reprojection and then stitched together to obtain the current time. t spliced ​​images M t The formula is:

[0013] In the formula, I k Indicates the first k The image of the frame; T k Indicates the first k The camera pose in a frame image; K Indicates camera intrinsic parameters; Indicates the ground reference plane; Indicates will I k in accordance withT k 、K、 Orthorectified reprojection onto the ground coordinate system; Indicates the first k A single-frame bird's-eye view map obtained by orthorectifying a frame image; Fuse (*) indicates an operation that stitches together a bird's-eye view map of all frames according to ground coordinates and performs weighted fusion in overlapping areas; M t Indicates the current time t spliced ​​images; S2.2: For the current moment t spliced ​​images M t The current time step is obtained by taking the smallest bounding rectangle and filling the invalid region with zeros. t Uniform size map R t The formula is:

[0014] In the formula, ZeroPad (*) indicates the operation of filling with zeros; MBR (*) indicates the minimum bounding rectangle operation; R t Indicates the current time t A map of uniform size; S2.3: For the current moment t Uniform size map R t First, visual features are extracted, and then flattening and encoding operations are performed to obtain the current time step. t Spatial features The formula is:

[0015] In the formula, f v (*) indicates a visual feature extraction operation; flatten (*) indicates the flattening operation; Indicates the encoding function; N S express The number of visual lexical elements; D for LLM The dimensions of hidden space; Indicates the current time t Spatial characteristics.

[0016] Further, step S3 includes the following steps: S3.1: Based on the current time tThe historical image feature sequence is used to obtain its feature sequence in the time dimension; the formula is:

[0017] In the formula, v 1:t-1 Indicates the current time t Historical image feature sequences; pooling 2D (*) indicates that the mean is calculated along the spatial dimension; flatten (*) indicates the flattening operation; Represents a feature sequence in the time dimension; S3.2: Set the current time t Uniform size map R t The AnyRes image encoder is used to obtain its global map coding features, and the formula is as follows:

[0018] In the formula, Indicates the current time t Global map coding features; R t Indicates the current time t A map of uniform size; This represents the encoding function of the AnyRes image encoder; S3.3: For the first i The clipping formula for a navigation waypoint at time t is:

[0019] In the formula, d j This indicates the set clipping radius; Indicates in In the middle, with the first i Using each navigation waypoint as the center, d j Trim based on the trim radius; Represents the current time t, the nth i A cropped image of navigation waypoints; S3.4: At the current time t, combine the time-dimensional features of each navigation waypoint with its i-th... i The cropped image of the nth navigation waypoint is obtained by performing a cross-attention operation to obtain the nth navigation waypoint image after the cross-attention operation. i The feature map of each navigation waypoint is given by the formula:

[0020] In the formula, Indicates the first iThe time-dimensional characteristics of navigation waypoints; cross _ atten This indicates a cross-attention operation; This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Represents the current time t, the nth i A cropped image of navigation waypoints; S3.5: Settings It integrates multi-scale feature information and enhances representation density; the formula is:

[0021] In the formula, This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Indicates the current time t ,exist d j At the scale, the first i Feature representation of each navigation waypoint; Indicates the first i The time-dimensional characteristics of navigation waypoints; Indicates the first i Each navigation waypoint can learn a set of parameters; Indicates the current time t , No. i The characteristic dimensions of each navigation waypoint; S3.6: Concatenate the feature dimensions of all historical navigation waypoints and process them through encoding to obtain the current time. t Temporal characteristics The formula is:

[0022] In the formula, Indicates the total number of navigation points; Indicates the current time t The feature dimensions of the first to the (t-1)th navigation waypoints; D For the dimension of the LLM hidden space, This represents the encoding function.

[0023] Furthermore, step S4 includes the following steps: S4.1: Will E text,t , E vis,t 、 、 The input sequence consists of four learnable lexical units.D t The formula is:

[0024] In the formula, and Two morphemes representing PnaQ; Indicates additional learnable lexical units; S4.2: Input sequence D t Input into LLM, output from LLM The formula is:

[0025] In the formula, LLM (*) indicates that by LLM deal with; This represents the hidden state used for horizontal displacement prediction; This represents the hidden state used for vertical displacement prediction; This represents the hidden state used to terminate probability prediction; This represents the hidden state used for navigation progress prediction; S4.3: Utilize Update the drone's horizontal and vertical displacement vectors; The formula is:

[0026] In the formula, and These represent the horizontal displacement vector prediction MLP layer and the vertical displacement vector prediction MLP layer, respectively. Indicates the current time t The drone's output horizontal and vertical displacement vectors; S4.4: Used to predict the termination probability; the formula is:

[0027] In the formula, Represents the sigmoid function; For terminating probability prediction MLP layers; Indicates the current time t The probability of drone termination; S4.5: Determine the navigation progress and determine whether a question is generated based on the navigation progress. If a question is generated, return an answer through the external response module, update the dialogue history data in S1 with the returned answer, and repeat S1-S4; if no question is generated, the drone continues to move until a question is generated. Drones use To estimate navigation progress, the formula is:

[0028] In the formula, MLP layer for navigation progress prediction; Represents the sigmoid function; This indicates the estimated navigation progress; the navigation progress is defined as the ratio of the distance from the UAV to the starting point of the trajectory to the total distance from the UAV to both the starting and ending points of the trajectory.

[0029] Furthermore, when Upon setup, the drone's speech engine automatically generates a question, the external response module returns an answer, and updates the dialogue history data in S1 with the answer. The drone also updates its own position. If the condition is not met, update the drone's position directly; when... When the drone stops, it stops; otherwise, it cycles through S1-S5. During the cycle, the drone navigates by continuously asking questions and returning answers.

[0030] Further, step S5 includes the following steps: S5.1: The drone completes the current training set. V After the sample, that is, the drone executes the model's actions. And move from the current location to the predicted location. ; S5.2: For predicted location Calculate the optimal action to make the drone's trajectory closer to the real trajectory. Then, the three sets of information, "dialogue-action-trajectory," are packaged into a triple; The meaning is: the current predicted location of the drone. To the manually marked location The horizontal and vertical displacement vectors required for the straight-line distance; S5.3: The generated triples will be saved to the augmented dataset buffer. V g middle; S5.4: Before each training iteration, the dataset will be manually labeled. V 0 and augmented datasets V g In this process, samples are drawn according to a pre-set ratio; S5.5: The extracted samples will be combined to form a new training set. V 新 Used for this round of training, looping S5.1-S5.5, using V 新 Training set to replace S5.1 V .

[0031] The beneficial effects that can be achieved by adopting the above-mentioned technology are: 1. This method fully integrates dialogue history data and image features, and obtains the current moment based on the observation history data of UAVs. t The spatial and temporal characteristics are fully considered in both time and space, avoiding the problem of chaotic historical spatiotemporal relationships in long-range navigation tasks.

[0032] 2. It adopts learnable lexical units and actively asks questions based on the navigation process, and updates the dialogue history data with the external response module to realize the interaction between the drone and the human. 3. Four learnable lexical units are used. It is used to estimate navigation progress and thus proactively ask questions. This is used to update the drone's horizontal and vertical displacement vectors. Actions and speech are routed by different features to avoid interference.

[0033] 4. One-time data augmentation expands the dataset. This method can ensure the quality, accuracy, and usability of the newly added dataset as much as possible. At the same time, through the set sampling strategy, it ensures that a portion of the original data or the data with the least disturbance is always retained in the training set, avoiding the distribution of the training data deviating from the true situation due to too much newly generated data, thereby affecting the model performance. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the model's modules; Figure 2 This is a schematic diagram of the global map and cross-attention; Figure 3 This is a diagram illustrating a drone proactively making inquiries. Figure 4 This is a diagram illustrating how a drone actively inquires about navigation progress. Figure 5 This is a schematic diagram of a one-time enhancement technology. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] like Figure 1 As shown, a visual language action control method for UAV aerial navigation with proactive dialogue capability specifically includes the following steps: S1: Encoding of UAV multimodal data. This multimodal data includes: dialogue history data, current frame image, and observation history data, specifically including the following steps: S1.1: Obtain the current moment from the drone's dialogue history data through the embedding layer of a Large Language Model (LLM). t The "dialogue history encoding vector". The formula is:

[0037] In the formula, X text,t Indicates the current time t Dialogue history data; Embed (*) indicates an embedded function operation; E text,t Indicates the current time t The dialogue history encoding vector can be... LLM The backbone network is directly used for feature representation in inference; L For the length of the marker; D for LLM The dimensions of hidden space.

[0038] S1.2: Get the current frame image of the drone. I t Features are extracted using a visual encoder, flattened, and linearly projected onto... LLM Space, to obtain the current moment t The "visual encoding vector". The formula is:

[0039] In the formula, E vis,t Indicates the current frame image I t The visual encoding vector; reshape (*) indicates the flattening operation; v t Indicates the current frame image I t Image features extracted by a visual encoder; W v Represents image features projected onto LLM The weight matrix of the space; b v Represents the weight matrix W v The bias term; N Indicates the number of visual markers; D for LLM The dimensions of hidden space.

[0040] S2: Based on historical observation data from UAVs Get the current time t Spatial features Specifically, it includes the following steps: S2.1: To eliminate perspective distortion and provide a unified ground coordinate system, orthorectified reprojection is used to project historical observation data onto the ground and then stitch them together to obtain the current time. t spliced ​​images M t The formula is:

[0041] In the formula, I k Indicates the first k The image of the frame; T k Indicates the first k The camera pose in a frame image; K Indicates camera intrinsic parameters; Indicates the ground reference plane; Indicates will I k in accordance with T k 、K、 Orthorectified reprojection onto the ground coordinate system; Indicates the first k A single-frame bird's-eye view map obtained by orthorectifying a frame image; Fuse (*) indicates an operation that stitches together a bird's-eye view map of all frames according to ground coordinates and performs weighted fusion in overlapping areas; M t Indicates the current time t A stitched image.

[0042] S2.2: For the current moment t spliced ​​images M t The current time step is obtained by taking the smallest bounding rectangle and filling the invalid region with zeros. t Uniform size map R t The formula is:

[0043] In the formula, ZeroPad (*) indicates the operation of filling with zeros; MBR (*) indicates the minimum bounding rectangle operation; R t Indicates the current time t A map of a uniform size.

[0044] S2.3: For the current momentt Uniform size map R t First, visual features are extracted, and then flattening and encoding operations are performed to obtain the current time step. t Spatial features The formula is:

[0045] In the formula, f v (*) indicates a visual feature extraction operation; flatten (*) indicates the flattening operation; Indicates the encoding function; N S express The number of visual lexical elements; D for LLM The dimensions of hidden space; Indicates the current time t Spatial characteristics.

[0046] S3: First, based on the current time... t The historical image feature sequence is used to obtain its feature sequence in the time dimension; then, based on a map of uniform size... R t Extract the global map encoding features at the current time; then employ a multi-granularity attention mechanism, through... and Linking these elements and ultimately revealing the temporal characteristics of historical exploration. Specifically, it includes the following steps: S3.1: Based on the current time t The historical image feature sequence is used to obtain its feature sequence in the time dimension; the formula is:

[0047] In the formula, v 1:t-1 Indicates the current time t The historical image feature sequence, namely the feature sequence composed of the images from frame 1 to frame t-1, is obtained by extracting features from each frame image through the visual encoder in step 2. pooling 2D (*) indicates that the mean is calculated along the spatial dimension; flatten (*) indicates the flattening operation; It represents a feature sequence in the time dimension.

[0048] S3.2: Set the current time t Uniform size map R tThe AnyRes image encoder is used to obtain its global map coding features. The formula is:

[0049] In the formula, Indicates the current time t Global map coding features; R t Indicates the current time t A map of uniform size; This represents the encoding function of the AnyRes image encoder.

[0050] S3.3: Employs a multi-granularity attention mechanism to... and Link, that is: link the current moment t Historical image feature sequences navigation waypoints Projected to The position on the feature map is obtained. The meaning of navigation waypoints is: the coordinates of the unmanned aerial vehicle (UAV). In each frame of historical image features, there is one and only one coordinate for the UAV. See the specific projected image for details. Figure 2 .exist Perform cropping to obtain the current time. t A cropped image of each navigation waypoint.

[0051] For the i The clipping formula for a navigation waypoint at time t is:

[0052] In the formula, d j This indicates the set clipping radius; Indicates in In the middle, with the first i Using each navigation waypoint as the center, d j Trim based on the trim radius. Represents the current time t, the nth i A cropped image of navigation waypoints.

[0053] S3.4: At the current time t, combine the time-dimensional features of each navigation waypoint with its i-th... i The cropped image of the nth navigation waypoint is obtained by performing a cross-attention operation to obtain the nth navigation waypoint image after the cross-attention operation. i Feature map of navigation waypoints.

[0054] The formula is:

[0055] In the formula, Indicates the first i The time-dimensional characteristics of navigation waypoints; cross _ atten This indicates a cross-attention operation; This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Represents the current time t, the nth i A cropped image of navigation waypoints.

[0056] S3.5: Settings This integrates multi-scale feature information and enhances representation density. The formula is:

[0057] In the formula, This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Indicates the current time t ,exist d j At the scale, the first i Feature representation of each navigation waypoint; Indicates the first i The time-dimensional characteristics of navigation waypoints; Indicates the first i Each navigation waypoint can learn a set of parameters; Indicates the current time t , No. i The characteristic dimensions of each navigation waypoint.

[0058] S3.6: Concatenate the feature dimensions of all historical navigation waypoints and process them through encoding to obtain the current time. t Temporal characteristics The formula is:

[0059] In the formula, Indicates the total number of navigation points; Indicates the current time t The feature dimensions of the first to the (t-1)th navigation waypoints; D For the dimension of the LLM hidden space, This represents the encoding function.

[0060] S4: Introduces the progress-driven navigation-query selection module PNaQ. PNaQ introduces two special lexical units, [act] and [que], where [act] is used for action prediction and [que] is used for language generation. [act] is followed by four learnable lexical units: , representing horizontal displacement, vertical displacement, termination probability, and navigation progress prediction terms, respectively, combined with E text,t , E vis,t 、 , Composition of input sequence D t ; Input sequence D t Input into LLM, output The same applies to the management of drones. Specifically, this includes the following steps: S4.1: Will E text,t , E vis,t 、 , The input sequence consists of four learnable lexical units. D t The formula is:

[0061] S4.2: Input sequence D t Input into LLM, output from LLM The formula is:

[0062] In the formula, They respectively represent the following: The hidden state after LLM processing; among which... This represents the hidden state used for horizontal displacement prediction; This represents the hidden state used for vertical displacement prediction; This represents the hidden state used to terminate probability prediction; This represents the hidden state used for navigation progress prediction.

[0063] S4.3: Utilize Update the drone's horizontal and vertical displacement vectors.

[0064] The formula is:

[0065] In the formula, and These represent the horizontal displacement vector prediction MLP layer and the vertical displacement vector prediction MLP layer, respectively. Indicates the current time t The drone outputs horizontal and vertical displacement vectors.

[0066] S4.4: Used to predict the termination probability; the formula is:

[0067] In the formula, Represents the sigmoid function; For terminating probability prediction MLP layers; Indicates the current time t The probability of drone termination.

[0068] S4.5: Determine the navigation progress and, based on the progress, decide whether to generate a question. If a question is generated, return an answer through the external response module, update the dialogue history data in S1 with the returned answer, and repeat S1-S4. If no question is generated, the drone continues to move until a question is generated. See details... Figure 3 and Figure 4 .

[0069] Drones use To estimate navigation progress, the formula is:

[0070] In the formula, MLP layer for navigation progress prediction; Represents the sigmoid function; This indicates the estimated navigation progress; the navigation progress is defined as the ratio of the distance from the UAV to the starting point of the trajectory to the total distance from the UAV to both the starting and ending points of the trajectory.

[0071] when Upon activation, the drone's speech head automatically generates a question, the external response module returns an answer, and updates the dialogue history data in S1 with the answer, while the drone updates its own position. when If the condition is not met, update the drone's location directly; The drone's position update formula is: Set the current position of the drone G t =[ x t , y t , z t If the position of the drone is determined by the given information, then the drone's position at the next moment will be:

[0072] In the formula, G t This indicates the position of the drone at the current time t; G t+1This indicates the position of the drone at time t+1; Indicates the current time t The drone outputs horizontal and vertical displacement vectors.

[0073] when When the condition is met, the drone stops; otherwise, it cycles through S1-S5. During this cycle, the drone navigates by continuously asking questions and returning answers.

[0074] S5: Based on One-Time Data Augmentation (OTDA), this method provides a large number of samples during the model training phase; it incorporates new "dialogue-action-trajectory triples" generated by LLM predictions into the augmented dataset and samples them at a set proportion in each training round; see details. Figure 5 Specifically, it includes the following steps: S5.1: The drone completes the current training set. V After the sample, that is, the drone executes the model's actions. And move from the current location to the predicted location. ; S5.2: For predicted location Calculate the optimal action to make the drone's trajectory closer to the real trajectory. Then, the three sets of information, "dialogue-action-trajectory", are packaged into a triple. The meaning is: the current predicted location of the drone. To the manually marked location The horizontal and vertical displacement vectors required for the straight-line distance; S5.3: The generated triples will be saved to the augmented dataset buffer. V g middle.

[0075] S5.4: Before each training iteration, the dataset will be manually labeled. V 0 and augmented datasets V g In this process, samples are drawn according to a pre-set ratio (e.g., 8:2); S5.5: The extracted samples will be combined to form a new training set. V 新 Used for this round of training, looping S5.1-S5.5, using V 新 Training set to replace S5.1 V The purpose of this sampling strategy is to ensure that a portion of the original data or the least disturbed data is always retained in the training set, avoiding excessive newly generated data that could cause the distribution of the training data to deviate from the true situation, thereby affecting model performance.

[0076] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A visual language action control method for aerial navigation of unmanned aerial vehicles (UAVs) with proactive dialogue capabilities, characterized in that, Includes the following steps: S1: The drone's dialogue history data is processed through the embedding layer of a large language model (LLM) to obtain the current moment. t Dialogue history encoding vector E text,t ; the current frame image of the drone I t The image features of the current frame are obtained by extracting features through a visual encoder. v t And flatten and linearly projected onto LLM Space, to obtain the current moment t Visual encoding vector E vis,t ; S2: Historical observation data based on UAVs The historical observation data is projected onto the ground using orthophoto reprojection and then stitched together to obtain the current time. t spliced ​​images M t ; for the current moment t spliced ​​images M t The current time step is obtained by taking the smallest bounding rectangle and filling the invalid region with zeros. t Uniform size map R t ; for the current moment t Uniform size map R t First, visual features are extracted, and then flattening and encoding operations are performed to obtain the current time step. t Spatial features In the formula, I k Indicates the first k The image of the frame; T k Indicates the first k The camera pose in a frame image; S3: First, based on the current time... t Historical image feature sequence v 1:t-1 To obtain its feature sequence in the time dimension. Then, based on a map of uniform size R t Extract the global map coding features at the current time. ; Subsequently, a multi-granularity attention mechanism was employed, through... and Linking these elements and ultimately revealing the temporal characteristics of historical exploration. ; S4: Introduces the progress-driven navigation-query selection module PNaQ. PNaQ introduces two special lexical units, [act] and [que], where [act] is used for action prediction and [que] is used for language generation. [act] is followed by four learnable lexical units: , combined E text,t , E vis,t 、 、 Composition of input sequence D t ; Input sequence D t Input into LLM, output Jointly manage the behavior of drones, utilize Update the drone's horizontal and vertical displacement vectors; utilize Predict the probability of drone termination, and determine whether the drone should stop navigation based on a set threshold; utilize... Assess navigation progress, determine whether to ask a question based on the set rules, and when a question is asked, the external response module returns an answer and updates the dialogue history data in S1 with the answer; S5: Based on one-time data augmentation, the new "dialogue-action-trajectory triples" generated by LLM predictions are incorporated into the augmented dataset and sampled at a set ratio in each round of training.

2. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, In step S1, Current moment t Dialogue history encoding vector E text,t The formula is: ； In the formula, X text,t Indicates the current time t Dialogue history data; Embed (*) indicates an embedded function operation; E text,t Indicates the current time t The dialogue history encoding vector L For the length of the marker; D for LLM The dimensions of hidden space; Current moment t Visual encoding vector E vis,t The formula is: ； In the formula, E vis,t Indicates the current frame image I t The visual encoding vector; reshape (*) indicates the flattening operation; v t Indicates the current frame image I t Image features extracted by a visual encoder; W v Represents image features projected onto LLM The weight matrix of the space; b v Represents the weight matrix W v The bias term; N Indicates the number of visual markers; D for LLM The dimensions of hidden space.

3. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, Step S2 includes the following steps: S2.1: The historical observation data is projected onto the ground using orthophoto reprojection and then stitched together to obtain the current time. t spliced ​​images M t The formula is: ； In the formula, I k Indicates the first k The image of the frame; T k Indicates the first k The camera pose in a frame image; K Indicates camera intrinsic parameters; Indicates the ground reference plane; Indicates will I k in accordance with T k 、K、 Orthorectified reprojection onto the ground coordinate system; Indicates the first k A single-frame bird's-eye view map obtained by orthorectifying a frame image; Fuse (*) indicates an operation that stitches together a bird's-eye view map of all frames according to ground coordinates and performs weighted fusion in overlapping areas; M t Indicates the current time t spliced ​​images; S2.2: For the current moment t spliced ​​images M t The current time step is obtained by taking the smallest bounding rectangle and filling the invalid region with zeros. t Uniform size map R t The formula is: ； In the formula, ZeroPad (*) indicates the operation of filling with zeros; MBR (*) indicates the minimum bounding rectangle operation; R t Indicates the current time t A map of uniform size; S2.3: For the current moment t Uniform size map R t First, visual features are extracted, and then flattening and encoding operations are performed to obtain the current time step. t Spatial features The formula is: ； In the formula, f v (*) indicates a visual feature extraction operation; flatten (*) indicates the flattening operation; Indicates the encoding function; N S express The number of visual lexical elements; D for LLM The dimensions of hidden space; Indicates the current time t Spatial characteristics.

4. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, Step S3 includes the following steps: S3.1: Based on the current time t The historical image feature sequence is used to obtain its feature sequence in the time dimension; the formula is: ； In the formula, v 1:t-1 Indicates the current time t Historical image feature sequences; pooling 2D (*) indicates that the mean is calculated along the spatial dimension; flatten (*) indicates the flattening operation; Represents a feature sequence in the time dimension; S3.2: Set the current time t Uniform size map R t The AnyRes image encoder is used to obtain its global map coding features, and the formula is as follows: ； In the formula, Indicates the current time t Global map coding features; R t Indicates the current time t A map of uniform size; This represents the encoding function of the AnyRes image encoder; S3.3: For the first i The clipping formula for a navigation waypoint at time t is: ； In the formula, d j This indicates the set clipping radius; Indicates in In the middle, with the first i Using each navigation waypoint as the center, d j Trim based on the trim radius; Represents the current time t, the nth i A cropped image of navigation waypoints; S3.4: At the current time t, combine the time-dimensional features of each navigation waypoint with its i-th... i The cropped image of the nth navigation waypoint is obtained by performing a cross-attention operation to obtain the nth navigation waypoint image after the cross-attention operation. i The feature map of each navigation waypoint is given by the formula: ； In the formula, Indicates the first i The time-dimensional characteristics of navigation waypoints; cross _ atten This indicates a cross-attention operation; This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Represents the current time t, the nth i A cropped image of navigation waypoints; S3.5: Settings It integrates multi-scale feature information and enhances representation density; the formula is: ； In the formula, This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Indicates the current time t ,exist d j At the scale, the first i Feature representation of each navigation waypoint; Indicates the first i The time-dimensional characteristics of navigation waypoints; Indicates the first i Each navigation waypoint can learn a set of parameters; Indicates the current time t , No. i The characteristic dimensions of each navigation waypoint; S3.6: Concatenate the feature dimensions of all historical navigation waypoints and process them through encoding to obtain the current time. t Temporal characteristics The formula is: ； In the formula, Indicates the total number of navigation points; Indicates the current time t The feature dimensions of the first to the (t-1)th navigation waypoints; D For the dimension of the LLM hidden space, This represents the encoding function.

5. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, Step S4 includes the following steps: S4.1: Will E text,t , E vis,t 、 、 The input sequence consists of four learnable lexical units. D t The formula is: ； In the formula, and Two morphemes representing PnaQ; Indicates additional learnable lexical units; S4.2: Input sequence D t Input into LLM, output from LLM ; The formula is: ； In the formula, LLM (*) indicates that by LLM deal with; This represents the hidden state used for horizontal displacement prediction; This represents the hidden state used for vertical displacement prediction; This represents the hidden state used to terminate probability prediction; This represents the hidden state used for navigation progress prediction; S4.3: Utilize Update the drone's horizontal and vertical displacement vectors; The formula is: ； In the formula, and These represent the horizontal displacement vector prediction MLP layer and the vertical displacement vector prediction MLP layer, respectively. Indicates the current time t The drone's output horizontal and vertical displacement vectors; S4.4: Used to predict the termination probability; the formula is: ； In the formula, Represents the sigmoid function; For terminating probability prediction MLP layers; Indicates the current time t The probability of drone termination; S4.5: Determine the navigation progress and determine whether a question is generated based on the navigation progress. If a question is generated, return an answer through the external response module, update the dialogue history data in S1 with the returned answer, and repeat S1-S4; if no question is generated, the drone continues to move until a question is generated. Drones use To estimate navigation progress, the formula is: ； In the formula, MLP layer for navigation progress prediction; Represents the sigmoid function; This indicates the estimated navigation progress; the navigation progress is defined as the ratio of the distance from the UAV to the starting point of the trajectory to the total distance from the UAV to both the starting and ending points of the trajectory.

6. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 5, characterized in that, when Upon setup, the drone's speech engine automatically generates a question, the external response module returns an answer, and updates the dialogue history data in S1 with the answer. The drone also updates its own position. If the condition is not met, update the drone's position directly; when... When the drone stops, it stops; otherwise, it cycles through S1-S5. During the loop, the drone navigates by continuously asking questions and returning answers.

7. A visual language action control method for UAV aerial navigation with active dialogue capability according to claim 5, characterized in that, Step S5 includes the following steps: S5.1: The UAV completes the current training set V After the sample, that is, the drone executes the model's actions. and move from the current location to the predicted location. ; S5.2: For predicted location Calculate the optimal action to make the drone's trajectory closer to the real trajectory. Then, the three sets of information, "dialogue-action-trajectory", are packaged into a triple; The meaning is: the current predicted location of the drone. To the manually marked location The horizontal and vertical displacement vectors required for the straight-line distance; S5.3: The generated triples will be saved to the augmented dataset buffer. V g middle; S5.4: Before each training iteration, the dataset will be manually labeled. V 0 and augmented datasets V g In this process, samples are drawn according to a pre-set ratio; S5.5: The extracted samples will be combined to form a new training set. V 新 Used for this round of training, looping S5.1-S5.5, using V 新 Training set to replace S5.1 V .

Images