A method for controlling the action of an unmanned aerial vehicle air navigation visual language with active dialogue capability

Abstract

The application provides an unmanned aerial vehicle air navigation visual language action control method with active dialogue capability, and belongs to the technical field of unmanned aerial vehicle navigation. Through LLM processing of dialogue history data and extraction of image features by a visual encoder, spatial and time sequence features are obtained in combination with observation history data, a PNaQ module is introduced to realize displacement updating, termination judgment and active questioning, and a one-time data enhancement technology is matched to expand the data set, so that the problems of space-time relationship confusion and lack of active questioning in long-range navigation are solved, and the navigation accuracy and interactivity are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicle navigation, and particularly relates to a visual language action control method for aerial navigation of unmanned aerial vehicles with active dialogue capability. BACKGROUND

[0002] The goal of visual language navigation (VLN) is to enable an agent to autonomously navigate according to human language instructions and visual observations.

[0003] Since the inherent ambiguity of language description usually makes a single instruction insufficient to accurately locate the target position, a visual dialogue navigation (VDN) method is needed, such as Thomason et al. who pioneered dialogue-aware navigation by integrating human dialogue history as text input, and Fan et al. who extended this paradigm to unmanned aerial vehicle navigation.

[0004] However, the current unmanned aerial vehicle navigation has the following problems, the main causes of which include:

[0005] 1. Under long-range navigation tasks, the historical spatio-temporal relationship is chaotic: long-range flight of unmanned aerial vehicles leads to significant changes in perspective and scale, existing methods often stack observation history in video sequence order or only make coarse-grained memory, lack of unified spatial reference system and explicit spatial structure modeling, making it difficult to correspond "what has been observed" to "global position", and thus causing chaotic historical spatio-temporal relationship, difficulty in backtracking retrieval, and untimely deviation correction after deviation.

[0006] 2. Lack of active questioning ability: existing models rely on pre-given dialogue history or template-based dialogue generation, lack explicit evaluation mechanism for navigation uncertainty, and cannot initiate clarification and supplementary inquiries actively when instructions are ambiguous or trajectories deviate, thus easily continuing to explore errors in complex environments.

[0007] 3. Coupling of action prediction and language generation causes mutual interference: the optimization objectives and output forms of the two types of tasks are obviously different; if a single fusion feature or a single prediction branch simultaneously undertakes action and language output, it is easy to cause representation and gradient conflict, resulting in unstable action decision or unreasonable questioning timing / content, thus reducing the overall navigation efficiency and success rate. SUMMARY

[0008] The application proposes a visual language action control method for aerial navigation of unmanned aerial vehicles with active dialogue capability, which can solve the above problems.

[0009] To solve the above problems, the application proposes the following technical content:

[0010] A visual language action control method for aerial navigation of unmanned aerial vehicles with active dialogue capability, comprising the following steps:

[0011] S1: Obtain the dialogue history encoding vector Etext,t of the current time t by embedding the dialogue history data of the UAV through the embedding layer of the large language model LLM; obtain the image feature vt of the current frame image by extracting the feature of the current frame image It of the UAV through the visual encoder, and flatten and linearly project it to the LLM space to obtain the visual encoding vector Evis,t of the current time t;

[0012] S2: Based on the observation history data of the UAV, the observation history data is projected to the ground by orthographic re-projection and then spliced to obtain the spliced image Mt of the current time t; the spliced image Mt of the current time t is taken to the minimum circumscribed rectangle, and the invalid area is filled with 0 operation to obtain the map Rt of the current time t with uniform size; the spatial feature of the current time t is obtained by first performing visual feature extraction on the map Rt of the current time t and then performing flattening and encoding operation; in the formula, Ik represents the kth frame image; Tk represents the pose of the camera in the kth frame image;

[0013] S3: First, based on the historical image feature sequence vt-1 of the current time t, the feature sequence in the time dimension is obtained; then, based on the map Rt with uniform size, the global map encoding feature of the current time is extracted; subsequently, a multi-granularity attention mechanism is adopted to link and finally obtain the time sequence feature of the historical exploration;

[0014] S4: Introduce the progress-driven navigation-question selection module PNaQ, PNaQ introduces two special word elements [act] and [que], [act] is used for action prediction, and [que] is used for language generation; four learnable word elements are appended after [act]:, which combines Etext,t, Evis,t, and to form an input sequence Dt; input the input sequence Dt into the LLM to output, which jointly manages the behavior of the UAV, and updates the horizontal and vertical displacement vectors of the UAV using ; predict the termination probability of the UAV, and determine whether the UAV stops navigation according to the set threshold; evaluate the navigation progress using, and determine whether to ask a question based on the set rules; when a question is asked, the external response module returns an answer, and the answer is updated in the dialogue history data in S1;

[0015] S5: Based on one-time data augmentation, the new "dialogue-action-trajectory triplets" predicted by LLM are included in the augmented data set, and are sampled at a set proportion in each training round.

[0016] Further, in step S1, the dialogue history encoding vector Etext,t of the current time t is obtained by embedding the dialogue history data of the UAV through the embedding layer of the large language model LLM; E text,t , the formula is:

[0017]

[0018] where, X text,t denotes the dialogue history data at the current time t ; Embed (*) denotes an embedding function operation; E text,t denotes the dialogue history encoding vector at the current time t , L is the token length; D is LLM the dimension of the hidden space;

[0019] the visual encoding vector at the current time t E vis,t , the formula is:

[0020]

[0021] where, E vis,t denotes the visual encoding vector of the current frame image I t ; reshape (*) denotes a flattening operation; v t denotes the image feature of the current frame image I t , extracted by the visual encoder; W v denotes the weight matrix of the image feature projection to LLM the hidden space; b v denotes the bias term of the weight matrix W v ; N denotes the number of visual tokens; D is LLM the dimension of the hidden space.

[0022] Further, step S2 includes the following steps:

[0023] S2.1: using orthographic re-projection to project the observation history data to the ground and then stitching to obtain the stitched image t at the current time M t ; the formula is:

[0024]

[0025] where, I k denotes the image of the k th frame; T k denotes the​k The camera pose in a frame image; K Indicates camera intrinsic parameters; Indicates the ground reference plane; Indicates will I k in accordance with T k 、K、 Orthorectified reprojection onto the ground coordinate system; Indicates the first k A single-frame bird's-eye view map obtained by orthorectifying a frame image; Fuse (*) indicates an operation that stitches together a bird's-eye view map of all frames according to ground coordinates and performs weighted fusion in overlapping areas; M t Indicates the current time t spliced ​​images;

[0026] S2.2: For the current moment t spliced ​​images M t The current time step is obtained by taking the smallest bounding rectangle and filling the invalid region with zeros. t Uniform size map R t The formula is:

[0027]

[0028] In the formula, ZeroPad (*) indicates the operation of filling with zeros; MBR (*) indicates the minimum bounding rectangle operation; R t Indicates the current time t A map of uniform size;

[0029] S2.3: For the current moment t Uniform size map R t First, visual features are extracted, and then flattening and encoding operations are performed to obtain the current time step. t Spatial features The formula is:

[0030]

[0031] In the formula, f v (*) indicates a visual feature extraction operation; flatten (*) indicates the flattening operation; Indicates the encoding function; N S express The number of visual lexical elements; D for LLM The dimensions of hidden space; Indicates the current time t Spatial characteristics.

[0032] Furthermore, step S3 includes the following steps:

[0033] S3.1: Based on the current time t The historical image feature sequence is used to obtain its feature sequence in the time dimension; the formula is:

[0034]

[0035] In the formula, v 1:t-1 Indicates the current time t Historical image feature sequences; pooling 2D (*) indicates that the mean is calculated along the spatial dimension; flatten (*) indicates the flattening operation; Represents a feature sequence in the time dimension;

[0036] S3.2: Set the current time t Uniform size map R t The AnyRes image encoder is used to obtain its global map coding features, and the formula is as follows:

[0037]

[0038] In the formula, Indicates the current time t Global map coding features; R t Indicates the current time t A map of uniform size; This represents the encoding function of the AnyRes image encoder;

[0039] S3.3: For the first i The clipping formula for a navigation waypoint at time t is:

[0040]

[0041] In the formula, d j This indicates the set clipping radius; Indicates in In the middle, with the first i Using each navigation waypoint as the center, d j Trim based on the trim radius; represents the cropping map of the i-th navigation point at the current time t; i

[0042] S3.4: At the current time t, the feature of each navigation point in the time dimension is adopted with the cross-attention operation of the cropping map of the i-th navigation point to obtain the feature map of the i-th navigation point after the cross-attention operation, and the formula is: i i

[0043]

[0044] In the formula, represents the feature of the i-th navigation point in the time dimension; represents the cross-attention operation; i represents the feature map of the i-th navigation point after the cross-attention operation at the current time t; cross represents the cropping map of the i-th navigation point at the current time t; atten S3.5: Set i , integrate multi-scale feature information and enhance the representation density; the formula is: i

[0045] S3.6: The feature sizes of all historical navigation points are concatenated, and the time sequence feature at the current time t is obtained through encoding processing; the formula is:

[0046]

[0047] In the formula, represents the feature map of the i-th navigation point after the cross-attention operation at the current time t; represents the feature of the i-th navigation point in the time dimension at the current time t; i represents the feature of the i-th navigation point in the time dimension at the current time t; represents the feature of the i-th navigation point in the time dimension at the current time t; t represents the feature of the i-th navigation point in the time dimension at the current time t; d j represents the feature of the i-th navigation point in the time dimension at the current time t; i represents the feature of the i-th navigation point in the time dimension at the current time t; represents the learnable parameter item of the i-th navigation point; i represents the feature size of the i-th navigation point at the current time t; represents the feature size of the i-th navigation point at the current time t; i S3.6: The feature sizes of all historical navigation points are concatenated, and the time sequence feature at the current time t is obtained through encoding processing; the formula is: t i

[0048] S3.6: The feature sizes of all historical navigation points are concatenated, and the time sequence feature at the current time t is obtained through encoding processing; the formula is: t

[0049]

[0050] ​​​​​​​​​​​​wherein denotes the total number of navigation points; denotes the current time t , the feature size of the 1st~t-1th navigation point; D is the dimension of the LLM hidden space, denotes the encoding function.

[0051] Further, step S4 comprises the following steps:

[0052] S4.1: inputting the sequence of four learnable tokens E text,t , E vis,t 、 、 and the input sequence of four learnable tokens D t ; the formula is:

[0053]

[0054] wherein and denote two tokens of PnaQ; denotes an additional learnable token;

[0055] S4.2: inputting the input sequence D t to the LLM, and outputting from the LLM; the formula is:

[0056]

[0057] wherein LLM (*) denotes processing by LLM ; denotes the hidden state for horizontal displacement prediction; denotes the hidden state for vertical displacement prediction; denotes the hidden state for termination probability prediction; denotes the hidden state for navigation progress prediction;

[0058] S4.3: updating the horizontal and vertical displacement vectors of the UAV using ;

[0059] the formula is:

[0060]

[0061] wherein and respectively represent the horizontal displacement vector prediction MLP layer and the vertical displacement vector prediction MLP layer; represents the current time t , the output horizontal and vertical displacement vector of the UAV;

[0062] S4.4: for predicting the termination probability; the formula is:

[0063]

[0064] wherein, represents the sigmoid function; is the termination probability prediction MLP layer; represents the current time t , the termination probability of the UAV;

[0065] S4.5: determine the navigation progress, and judge whether to generate a question sentence according to the navigation progress; if a question sentence is generated, return an answer through an external response module, update the returned answer in the dialogue history data in S1, and repeat S1-S4; if a question sentence is not generated, the UAV continues to move until a question sentence is generated;

[0066] The UAV uses to estimate the navigation progress, and the formula is:

[0067]

[0068] wherein, is the navigation progress prediction MLP layer; represents the sigmoid function; represents the estimated navigation progress; the meaning of the navigation progress is that the distance of the UAV to the starting point of the trajectory accounts for the ratio of the total distance of the UAV to the starting point and the ending point of the trajectory.

[0069] Further, when is established, the language head of the UAV automatically generates a question sentence, the external response module returns an answer, and the answer is updated in the dialogue history data in S1, and the UAV updates its own position; when is not established, the position of the UAV is directly updated; when , the UAV stops; otherwise, cycle S1-S5; in the process of the cycle, the UAV realizes navigation by continuously asking questions and returning answers.

[0070] Further, step S5 includes the following steps:

[0071] S5.1: after the UAV completes the samples of the current training set V , the UAV executes the action of the model and moves from the current position to the predicted position ;

[0072] S5.2: Calculate the optimal action that makes the UAV trajectory closer to the true trajectory for the predicted position ; Then pack the "dialogue-action-trajectory" three groups of information into a triple; The meaning of "predicted position" is the current UAV predicted position to the manually annotated position The horizontal and vertical displacement vectors required for the straight-line distance;

[0073] S5.3: The generated triple is saved to the augmented dataset buffer V g ;

[0074] S5.4: Before each round of training iteration, samples are extracted from the manually annotated dataset V 0 and the augmented dataset V g According to the pre-set proportion;

[0075] S5.5: The extracted samples are combined into a new training set V 新 for this round of training, and the cycle S5.1-S5.5 replaces the training set of S5.1 V 新 . V

[0076] The beneficial effects achieved by using the above technical content are:

[0077] 1、This method fully integrates dialogue history data and image features, and obtains spatial and temporal features at the current time based on the observation history of the UAV, fully considering time and space, and avoiding the problem of chaotic historical spatio-temporal relationship in long-range navigation tasks. t

[0078] 2、The method uses learnable tokens and actively asks questions based on the navigation process, and updates the dialogue history data with an external response module to realize the interaction between the UAV and the person.

[0079] 3、The four learnable tokens are used to estimate the navigation progress and actively ask questions, and the horizontal and vertical displacement vectors of the UAV are used to update the action and language, which are routed by different features to avoid interference.

[0080] ​​​4. One-time data augmentation expands the dataset. This method can ensure the quality, accuracy, and usability of the newly added dataset as much as possible. At the same time, through the set sampling strategy, it ensures that a portion of the original data or the data with the least disturbance is always retained in the training set, avoiding the distribution of the training data deviating from the true situation due to too much newly generated data, thereby affecting the model performance. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the model's modules;

[0082] Figure 2 This is a schematic diagram of the global map and cross-attention;

[0083] Figure 3 This is a diagram illustrating a drone proactively making inquiries.

[0084] Figure 4 This is a diagram illustrating how a drone actively inquires about navigation progress.

[0085] Figure 5 This is a schematic diagram of a one-time enhancement technology. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] like Figure 1 As shown, a visual language action control method for UAV aerial navigation with proactive dialogue capability specifically includes the following steps:

[0088] S1: Encoding of UAV multimodal data. This multimodal data includes: dialogue history data, current frame image, and observation history data, specifically including the following steps:

[0089] S1.1: Obtain the current moment from the drone's dialogue history data through the embedding layer of a Large Language Model (LLM). t The "dialogue history encoding vector". The formula is:

[0090]

[0091] In the formula, X text,t Indicates the current time t Dialogue history data; Embed (*) indicates an embedded function operation; Etext,t represents the current time t dialog history encoding vector, which can be LLM used directly by the backbone network for inference feature representation; L is the label length; D is LLM the dimension of the hidden space.

[0092] S1.2: The current frame image of the UAV I t is extracted by the visual encoder, flattened and linearly projected into LLM space, obtaining the "visual encoding vector" of the current time t . The formula is:

[0093]

[0094] In the formula, E vis,t represents the visual encoding vector of the current frame image I t ; reshape (*) represents the flattening operation; v t represents the image feature of the current frame image I t extracted by the visual encoder; W v represents the weight matrix of the image feature projected into LLM space; b v represents the bias term of the weight matrix W v ; N represents the number of visual labels; D is LLM the dimension of the hidden space.

[0095] S2: Based on the observation history data of the UAV , the spatial feature t of the current time is obtained. Specifically, the following steps are included:

[0096] S2.1: In order to eliminate perspective distortion and provide a unified ground coordinate system, the observation history data is projected to the ground after orthographic re-projection and then spliced, obtaining the spliced image t of the current time M t . The formula is:

[0097]

[0098] In the formula, I k represents the firstk image of a frame; T k representing the first k pose of the camera in the frame image; K representing the camera intrinsic parameters; representing the ground reference plane; representing the operation of I k according to T k 、K、 orthographic re-projection to the ground coordinate system;

[0099] representing the first k orthographic re-projected single frame aerial view map of the frame image; Fuse (*) represents the operation of splicing all frames of aerial view maps according to the ground coordinates, and weighted fusion in the overlapping area; M t representing the spliced image at the current time t

[0100] S2.2: extracting the spatial features of the spliced image at the current time t M t by taking the minimum bounding rectangle, and filling 0 operation to the invalid area, to obtain the uniform size map at the current time t R t The formula is:

[0101]

[0102] In the formula, ZeroPad (*) represents the filling 0 operation; MBR (*) represents the minimum bounding rectangle operation; R t representing the uniform size map at the current time t

[0103] S2.3: extracting the spatial features of the uniform size map at the current time t R t , and then through the flattening and encoding operation, to obtain the spatial features of the current time t The formula is:

[0104]

[0105] In the formula, f v (*) represents the visual feature extraction operation; flatten ​​​​​​(*) denotes flattening operation; denotes the encoding function; N S denotes the number of visual words; D is LLM the dimension of the hidden space; denotes the spatial feature of the current time t .

[0106] S3: Firstly, based on the historical image feature sequence of the current time t , obtain its feature sequence in the time dimension; then, based on the unified size map R t , extract the global map encoding feature of the current time; subsequently, adopt the multi-granularity attention mechanism, link and , and finally obtain the time sequence feature of historical exploration . Specifically, the following steps are included:

[0107] S3.1: Based on the historical image feature sequence of the current time t , obtain its feature sequence in the time dimension; the formula is:

[0108]

[0109] In the formula, v 1:t-1 denotes the historical image feature sequence of the current time t , that is, the feature sequence composed of the first frame to the t-1 frame image, which obtains the feature after extracting each frame image through the visual encoder in step 2; pooling 2D (*) denotes mean value calculation along the spatial dimension; flatten (*) denotes flattening operation; denotes the feature sequence in the time dimension.

[0110] S3.2: Based on the unified size map t of the current time R t , adopt the AnyRes image encoder to obtain its global map encoding feature. The formula is:

[0111]

[0112] In the formula, denotes the global map encoding feature of the current time t ; R t denotes the unified size map of the current time t ; This represents the encoding function of the AnyRes image encoder.

[0113] S3.3: Employs a multi-granularity attention mechanism to... and Link, that is: link the current moment t Historical image feature sequences navigation waypoints Projected to The position on the feature map is obtained. The meaning of navigation waypoints is: the coordinates of the unmanned aerial vehicle (UAV). In each frame of historical image features, there is one and only one coordinate for the UAV. See the specific projected image for details. Figure 2 .exist Perform cropping to obtain the current time. t A cropped image of each navigation waypoint.

[0114] For the i The clipping formula for a navigation waypoint at time t is:

[0115]

[0116] In the formula, d j This indicates the set clipping radius; Indicates in In the middle, with the first i Using each navigation waypoint as the center, d j Trim based on the trim radius. Represents the current time t, the nth i A cropped image of navigation waypoints.

[0117] S3.4: At the current time t, combine the time-dimensional features of each navigation waypoint with its i-th... i The cropped image of the nth navigation waypoint is obtained by performing a cross-attention operation to obtain the nth navigation waypoint image after the cross-attention operation. i Feature map of navigation waypoints.

[0118] The formula is:

[0119]

[0120] In the formula, Indicates the first i The time-dimensional characteristics of navigation waypoints; cross _ atten This indicates a cross-attention operation; This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Represents the current time t, the nthi A cropped image of navigation waypoints.

[0121] S3.5: Settings This integrates multi-scale feature information and enhances representation density. The formula is:

[0122]

[0123] In the formula, This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Indicates the current time t ,exist d j At the scale, the first i Feature representation of each navigation waypoint; Indicates the first i The time-dimensional characteristics of navigation waypoints; Indicates the first i Each navigation waypoint can learn a set of parameters; Indicates the current time t , No. i The characteristic dimensions of each navigation waypoint.

[0124] S3.6: Concatenate the feature dimensions of all historical navigation waypoints and process them through encoding to obtain the current time. t Temporal characteristics The formula is:

[0125]

[0126] In the formula, Indicates the total number of navigation points; Indicates the current time t The feature dimensions of the first to the (t-1)th navigation waypoints; D For the dimension of the LLM hidden space, This represents the encoding function.

[0127] S4: Introduces the progress-driven navigation-query selection module PNaQ. PNaQ introduces two special lexical units, [act] and [que], where [act] is used for action prediction and [que] is used for language generation. [act] is followed by four learnable lexical units: , representing horizontal displacement, vertical displacement, termination probability, and navigation progress prediction terms, respectively, combined with E text,t , E vis,t 、 , Composition of input sequenceD t ; input sequence D t into the LLM, output the behavior of the drone. Specifically, the steps include:

[0128] S4.1: input sequence E text,t , E vis,t 、 , into the LLM, output D t ; the formula is:

[0129]

[0130] S4.2: input sequence D t into the LLM, output ; the formula is:

[0131]

[0132] wherein, respectively, represent the hidden state of the LLM after processing ; wherein, represents the hidden state for horizontal displacement prediction; represents the hidden state for vertical displacement prediction; represents the hidden state for termination probability prediction; represents the hidden state for navigation progress prediction.

[0133] S4.3: update the horizontal and vertical displacement vectors of the drone using .

[0134] The formula is:

[0135] wherein, and respectively represent the horizontal displacement vector prediction MLP layer and the vertical displacement vector prediction MLP layer; represents the output horizontal and vertical displacement vectors of the drone at the current time t .

[0136] S4.4: for predicting the termination probability; the formula is:

[0137]

[0138] wherein, sigmoid function; is the termination probability prediction MLP layer; denotes the current time t , the termination probability of the UAV.

[0139] S4.5: Determine the navigation progress, and determine whether to generate a question according to the navigation progress. If a question is generated, return an answer through an external response module, update the returned answer in the dialogue history data in S1, and repeat S1-S4. If a question is not generated, the UAV continues to move until a question is generated. See Figure 3 and Figure 4 .

[0140] The UAV uses to estimate the navigation progress, and the formula is:

[0141]

[0142] wherein, is the navigation progress prediction MLP layer; sigmoid function; denotes the estimated navigation progress. The meaning of the navigation progress is that the distance of the UAV to the starting point of the trajectory accounts for the ratio of the total distance of the UAV to the starting point and the ending point of the trajectory.

[0143] When is true, the language head of the UAV automatically generates a question, the external response module returns an answer, and the answer is updated in the dialogue history data in S1, and the UAV updates its own position.

[0144] When is not true, the position of the UAV is directly updated.

[0145] The position update formula of the UAV is:

[0146] Let the current position of the UAV be G t [ x t , y t , z t ], then the position of the UAV at the next time is:

[0147]

[0148] wherein, G t denotes the position of the UAV at the current time t; G t+1 denotes the position of the UAV at time t+1; Indicates the current time t The drone outputs horizontal and vertical displacement vectors.

[0149] when When the condition is met, the drone stops; otherwise, it cycles through S1-S5. During this cycle, the drone navigates by continuously asking questions and returning answers.

[0150] S5: Based on One-Time Data Augmentation (OTDA), this method provides a large number of samples during the model training phase; it incorporates new "dialogue-action-trajectory triples" generated by LLM predictions into the augmented dataset and samples them at a set ratio in each training round; see details. Figure 5 Specifically, it includes the following steps:

[0151] S5.1: The drone completes the current training set. V After the sample, that is, the drone executes the model's actions. And move from the current location to the predicted location. ;

[0152] S5.2: For predicted location Calculate the optimal action to make the drone's trajectory closer to the real trajectory. Then, the three sets of information, "dialogue-action-trajectory", are packaged into a triple. The meaning is: the current predicted location of the drone. To the manually marked location The horizontal and vertical displacement vectors required for the straight-line distance;

[0153] S5.3: The generated triples will be saved to the augmented dataset buffer. V g middle.

[0154] S5.4: Before each training iteration, the dataset will be manually labeled. V 0 and augmented datasets V g In this process, samples are drawn according to a pre-set ratio (e.g., 8:2);

[0155] S5.5: The extracted samples will be combined to form a new training set. V 新 Used for this round of training, looping S5.1-S5.5, using V 新 Training set to replace S5.1 V The purpose of this sampling strategy is to ensure that a portion of the original data or the least disturbed data is always retained in the training set, avoiding excessive newly generated data that could cause the distribution of the training data to deviate from the true situation, thereby affecting model performance.

[0156] With the above ideal embodiments according to the present application as the inspiration, through the above description, relevant staff can definitely make various changes and modifications without deviating from the scope of the technical idea of the present application. The technical scope of the present application is not limited to the content in the specification, and must be determined according to the scope of claims.

Claims

1. A visual language action control method for aerial navigation of unmanned aerial vehicles (UAVs) with proactive dialogue capabilities, characterized in that, Includes the following steps: S1: The drone's dialogue history data is processed through the embedding layer of a large language model (LLM) to obtain the current moment. t Dialogue history encoding vector E text,t ; the current frame image of the drone I t The image features of the current frame are obtained by extracting features through a visual encoder. v t And flatten and linearly projected onto LLM Space, to obtain the current moment t Visual encoding vector E vis,t ; S2: Historical observation data based on UAVs The historical observation data is projected onto the ground using orthophoto reprojection and then stitched together to obtain the current time. t spliced ​​images M t ; for the current moment t spliced ​​images M t The current time step is obtained by taking the smallest bounding rectangle and filling the invalid region with zeros. t Uniform size map R t ; for the current moment t Uniform size map R t First, visual features are extracted, and then flattening and encoding operations are performed to obtain the current time step. t Spatial features In the formula, I k Indicates the first k The image of the frame; T k Indicates the first k The camera pose in a frame image; S3: First, based on the current time... t Historical image feature sequence v 1:t-1 To obtain its feature sequence in the time dimension. Then, based on a map of uniform size R t Extract the global map coding features at the current moment. ; Subsequently, a multi-granularity attention mechanism was employed, through... and Linking these elements and ultimately revealing the temporal characteristics of historical exploration. ; S4: Introduces the progress-driven navigation-query selection module PNaQ. PNaQ introduces two special lexical units, [act] and [que], where [act] is used for action prediction and [que] is used for language generation. [act] is followed by four learnable lexical units: , combined E text,t , E vis,t 、 、 Composition of input sequence D t ; Input sequence D t Input into LLM, output Jointly manage the behavior of drones, and utilize Update the drone's horizontal and vertical displacement vectors; utilize Predict the probability of drone termination, and determine whether the drone should stop navigation based on a set threshold; utilize... Assess navigation progress, determine whether to ask a question based on the set rules, and when a question is asked, the external response module returns an answer and updates the dialogue history data in S1 with the answer; S5: Based on one-time data augmentation, the new "dialogue-action-trajectory triples" generated by LLM predictions are incorporated into the augmented dataset and sampled at a set ratio in each round of training.

2. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, In step S1, Current moment t Dialogue history encoding vector E text,t The formula is: ； In the formula, X text,t Indicates the current time t Dialogue history data; Embed (*) indicates an embedded function operation; E text,t Indicates the current time t The dialogue history encoding vector, L For the length of the marker; D for LLM The dimensions of hidden space; Current moment t Visual encoding vector E vis,t The formula is: ； In the formula, E vis,t Indicates the current frame image I t The visual encoding vector; reshape (*) indicates the flattening operation; v t Indicates the current frame image I t Image features extracted by a visual encoder; W v Represents image features projected onto LLM The weight matrix of the space; b v Represents the weight matrix W v The bias term; N Indicates the number of visual markers; D for LLM The dimensions of hidden space.

3. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, Step S2 includes the following steps: S2.1: The historical observation data is projected onto the ground using orthophoto reprojection and then stitched together to obtain the current time. t spliced ​​images M t The formula is: ； In the formula, I k Indicates the first k The image of the frame; T k Indicates the first k The camera pose in a frame image; K Indicates camera intrinsic parameters; Indicates the ground reference plane; Indicates will I k in accordance with T k 、K、 Orthorectified reprojection onto the ground coordinate system; Indicates the first k A single-frame bird's-eye view map obtained by orthorectifying a frame image; Fuse (*) indicates an operation that stitches together a bird's-eye view map of all frames according to ground coordinates and performs weighted fusion in overlapping areas; M t Indicates the current time t spliced ​​images; S2.2: For the current moment t spliced ​​images M t The current time step is obtained by taking the smallest bounding rectangle and filling the invalid region with zeros. t Uniform size map R t The formula is: ； In the formula, ZeroPad (*) indicates the operation of filling with zeros; MBR (*) indicates the minimum bounding rectangle operation; R t Indicates the current time t A map of uniform size; S2.3: For the current moment t Uniform size map R t First, visual features are extracted, and then flattening and encoding operations are performed to obtain the current time step. t Spatial features The formula is: ； In the formula, f v (*) indicates a visual feature extraction operation; flatten (*) indicates the flattening operation; Indicates the encoding function; N S express The number of visual lexical elements; D for LLM The dimensions of hidden space; Indicates the current time t Spatial characteristics.

4. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, Step S3 includes the following steps: S3.1: Based on the current time t The historical image feature sequence is used to obtain its feature sequence in the time dimension; the formula is: ； In the formula, v 1:t-1 Indicates the current time t Historical image feature sequences; pooling 2D (*) indicates that the mean is calculated along the spatial dimension; flatten (*) indicates the flattening operation; Represents a feature sequence in the time dimension; S3.2: Set the current time t Uniform size map R t The AnyRes image encoder is used to obtain its global map coding features, and the formula is as follows: ； In the formula, Indicates the current time t Global map coding features; R t Indicates the current time t A map of uniform size; This represents the encoding function of the AnyRes image encoder; S3.3: For the first i The clipping formula for a navigation waypoint at time t is: ； In the formula, d j This indicates the set clipping radius; Indicates in In the middle, with the first i Using each navigation waypoint as the center, d j Trim based on the trim radius; Represents the current time t, the nth i A cropped image of navigation waypoints; S3.4: At the current time t, combine the time-dimensional features of each navigation waypoint with its i-th... i The cropped image of the nth navigation waypoint is obtained by performing a cross-attention operation to obtain the nth navigation waypoint image after the cross-attention operation. i The feature map of each navigation waypoint is given by the formula: ； In the formula, Indicates the first i The time-dimensional characteristics of navigation waypoints; cross _ atten This indicates a cross-attention operation; This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Represents the current time t, the nth i A cropped image of navigation waypoints; S3.5: Settings It integrates multi-scale feature information and enhances representation density; the formula is: ； In the formula, This represents the current time t, after the cross-attention operation. i Feature map of each navigation waypoint; Indicates the current time t ,exist d j At the scale, the first i Feature representation of each navigation waypoint; Indicates the first i The time-dimensional characteristics of navigation waypoints; Indicates the first i Each navigation waypoint can learn a set of parameters; Indicates the current time t , No. i The characteristic dimensions of each navigation waypoint; S3.6: Concatenate the feature dimensions of all historical navigation waypoints and process them through encoding to obtain the current time. t Temporal characteristics The formula is: ； In the formula, Indicates the total number of navigation points; Indicates the current time t The feature dimensions of the first to the (t-1)th navigation waypoints; D For the dimension of the LLM hidden space, This represents the encoding function.

5. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 1, characterized in that, Step S4 includes the following steps: S4.1: Will E text,t , E vis,t 、 、 The input sequence consists of four learnable lexical units. D t The formula is: ； In the formula, and Two morphemes representing PnaQ; Indicates additional learnable lexical units; S4.2: Input sequence D t Input into LLM, output from LLM ; The formula is: ； In the formula, LLM (*) indicates that by LLM deal with; This represents the hidden state used for horizontal displacement prediction; This represents the hidden state used for vertical displacement prediction; This represents the hidden state used to terminate probability prediction; This represents the hidden state used for navigation progress prediction; S4.3: Utilize Update the drone's horizontal and vertical displacement vectors; The formula is: ； In the formula, and These represent the horizontal displacement vector prediction MLP layer and the vertical displacement vector prediction MLP layer, respectively. Indicates the current time t The drone's output horizontal and vertical displacement vectors; S4.4: Used to predict the termination probability; the formula is: ； In the formula, Represents the sigmoid function; For terminating probability prediction MLP layers; Indicates the current time t The probability of drone termination; S4.5: Determine the navigation progress and determine whether a question is generated based on the navigation progress. If a question is generated, return an answer through the external response module, update the dialogue history data in S1 with the returned answer, and repeat S1-S4; if no question is generated, the drone continues to move until a question is generated. Drones use To estimate navigation progress, the formula is: ； In the formula, MLP layer for navigation progress prediction; Represents the sigmoid function; This indicates the estimated navigation progress; the navigation progress is defined as the ratio of the distance from the UAV to the starting point of the trajectory to the total distance from the UAV to both the starting and ending points of the trajectory.

6. The UAV aerial navigation visual language action control method with active dialogue capability according to claim 5, characterized in that, when Upon setup, the drone's speech engine automatically generates a question, the external response module returns an answer, and updates the dialogue history data in S1 with the answer. The drone also updates its own position. If the condition is not met, update the drone's position directly; when... When the drone stops, it stops; otherwise, it cycles through S1-S5. During the loop, the drone navigates by continuously asking questions and returning answers.

7. A visual language action control method for UAV aerial navigation with active dialogue capability according to claim 5, characterized in that, Step S5 includes the following steps: S5.1: The UAV completes the current training set V After the sample, that is, the drone executes the model's actions. and move from the current location to the predicted location. ; S5.2: For predicted location Calculate the optimal action to make the drone's trajectory closer to the real trajectory. Then, the three sets of information, "dialogue-action-trajectory", are packaged into a triple; The meaning is: the current predicted location of the drone. To the manually marked location The horizontal and vertical displacement vectors required for the straight-line distance; S5.3: The generated triples will be saved to the augmented dataset buffer. V g middle; S5.4: Before each training iteration, the dataset will be manually labeled. V 0 and augmented datasets V g In this process, samples are drawn according to a pre-set ratio; S5.5: The extracted samples will be combined to form a new training set. V 新 Used for this round of training, looping S5.1-S5.5, using V 新 Training set to replace S5.1 V .

Images