Multi-modal fusion unmanned aerial vehicle intelligent interaction control method

Abstract

The invention relates to the field of unmanned aerial vehicle control, and discloses a multi-modal fusion unmanned aerial vehicle intelligent interaction control method, which comprises the following steps: S1, receiving an interaction instruction, and carrying out analysis and expected generation; s2, executing fine granularity perception and consistency verification; the unmanned aerial vehicle end constructs an actual observation semantic distribution map at the current moment based on the environment data; the unmanned aerial vehicle end compares the actual observation semantic distribution map with the expected nine-sector semantic distribution map, and calculates an expected-observation consistency score between the actual observation semantic distribution map and the expected nine-sector semantic distribution map; S3, constructing a semantic weight topology memory index chain; and S4, executing navigation failure judgment and self-adaptive backtracking. The multi-mode instruction is analyzed through a base station end large language model, a fuzzy natural language or gesture is converted into a specific verifiable visual expectation standard, and the effect that closed-loop verification is conducted on the instruction execution condition through environment semantics while flight control is executed is achieved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, specifically to a multimodal fusion-based intelligent interactive control method for UAVs. Background Technology

[0002] With the development of artificial intelligence and robotics, the interaction mode of drones is shifting from traditional remote control operation to more intelligent multimodal interaction. Current drone control systems can receive users' natural language voice commands or gesture signals, using algorithms such as speech recognition and visual analysis to convert these inputs into specific flight action commands. At the mission execution level, drones typically rely on Simultaneous Localization and Mapping (SLAM) technology and onboard sensors to perceive the environment, combining pre-built maps or real-time perception data to complete flight tasks such as path planning and target tracking. Some advanced systems are also beginning to introduce large language models, decomposing complex user commands into specific task sequences to achieve higher levels of automated control.

[0003] However, existing UAV interactive control technologies still have limitations in practical applications. First, current control logic mainly focuses on converting commands into mechanical motion coordinates or velocity vectors, lacking a mechanism for semantic-level visual anticipation of command execution results. The system cannot verify whether the scene features in the current field of view truly match the user's interaction intent, making it difficult to guarantee the accuracy of task execution in unstructured environments. Second, when facing complex dynamic scenes, existing visual navigation methods often rely on the recognition results of a single target or sparse feature points for guidance, lacking consistency verification of fine-grained spatial semantic distribution within the field of view. When target occlusion, lighting changes, or recognition errors occur, the system is prone to navigation misjudgments, leading to unstable control states. Furthermore, when UAVs encounter positioning loss or get stuck in a deadlock state in unfamiliar environments, existing recovery strategies usually only employ simple hovering or mechanical return along the original path, lacking autonomous backtracking decision-making capabilities based on environmental visual saliency and contextual relevance. This results in low efficiency in self-recovery and task replanning after a failure, often making it difficult to adapt to complex and changing operating environments. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a multimodal fusion-based intelligent interactive control method for unmanned aerial vehicles (UAVs). This method solves the problems of low task execution accuracy caused by the lack of semantic-level visual expectation verification of command intent in existing UAV interactive control systems, and the difficulty in achieving efficient autonomous recovery when navigation anomalies occur in complex environments due to the lack of a backtracking strategy based on environmental salience.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal fusion-based intelligent interactive control method for unmanned aerial vehicles, comprising:

[0006] S1. Receive interactive instructions and parse and generate expected results: The base station receives the interactive instructions input by the user, uses a large language model to decompose the interactive instructions into a sequence of subtasks composed of action primitives and landmark phrases, and combines prior environmental information to generate the expected nine-sector semantic distribution map for the next moment.

[0007] S2. Perform fine-grained perception and consistency verification: The UAV receives the action primitive and controls the aircraft to perform the corresponding action, and collects environmental data in real time; the UAV constructs the actual observation semantic distribution map at the current moment based on the environmental data; the UAV compares the actual observation semantic distribution map with the expected nine-sector semantic distribution map and calculates the expected-observation consistency score between the two.

[0008] S3. Construct a semantic weight topology memory index chain: During the UAV's mission, the system records the UAV's pose data, actual observed semantic distribution map, and environmental saliency weights at each moment in a time sequence, forming a historical path memory array.

[0009] S4. Navigation Failure Judgment and Adaptive Backtracking: When the expected-observation consistency score is less than the preset safety threshold, the system determines that the navigation is abnormal and triggers the backtracking mode; the system selects the optimal backtracking anchor point from the historical path memory array based on the backtracking utility value, controls the UAV to return to the optimal backtracking anchor point and performs path replanning.

[0010] Preferably, in step S1, the generation of the expected nine-sector semantic distribution map specifically includes:

[0011] The system divides the field of view of the drone camera into a nine-grid structure, forming nine sectors;

[0012] The cloud generates an expected feature descriptor for each sector, which includes the expected landmark semantic label and the expected relative depth range value for the corresponding sector;

[0013] The landmark semantic tags are derived from landmark phrases or environmental background categories in the subtask sequence; the relative depth range value is calculated based on the object coordinates in the prior map and the current predicted trajectory of the UAV.

[0014] Preferably, in step S2, constructing the actual observation semantic distribution map at the current moment specifically includes:

[0015] The UAV uses an RGB-D depth camera to acquire RGB color images and depth images, and divides them into nine observation sectors corresponding to the spatial definition of the expected nine-sector semantic distribution map;

[0016] The airborne terminal constructs an observation feature vector for each observation sector. The observation feature vector includes the semantic labels of the objects actually identified in the corresponding sector and the measured average depth value.

[0017] The object semantic label is extracted from the RGB image region using an object detection algorithm, and the average depth value is calculated from the depth image region.

[0018] Preferably, in step S2, calculating the expected-observation consistency score between the two specifically includes:

[0019] For each sector, calculate the matching between its actual observed semantic labels and expected landmark semantic labels, and calculate the depth observation residual between the actual average depth value and the expected relative depth range value;

[0020] The expected-observation consistency score is obtained by calculating the weighted sum of all sectors based on the preset task attention weights, semantic matching results, and deep observation residuals.

[0021] The task attention weight is determined by the currently executed action instruction, and the expectation-observation consistency score decreases as the depth observation residual increases.

[0022] Preferably, in step S3, the historical path memory array consists of memory nodes arranged in a time sequence, and each memory node includes:

[0023] The six-degree-of-freedom pose data of the UAV at the time corresponding to the memory node;

[0024] The actual observation semantic distribution map corresponding to this moment;

[0025] The environmental saliency weight of the memory node;

[0026] The calculation of the environmental salience weight specifically includes:

[0027] For each sector, obtain its observation semantic label information gain value and observation depth value;

[0028] The weighted components of the corresponding sector are calculated based on the preset position weighting coefficient, the information gain value, and the observation depth value.

[0029] The environmental saliency weight of the memory node is obtained by summing the weight components of the nine sectors.

[0030] The position weighting coefficient has a higher value in the central region of the field of view than in the edge region, and the weight component is negatively correlated with the observation depth value.

[0031] Preferably, the rule for determining the information gain value is as follows:

[0032] If the observed semantic label belongs to a preset landmark category with navigation indication significance, a higher positive value is assigned;

[0033] If the observed semantic label belongs to the background category that lacks texture features or has high repetition, a lower positive value is assigned.

[0034] Preferably, in step S4, selecting the optimal backtracking anchor point from the historical path memory array based on the backtracking utility value specifically includes:

[0035] Traverse the memory nodes in the historical path memory array as candidate nodes, and calculate the backtracking utility value of each candidate node;

[0036] The retrospective utility value is calculated based on a weighted combination of the following three factors:

[0037] The environmental saliency weight of the candidate nodes;

[0038] The Euclidean distance from the current position to the pose of the candidate node;

[0039] The semantic similarity between the actual observed semantic distribution map at the current moment and the actual observed semantic distribution map recorded by the candidate node;

[0040] The retrospective utility value is positively correlated with the environmental saliency weight, negatively correlated with the Euclidean distance, and positively correlated with the semantic similarity.

[0041] The memory node with the largest backtracking utility value is selected as the optimal backtracking anchor point.

[0042] Preferably, the calculation of semantic similarity specifically includes:

[0043] Extract the observation feature vectors of each sector in the current moment's actual observation semantic distribution map;

[0044] Extract the observation feature vectors of each sector from the actual observation semantic distribution map of the candidate node records;

[0045] Calculate the cosine similarity or Jaccard similarity coefficient between the observed feature vectors of the corresponding sectors, and determine the overall semantic similarity based on the calculation results.

[0046] Preferably, in step S4, controlling the UAV to return to the optimal backtracking anchor point and perform path replanning specifically includes:

[0047] Extract the location coordinates stored in the optimal backtracking anchor point, plan a collision-free path from the current location to the location coordinates based on the local map, and control the drone to return;

[0048] After reaching the specified location coordinates, the drone collects real-time images and uploads them to the base station.

[0049] The base station uses a large language model to reassess the current environmental state and, combined with the regional information that caused the navigation anomaly, regenerates the sub-task sequence leading to the final goal.

[0050] Preferably, in step S1, receiving and parsing the interactive command specifically includes:

[0051] When the interactive command is in the form of voice, the audio stream is transcribed into a text sequence using automatic speech recognition technology;

[0052] When the interaction command is in the form of a gesture image, the gesture features are extracted using a visual encoder and mapped to the corresponding text or semantic label using a classifier.

[0053] The preprocessed text or semantic labels are input into a large language model, and the association between action primitives and landmark phrases is established through an attention mechanism.

[0054] This invention provides a multimodal fusion-based intelligent interactive control method for unmanned aerial vehicles (UAVs). It offers the following advantages:

[0055] 1. This invention uses a large language model at the base station to parse multimodal commands and generate a semantic distribution map of nine sectors containing spatial layout information. It transforms ambiguous natural language or gestures into specific and verifiable visual expectation standards, achieving the effect of closed-loop verification of command execution using environmental semantics while executing flight control. This solves the technical problem that traditional UAV interactive control relies solely on mechanical execution of coordinate commands and cannot autonomously verify whether scene features match user intent in unstructured environments.

[0056] 2. This invention constructs an actual observed semantic distribution map on the airborne end and calculates its fine-grained consistency score with the expected distribution map. It introduces task attention weight and depth observation residual to perform weighted evaluation of the matching results of different sectors, achieving the effect of accurately perceiving the spatial distribution of targets within the field of view and the matching degree of command description. It solves the technical problems of existing visual navigation methods in complex dynamic scenes, such as navigation judgment errors and unstable control caused by single target recognition errors, spatial position deviations or occlusions.

[0057] 3. This invention constructs a semantic topological memory index chain that includes environmental saliency weights, and selects the optimal backtracking anchor point based on the backtracking utility value when navigation is abnormal. It comprehensively considers the backtracking path cost, node environmental saliency, and visual context relevance, achieving the effect of autonomously finding a visually recognizable location for task replanning in the event of navigation failure or environmental disorientation. This solves the technical problem that UAVs are unable to efficiently and autonomously recover normal navigation status when encountering deadlock or positioning failure in unfamiliar environments. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method steps of the present invention;

[0059] Figure 2 This is a schematic diagram illustrating the instruction parsing and subtask generation of the present invention. Detailed Implementation

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Please see the appendix Figure 1 -Appendix Figure 2 This invention provides a multimodal fusion-based intelligent interactive control method for unmanned aerial vehicles (UAVs), comprising the following steps:

[0062] Step S1: Receive interactive instructions and parse and generate the expected results.

[0063] The base station receives natural language voice commands or gesture image data input by the user. For voice commands, the base station converts them into text; for gesture images, the drone or the base station identifies the meaning of the corresponding command.

[0064] The base station uses a pre-built large language model to perform semantic analysis on the commands, breaking them down into a sequence of subtasks consisting of action primitives and landmark phrases. Action primitives indicate the UAV's flight actions, while landmark phrases indicate navigation target references.

[0065] Based on the decomposed sub-task sequence and combined with prior environmental information, the base station predicts the scene features that should appear in the field of view after the drone performs the next action. The base station generates a semantic distribution map of nine sectors, including expected landmark semantic labels and expected relative depth ranges.

[0066] Step S2: Perform fine-grained awareness and consistency verification.

[0067] The drone receives action primitives and controls the aircraft to perform corresponding actions. During execution, the drone uses an RGB-D depth camera to collect environmental data in real time.

[0068] The drone divides the acquired field-of-view image into a 3×3 grid. The drone then extracts semantic and depth information of objects within each area to construct a semantic distribution map of the actual observation at the current moment.

[0069] The UAV compares the actual observed semantic distribution map with the expected nine-sector semantic distribution map sent by the base station. The UAV calculates the consistency score between the two. If the consistency score is greater than or equal to a preset safety threshold, the navigation status is determined to be normal, and the UAV continues to execute subsequent instructions; if the consistency score is less than the safety threshold, the navigation is determined to be abnormal, the UAV hovers and sends a replanning request to the base station.

[0070] Step S3: Construct the semantic weight topological memory index chain.

[0071] During the UAV's mission, the system records the UAV's state data in a time-series manner to form a historical path memory array. The recorded data includes the UAV's three-dimensional pose coordinates at each moment and the corresponding actual observation semantic distribution map.

[0072] The system calculates a saliency weight for each recorded node. The saliency weight is determined based on the information gain value of the semantic labels within the nine sectors and the distance to the target. Nodes with high information gain semantic labels and closer proximity are assigned higher saliency weights. The system stores nodes containing pose, semantic distribution, and saliency weights in a historical path memory array, forming a semantic topology index chain.

[0073] Step S4: Perform navigation failure determination and adaptive backtracking.

[0074] When a replanning request is triggered in step S2, the system enters the backtracking decision mode. The system traverses the memory nodes in the historical path memory array.

[0075] The system calculates the backtracking utility value for each memory node. The calculation of the backtracking utility value takes into account the salience weight of the memory node, the path cost from the current position to the memory node, and the similarity between the current observation and the visual features of the memory node.

[0076] The system selects the memory node with the highest backtracking utility value as the backtracking anchor point. The system plans the path from the current position back to the backtracking anchor point and controls the drone to perform the backtracking. After reaching the backtracking anchor point, the base station regenerates the subsequent task path based on the visual information of that location.

[0077] Step S1 is primarily executed at the base station (cloud), utilizing high-performance computing resources to handle complex semantic understanding and reasoning tasks. This step aims to transform unstructured user instructions into machine-executable control sequences and establish a visual expectation of the environment, providing a benchmark for subsequent edge-cloud collaborative verification. Step S1 specifically includes the following sub-steps:

[0078] S101, reception and preprocessing of multimodal command data.

[0079] The system receives user-input command data through a communication interface. When the command is in speech form, automatic speech recognition (ASR) technology is used to transcribe the audio stream into a text sequence; when the command is in the form of a gesture image, a visual encoder (such as the ResNet model) is used to extract gesture features, and a classifier is used to map them into corresponding text or semantic labels. For the specific algorithms of automatic speech recognition and gesture recognition, those skilled in the art can use existing mature technologies, and will not be elaborated upon here.

[0080] S102, Task decomposition and primitive mapping based on large language model.

[0081] The base station deploys a pre-trained Large Language Model (LLM) as the inference core. The system inputs the preprocessed text sequence into the LLM, and through prompt engineering or fine-tuning, the model outputs a structured task sequence. This task sequence contains an action primitive sequence A and a set of landmark phrases O.

[0082] Among them, action primitives refer to the basic control commands that the UAV flight control system can directly parse and execute, including but not limited to hovering, forward flight, yaw rotation, climb, descent, and speed adjustment commands. The sequence of action primitives is represented as follows: ,in, Representing the The action vector for each execution step.

[0083] Landmark phrases are semantic descriptions of environmental references in instructions, used to assist navigation and positioning. A set of landmark phrases is represented as... ,in, Representing the Semantic labels for key landmarks (e.g., red high-rise buildings, lawns, blue rooftops). Large language models establish action primitives through attention mechanisms. With landmark phrases The relationships between them should be clarified to identify the environmental objectives that should be considered when performing specific actions.

[0084] S103, Construct the expected semantic distribution map of the nine sectors.

[0085] While generating action primitives, the system also needs to generate an expectation of the visual scene at the next moment. For each action in the action sequence... The reasoning core combines prior knowledge of the current environment to predict the field of view after the drone performs the action.

[0086] Prior knowledge includes existing local environment maps, historical observation data, or common-sense physical laws (e.g., when performing an upward flight maneuver, the semantic area of ​​the sky should be expanded; when performing an approach maneuver to a red building, the proportion of the red building in the field of view should be increased).

[0087] To achieve fine-grained spatial semantic description, the system logically divides the UAV camera's field of view into a 3×3 grid structure, i.e., nine sectors. For the... Carving (i.e., performing an action) (At the next moment), a predicted nine-sector semantic distribution map is generated in the cloud. This distribution map consists of nine predicted feature descriptors, whose mathematical expression is:

[0088]

[0089] In the formula, express The set of expected semantic distributions at any given time; Indicates the first The expected feature descriptor for each sector, and The sector numbering rules can be preset, for example, numbered 1 to 9 in order from left to right and from top to bottom, with sector 5 being the center area of ​​the field of view.

[0090] S104, Generate the expected feature descriptor.

[0091] Expected feature descriptor It is the first A parametric description of the visual content within each sector, used for subsequent comparison with actual observations at the edges. Each expected feature descriptor contains the expected landmark semantic label and the expected relative depth range for that sector. Its definition is as follows:

[0092]

[0093] In the formula, Indicates the first The expected semantic label of the object in the sector. This label is derived from the set of landmark phrases O or the environmental background category extracted in step S102. For example, if the predicted action is performed so that the drone is facing a red building, then the central sector (sector 5) will have the following semantic label: The value is assigned the semantic code corresponding to the red building;

[0094] Indicates the first The expected average depth value of the sector. This value is calculated by the system based on the object coordinates in the prior map and the current predicted trajectory of the UAV. If there is no accurate map information, the large language model can assign a relative depth level based on common sense logic (e.g., the depth of the sky area is infinite, and the depth of the near ground is a small value).

[0095] Through the above steps, the base station not only issued control commands on how to fly, but also issued expected verification standards on what should be seen, providing a data foundation for subsequent autonomous consistency verification at the edge.

[0096] In step S2, the system performs a 9-sector fine-grained perception and edge-cloud consistency verification. This step is executed by the UAV's onboard computing module and aims to verify the accuracy of the expectations generated in step S1 using local real-time perception data, thereby determining the control transfer strategy. This step specifically includes the following processing flow:

[0097] S2.1 Acquisition of multimodal sensing data and spatial grid division.

[0098] During the execution of action commands, the drone uses its onboard RGB-D depth camera to collect environmental data at a preset frame rate. Each frame contains an RGB color image and an aligned depth image. The drone's onboard unit spatially divides the collected images in a pixel coordinate system, dividing them into a 3×3 grid area, forming nine independent observation sectors. The spatial distribution of these nine observation sectors strictly corresponds to the spatial definition of the expected nine-sector semantic distribution map in step S1.

[0099] S2.2 Construction of the actual observation semantic distribution map.

[0100] For each segmented sector, the onboard unit runs a lightweight object detection algorithm to perform semantic recognition on the RGB image region and obtain the category label of the main object within that region. Simultaneously, the onboard unit calculates the average depth value of the corresponding depth image region, which is used as the observation distance for that sector. For the specific implementation of the object detection algorithm, those skilled in the art can use lightweight neural network models such as YOLO-Nano and MobileNet-SSD, which will not be elaborated upon here.

[0101] Based on the above processing results, the airborne terminal constructs the current time. Actual observation semantic distribution map The distribution map consists of 9 observed feature vectors, and its mathematical expression is as follows:

[0102]

[0103] In the formula, This represents the semantic distribution map of actual observations. For any given... Each sector, observing feature vector The definition is as follows:

[0104]

[0105] In the formula, Indicates the first The semantic labels of objects actually identified in each sector; Indicates the first The average depth value measured in each sector, Indicates the first The observed feature vector of each sector.

[0106] S2.3 Calculation of Expectation-Observation Consistency Score. To quantitatively assess the degree of matching between the current actual environment and the cloud-based expectations, the airborne terminal calculates the expectation-observation consistency score in real time. This calculation process introduces a task-focused weighting mechanism to distinguish the importance of different sectors in the current navigation task. Consistency Score The calculation formula is as follows:

[0107]

[0108] In the formula, express The consistency score at time step [0, 1] has a value range of [0, 1]. Indicates the first The task attention weight for each sector. This weight is determined by the currently executing action instruction and satisfies the normalization condition. If the current command is to follow the target in front, then the weight of the sector in the center of the field of view is... The weights of edge sectors are set to higher values, while the weights of edge sectors are set to lower values; if the command is a panoramic search, the weight distribution of each sector tends to be more uniform.

[0109] This is a semantic indicator function. When the actual observed semantic label... semantic tags as expected in the cloud The function value is 1 if the two terms are identical, or belong to the set of synonyms in the pre-defined semantic library (e.g., car and sedan); otherwise, the function value is 0. Indicates the first The depth observation residual of each sector reflects the deviation between the actual distance to the object and the expected distance; This is a distance normalization parameter used to adjust the sensitivity of depth deviation to score decay. This parameter is preset according to the environmental scale; for example, a smaller value is set in an indoor environment, and a larger value is set in an open outdoor environment.

[0110] S2.4 Execution of dynamic control strategy based on verification results.

[0111] The onboard unit will calculate the consistency score. With the preset safety threshold A comparison is made, and different control strategies are implemented based on the comparison results:

[0112] like The system determines that the current navigation status meets expectations and the environmental characteristics match well. At this point, the onboard unit decides that cloud intervention is unnecessary and continues to execute the current sequence of action primitives using the local control law to maintain autonomous flight. This approach reduces frequent communication interactions between the UAV and the base station, lowering control latency.

[0113] like The system determines that the current navigation state is abnormal. Causes of this abnormality include unexpected dynamic changes in the environment, loss of visual positioning, or target obstruction. In this case, the onboard unit immediately interrupts the execution of the current action sequence, controls the UAV to enter a hovering state, and displays the current actual observation semantic distribution map. The corresponding abnormal image frames are transmitted back to the base station via the communication link, triggering a replanning request.

[0114] In step S3, the system constructs the semantic weighted topology memory index chain. This step is integral to the entire process of the UAV's mission and is used to record environmental semantic features and spatial location information, forming a historical path memory array to support subsequent backtracking operations. Specifically, it includes the following processing flow:

[0115] S3.1 Generation and structure definition of memory nodes.

[0116] During the execution of the drone's action sequence, the system records the current flight status and perception data according to a preset sampling frequency or spatial distance interval, generating a historical path memory array. The array consists of memory nodes arranged in a time sequence, represented as follows: .

[0117] Unlike traditional trajectory recording methods that only store three-dimensional coordinates, the memory nodes in this embodiment... It is defined as a composite data structure containing semantic topological information. Specifically, each memory node... It includes three sets of core data: drones in Six-DOF pose data at time step The semantic distribution map of actual observations at this moment. And the environmental salience weight of the node. .

[0118] Among them, pose data Includes position coordinates (x, y, z) and attitude quaternions or Euler angles, used to describe the spatial state of the UAV. Actual observation semantic distribution map. Consistent with the data structure constructed in step S2, it contains semantic labels and depth information for 9 sectors, used to describe the visual environment features of the location.

[0119] S3.2 Calculation of the salience weight of the node environment.

[0120] To evaluate the value of each memory node in subsequent backtracking and relocation tasks, the system calculates an environmental saliency weight for each generated node. This weight reflects the uniqueness and recognizability of the location's visual features. If the location contains landmarks (such as buildings or facilities), the salience weight is higher and can be used as a backtracking anchor point; if the location contains the sky or road surface, the salience weight is lower.

[0121] Based on the observation data of 9 sectors, the system calculates the first... The significance weight of each memory node :

[0122]

[0123] In the formula, Represents a node Environmental significance weight value; Indicates the first The position weighting coefficients for each sector. Considering the optical characteristics of camera imaging and visual attention mechanisms, the central region of the field of view usually has the least distortion and contains the main target; therefore, the 5th sector (central sector) is pre-weighted. Value greater than edge sector value; Indicates the first Semantic tags for sector observation The information gain function. The system has a pre-set semantic category weight table. For landmark categories with navigational indication significance (such as red fire hydrants and signal towers), the function... Returns a large positive value; for background categories that lack texture features or have high repetition (such as white walls or blue skies), the function... Returns the smaller positive value;

[0124] Indicates the first Observation depth values ​​for each sector; To prevent extremely small constants with a denominator of zero (e.g., values ​​of 0.1), this term... This reflects the distance decay law, which means that when the semantic categories are the same, the closer the target is to the drone, the clearer its visual features are, the larger the pixel area it occupies in the image, and therefore the higher its contribution to localization, and the greater its corresponding weight component.

[0125] S3.3 Dynamic maintenance of the historical path memory array.

[0126] The system will calculate Store in the corresponding memory node In the process, the node is added to the historical path memory array. At the end. To control storage size and improve subsequent retrieval efficiency, the system can implement a keyframe filtering strategy:

[0127] When the pose change of the newly generated node is less than the preset threshold and the salience weight change is less than the preset threshold, the system determines that the current environment features are redundant, discards the new node, and only updates the current pose index.

[0128] Through the above mechanism, the system constructs a topological index chain containing rich semantic weight information, which can characterize the distribution of environmental features in the explored area.

[0129] In step S4, the system performs navigation failure determination and utility-based adaptive backtracking decision-making. This step pertains to the decision-making and planning level, aiming to intelligently select the optimal recovery point using historical memory data when the UAV encounters deadlock, positioning loss, or unexpected environmental changes in an unfamiliar environment, thereby achieving autonomous recovery from a faulty state to a normal navigation state. This step specifically includes the following processing flow:

[0130] S4.1 Trigger determination of backtracking mode.

[0131] The system monitors the consistency score output in step S2 in real time. The system monitors the changing trend. When a preset fault trigger condition is detected, the system switches the control mode from autonomous cruise mode to adaptive backtracking mode.

[0132] Fault triggering conditions include, but are not limited to:

[0133] Consistency score continuous Frames below the safety threshold (Indicating a persistently abnormal environment);

[0134] The drone detected a physical obstacle ahead and the local obstacle avoidance algorithm was unable to plan a detour (indicating it was stuck in a dead end).

[0135] Once the backtracking mode is triggered, the drone immediately stops its current mission, hovers in its current position, and prepares to perform the backtracking operation.

[0136] S4.2 Utility evaluation of backtracking anchor points.

[0137] To select the optimal backtracking position, the system traverses the historical path memory array. The system uses memory nodes in the memory as candidate backtracking anchors. The system calculates each candidate node. Retrospective utility value The retrospective utility value is a comprehensive evaluation index designed to balance the environmental salience of the retrospective point (ease of relocation), the cost of retrospection (distance), and the environmental relevance.

[0138] The system calculates the first based on the following formula. The backtracking utility value of a memory node :

[0139]

[0140] In the formula, Indicates the first The backtracking utility value of each memory node; Indicates the first The environmental saliency weight of each memory node. This value directly references the result calculated and stored in the node data structure in step S3. The larger the value, the richer the landmark features of the location. When the drone returns to the location, the base station can more easily reinterpret the environment and plan a new route based on the clear landmarks. Indicates the current pose of the drone With candidate node pose The Euclidean distance between them. This term represents the energy and time cost of backtracking. The greater the distance, the higher the cost and the greater the negative contribution to the utility value.

[0141] Represents the semantic distribution of observations at the current moment. Semantic distribution of candidate node records The semantic similarity between the two distributions. This similarity can be obtained by calculating the cosine similarity or Jaccard similarity coefficient of the semantic label vectors of corresponding sectors in the two distributions. This term is introduced to prioritize finding nodes that are visually related to the current environment and prevent jumping back to completely unrelated regions; , , These are non-negative weighted adjustment coefficients. The ratio of these three coefficients can be adjusted depending on the application scenario's emphasis on positioning stability or energy consumption. For example, in scenarios with sufficient battery power but poor environmental texture, increasing... The proportion of backtracking points should be prioritized to ensure high significance.

[0142] S4.3 Execution of the optimal backtracking strategy.

[0143] The system compares the backtracking utility values ​​of all candidate nodes and selects the node with the highest utility value as the optimal backtracking anchor. :

[0144]

[0145] In the formula, The optimal backtracking anchor point is the memory node with the highest utility value selected from the historical path memory array. This node will be used as the recovery position for the UAV to return to, so as to achieve autonomous replanning in the event of a failure. The operator represents the operator for maximizing the function by taking the independent variable, and is used to select the element from the set that maximizes the value of the objective function; Represents a historical path memory array The first in There are 10 candidate memory nodes, each containing pose data, an observation semantic distribution map, and a salience weight; This represents a historical path memory array, which is a set of memory nodes arranged in a time sequence. It is used to store environmental information about the paths explored by the drone; Indicates the first The backtracking utility value of each memory node is used to comprehensively evaluate the positioning value, cost, and relevance of the backtracking anchor point.

[0146] After determining the optimal backtracking anchor point, the system extracts the location coordinates stored in that node. The drone's onboard path planning module plans a route from the current location to... based on the current local map information. The system determines a collision-free path and controls the aircraft to return along that path.

[0147] S4.4, Cloud-based replanning based on landmark confirmation.

[0148] When the drone reaches the homing anchor point Afterwards, the system executes a replanning process. The drone collects real-time images and uploads them to the base station. Since the backtracking anchor points are selected based on high saliency weights, their corresponding images must contain clear and easily identifiable landmark features (such as high information gain objects described in step S3).

[0149] After receiving the image, the base station uses a large language model to reassess the current environmental state. The base station identifies the current location as the new starting point and, combined with information about areas that previously caused navigation failures (marked as impassable areas), regenerates a sequence of subtasks leading to the final goal. Through this process, the system utilizes a memory backtracking mechanism to avoid previous erroneous paths, achieving trial and error and self-correction in invisible environments.

[0150] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal fusion intelligent interactive control method for unmanned aerial vehicles (UAVs), characterized in that, include: S1. Receive interactive instructions and parse and generate expected results: The base station receives the interactive instructions input by the user, uses a large language model to decompose the interactive instructions into a sequence of subtasks composed of action primitives and landmark phrases, and combines prior environmental information to generate the expected nine-sector semantic distribution map for the next moment. S2. Perform fine-grained perception and consistency verification: The UAV receives the action primitive and controls the aircraft to perform the corresponding action, and collects environmental data in real time; the UAV constructs the actual observation semantic distribution map at the current moment based on the environmental data; the UAV compares the actual observation semantic distribution map with the expected nine-sector semantic distribution map and calculates the expected-observation consistency score between the two. S3. Construct a semantic weight topology memory index chain: During the UAV's mission, the system records the UAV's pose data, actual observed semantic distribution map, and environmental saliency weights at each moment in a time sequence, forming a historical path memory array. S4. Navigation Failure Judgment and Adaptive Backtracking: When the expected-observation consistency score is less than the preset safety threshold, the system determines that the navigation is abnormal and triggers the backtracking mode; the system selects the optimal backtracking anchor point from the historical path memory array based on the backtracking utility value, controls the UAV to return to the optimal backtracking anchor point and performs path replanning.

2. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 1, characterized in that, In step S1, the generation of the expected nine-sector semantic distribution map specifically includes: The system divides the field of view of the drone camera into a nine-grid structure, forming nine sectors; The cloud generates an expected feature descriptor for each sector, which includes the expected landmark semantic label and the expected relative depth range value for the corresponding sector; The landmark semantic tags are derived from landmark phrases or environmental background categories in the subtask sequence; the relative depth range value is calculated based on the object coordinates in the prior map and the current predicted trajectory of the UAV.

3. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 1, characterized in that, In step S2, constructing the actual observation semantic distribution map at the current moment specifically includes: The UAV uses an RGB-D depth camera to acquire RGB color images and depth images, and divides them into nine observation sectors corresponding to the spatial definition of the expected nine-sector semantic distribution map. The airborne terminal constructs an observation feature vector for each observation sector. The observation feature vector includes the semantic labels of the objects actually identified in the corresponding sector and the measured average depth value. The object semantic label is extracted from the RGB image region using an object detection algorithm, and the average depth value is calculated from the depth image region.

4. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 1, characterized in that, In step S2, calculating the expected-observation consistency score between the two specifically includes: For each sector, calculate the matching between its actual observed semantic labels and expected landmark semantic labels, and calculate the depth observation residual between the actual average depth value and the expected relative depth range value; The expected-observation consistency score is obtained by calculating the weighted sum of all sectors based on the preset task attention weights, semantic matching results, and deep observation residuals. The task attention weight is determined by the currently executed action instruction, and the expectation-observation consistency score decreases as the depth observation residual increases.

5. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 1, characterized in that, In step S3, the historical path memory array consists of memory nodes arranged in a time sequence, and each memory node contains: The six-degree-of-freedom pose data of the UAV at the time corresponding to the memory node; The actual observation semantic distribution map corresponding to this moment; The environmental saliency weight of the memory node; The calculation of the environmental salience weight specifically includes: For each sector, obtain its observation semantic label information gain value and observation depth value; The weighted components of the corresponding sector are calculated based on the preset position weighting coefficient, the information gain value, and the observation depth value. The environmental saliency weight of the memory node is obtained by summing the weight components of the nine sectors. The position weighting coefficient has a higher value in the central region of the field of view than in the edge region, and the weight component is negatively correlated with the observation depth value.

6. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 5, characterized in that, The rule for determining the information gain value is as follows: If the observed semantic label belongs to a preset landmark category with navigation indication significance, a higher positive value is assigned; If the observed semantic label belongs to the background category that lacks texture features or has high repetition, a lower positive value is assigned.

7. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 6, characterized in that, In step S4, selecting the optimal backtracking anchor point from the historical path memory array based on the backtracking utility value specifically includes: Traverse the memory nodes in the historical path memory array as candidate nodes, and calculate the backtracking utility value of each candidate node; The retrospective utility value is calculated based on a weighted combination of the following three factors: The environmental saliency weight of the candidate nodes; The Euclidean distance from the current position to the pose of the candidate node; The semantic similarity between the actual observed semantic distribution map at the current moment and the actual observed semantic distribution map recorded by the candidate node; The retrospective utility value is positively correlated with the environmental saliency weight, negatively correlated with the Euclidean distance, and positively correlated with the semantic similarity. The memory node with the largest backtracking utility value is selected as the optimal backtracking anchor point.

8. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 7, characterized in that, The calculation of semantic similarity specifically includes: Extract the observation feature vectors of each sector in the current moment's actual observation semantic distribution map; Extract the observation feature vectors of each sector from the actual observation semantic distribution map of the candidate node records; Calculate the cosine similarity or Jaccard similarity coefficient between the observed feature vectors of corresponding sectors, and determine the overall semantic similarity based on the calculation results.

9. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 1, characterized in that, In step S4, controlling the UAV to return to the optimal backtracking anchor point and perform path replanning specifically includes: Extract the location coordinates stored in the optimal backtracking anchor point, plan a collision-free path from the current location to the location coordinates based on the local map, and control the drone to return; After reaching the specified location coordinates, the drone collects real-time images and uploads them to the base station. The base station uses a large language model to reassess the current environmental state and, combined with the regional information that caused the navigation anomaly, regenerates the sub-task sequence leading to the final goal.

10. The multimodal fusion intelligent interactive control method for unmanned aerial vehicles according to claim 1, characterized in that, In step S1, receiving and parsing the interactive command specifically includes: When the interactive command is in the form of voice, the audio stream is transcribed into a text sequence using automatic speech recognition technology; When the interaction command is in the form of a gesture image, the gesture features are extracted using a visual encoder and mapped to the corresponding text or semantic label using a classifier. The preprocessed text or semantic labels are input into a large language model, and the association between action primitives and landmark phrases is established through an attention mechanism.

Images