An unmanned aerial vehicle autonomous obstacle avoidance navigation method and system based on visual language action model and world model

Abstract

The application provides an unmanned aerial vehicle autonomous obstacle avoidance navigation method and system based on a visual language action model and a world model, the method comprising the following steps: S1, obtaining a high-level semantic action of a cross-vision target guide by using a double-end joint training VLA model based on a Transformer; S2, compressing a latent space state by using a world model, predicting a semantic expression of an obstacle by using a JEPA style predictor, taking the semantic expression of the obstacle as the main factor to mold a reward function of the world model, and generating an action based on environmental semantics; and S3, adaptively adjusting a target guide and an action weight of environmental semantics according to actual environmental information, and then taking a current and past state of the unmanned aerial vehicle as a reinforcement learning reward function to optimize a current trajectory, so that a more intelligent and reasonable navigation strategy is generated.

Description

Technical Field

[0001] This invention relates to the fields of autonomous navigation and artificial intelligence technology for unmanned aerial vehicles (UAVs), and particularly to an autonomous obstacle avoidance navigation method and system for UAVs based on visual language action models and world models. Background Technology

[0002] Autonomous navigation of unmanned aerial vehicles (UAVs) in complex environments is one of the key technologies for realizing their intelligent applications. Among these technologies, reinforcement learning-based UAV navigation methods can learn local obstacle avoidance and action decision-making strategies through interaction with the environment, exhibiting strong adaptability and online optimization capabilities. VLA-based navigation methods can integrate task text and onboard visual information to achieve a high-level understanding of target semantics, task intent, and visual field images, thereby enhancing the UAV's intent understanding capabilities. World model-based navigation methods can learn the evolutionary patterns of environmental states in latent space, predict future states, and assist in strategy generation and trajectory planning, thus improving sample utilization and training efficiency. The development of these technologies provides crucial support for the autonomous navigation, target guidance, and obstacle avoidance control of UAVs in complex environments.

[0003] However, existing UAV autonomous navigation methods based on reinforcement learning, VLA, and world models still have certain limitations in practical applications. First, reinforcement learning-based methods mainly rely on current observations or local environmental information for decision-making. While they possess good local obstacle avoidance capabilities, the high dimensionality of the input state space during long-distance target navigation leads to a lack of global awareness in decisions based on local environmental information, resulting in low exploration efficiency or suboptimal navigation strategies. Second, while VLA-based methods can provide high-level semantic guidance across targets in different fields of view, their ability to express obstacle semantics in complex environments is limited, lacking obstacle avoidance-related navigation guidance, and they struggle to achieve continuous and stable trajectory control by combining UAV flight dynamics. Third, although world model-based methods can predict and plan latent space, reducing the dimensionality of the state space, existing models mostly focus on state evolution and reward estimation, lacking explicit modeling of obstacle semantics in the environment and their impact on navigation safety, and their ability to express the latent space model of the target area is insufficient. Furthermore, most of the above methods adopt end-to-end training methods, resulting in strong model dependence, low training efficiency, and difficulty in achieving unified coordination between target semantic guidance, environmental semantic obstacle avoidance, and flight trajectory optimization. Based on the aforementioned problems, there is an urgent need for an autonomous obstacle avoidance and navigation method and system for unmanned aerial vehicles (UAVs) that integrates reinforcement learning, visual language action models, and world models. This system should organically unify cross-viewpoint target semantic guidance, environmental obstacle semantic modeling, and fine-grained trajectory optimization based on UAV state constraints, thereby improving the intelligence and stability of UAVs' autonomous navigation in complex environments. Summary of the Invention

[0004] Purpose of the Invention: This invention aims to address the shortcomings of existing vision-based autonomous navigation methods for unmanned aerial vehicles (UAVs), such as insufficient multimodal semantic modeling capabilities, weak safety in complex environments, and low training efficiency. This invention provides an autonomous obstacle avoidance and navigation method and system for UAVs based on a Vision Language Action (VLA) model and a world model. Building upon the VLA model's modeling of target semantics for navigation and the world model's modeling of environmental semantics for decision-making, this invention enhances the intelligent autonomous navigation decision-making capabilities and obstacle avoidance performance of UAVs by introducing adaptive policy weight adjustment based on real-world feedback and state constraints based on reinforcement learning for modal fusion and trajectory optimization.

[0005] The method includes the following steps:

[0006] Step S1: Use the visual sensing device installed at the front of the drone to collect environmental images, and input the environmental images and text containing target semantics into the Vision Language Action (VLA) network. Based on the degree of correlation between the text and the image region, predict the direction of the target outside the field of view, adaptively adjust the drone's orientation, and plan the drone's actions guided by the target text.

[0007] Step S2: Based on the target information outside the field of view, combined with the environmental obstacle information obtained by the sensors and visual sensing devices, the latent space of the world model is extracted and semantic continuous representation is performed. The semantic representation of obstacles and targets in the future state is predicted using the joint embedding prediction architecture JEPA-style predictor. After reinforcement learning based on the world model, an obstacle avoidance-based UAV navigation strategy is generated.

[0008] Step S3: The obstacle avoidance-based UAV navigation strategy generated in step S2 is combined with the UAV actions guided by the target text. The dominant weights of the UAV navigation strategy and the UAV actions guided by the target text are dynamically adjusted according to the specific environmental conditions to achieve safe and stable UAV flight control. In order to ensure the attitude stability of the UAV during flight, the current and historical states of the UAV are used as inputs to the reward function through reinforcement learning to achieve trajectory optimization based on the physical characteristics of the UAV itself.

[0009] Step S1 includes: acquiring environmental images using a visual sensing device installed at the front of the drone. :

[0010] ,

[0011] Where t represents the current time, Represents the real number space, where H, W, and C represent the height, width, and number of channels of the image, respectively;

[0012] After acquiring environmental images, the target text is filtered for importance. By performing hierarchical identification and weight allocation of target objects, location information, action constraint information, and environmental cues in the input text, key semantic text that plays a dominant role in UAV navigation decisions is selected. The resulting semantic text command T c for:

[0013] ,

[0014] in, This represents the l-th word element in the text, where l ranges from 1 to n, and n is the total number of words.

[0015] Features were extracted using both a visual encoder and a text encoder:

[0016] ,

[0017] ,

[0018] in, For visual feature extraction networks; For text semantic encoding networks; Image region features; d represents the text features; d represents the unified feature dimension after feature extraction and semantic encoding; M represents the number of visual image region features; and N represents the number of lexical units after text filtering.

[0019] Step S1 also includes: introducing a dual-end collaborative training mechanism containing intermediate alignment states during the training of visual and text encoders, so that the visual language action model can simultaneously establish constraint relationships in two directions: from text semantics to intermediate alignment states and then to visual region features, and from visual region features to intermediate alignment states and then to text semantics.

[0020] Among them, the intermediate alignment state is used to characterize the shared potential representation between the semantics of the text target and the semantics of the image region;

[0021] Based on dual-end collaborative training, cross-modal attention fusion of text features and visual region features after center alignment is performed using Transformer. Based on the multi-head attention structure of Transformer, The text feature vectors of n categories contained within are mapped to n text heads in multi-head attention, thus mapping the target text... Decompose into two or more text headers with clearly defined semantic responsibilities:

[0022] ,

[0023] in This represents the i-th text header in the target text, where i can be 1 to n, and n is the total number of text headers.

[0024] The n text heads are used as query vectors in cross-modal attention, and the visual region feature vectors are used as key and value inputs, thereby realizing multi-head attention fusion of text semantics to visual region features;

[0025] Suppose the visual encoder extracts visual region features for:

[0026] ,

[0027] in This represents the j-th visual region in the visual input, where j ranges from 1 to m, and m is the total number of visual regions.

[0028] Using visual region features as query vectors and text header features as key and value inputs, multi-head attention is performed on all visual regions in the visual image with n text headers to obtain the degree of semantic matching of each visual region relative to each text header category.

[0029] Step S1 also includes: for the j-th visual region and the i-th text header First, query vectors are generated through linear mapping. Key vector Sum value vector :

[0030] ,

[0031] ,

[0032] ,

[0033] in, This is the query mapping matrix corresponding to the visual region features; Let i be the key mapping matrix corresponding to the i-th text header; This is the value mapping matrix corresponding to the i-th text header;

[0034] Matching score between the j-th visual region and the i-th text header for:

[0035] ,

[0036] in, Represents the dimension of the key vector. This indicates transpose.

[0037] Step S1 further includes: normalizing the matching scores of the j-th visual region with all n text headers to obtain the attention weights of the j-th visual region for each text header:

[0038] ,

[0039] in, exp represents the degree of semantic matching between the j-th visual region and the i-th text header category;

[0040] The text semantic fusion result corresponding to the j-th visual region is obtained as follows:

[0041] ,

[0042] in, This represents the fused text representation of the j-th visual region obtained by semantically weighting all text headers.

[0043] Step S1 further includes: summing the matching scores of each text header to obtain the overall matching degree of the j-th visual region. :

[0044] ,

[0045] Obtain the comprehensive matching degree set of all visual regions :

[0046] ,

[0047] in This indicates the overall matching degree of the m-th visual region;

[0048] The visual region with the highest overall matching degree is selected as the key visual region most relevant to the semantics of the target text, and indexed. Represented as:

[0049] ,

[0050] Ultimately, the visual region with the highest matching degree is obtained. and the corresponding text semantic fusion results The result of fusing the visual region with the highest matching degree with the text semantics is input into the action mapping network. The action output of the drone based on the visual text action network is obtained. :

[0051] ,

[0052] The matching value of the visual region with the highest matching degree Compared with the set threshold, when When the current viewpoint is below a threshold, it is determined that the current drone viewpoint cannot effectively support navigation decisions. In this case, the result of visual-text semantic fusion with the highest matching degree within the current visual range is used. The system uses a prediction algorithm to infer the possible direction of targets outside the field of view, predicts the next navigation direction of the drone, and controls the drone to perform rotational actions, adjust the angle of the captured images, re-acquire environmental images, and recalculate the degree of visual region matching.

[0053] Step S2 includes: assuming the data obtained by the UAV from the environmental sensor at time t for:

[0054] ,

[0055] in, This represents distance sensor or depth sensor data. This indicates the distance distribution of lidar point clouds or obstacles.

[0056] Further construct the current world model input state :

[0057] ,

[0058] in, Represents the state fusion function;

[0059] Suppose that the current state is mapped to the potential state by the encoder. :

[0060] ,

[0061] in, Represents the world model state encoder. This represents the potential state representation of the current environment;

[0062] Use a semantic decoupler to decouple the obstacle semantics and target semantics in the current state:

[0063] ,

[0064] ,

[0065] in, This represents the semantic representation of the predicted obstacles. This represents the predicted semantic representation of the target. This represents a semantic decoupler for obstacles. This represents the target semantic decoupler;

[0066] The semantic representations of the target and obstacles are continuously vector-encoded using a continuous representation encoder.

[0067] ,

[0068] ,

[0069] in, This represents a continuous semantic representation of the target. Represents the semantic continuity of obstacles; and These are the target semantic encoder and the obstacle semantic encoder, respectively.

[0070] Obtain the continuous deterministic representation of the current moment. :

[0071] ,

[0072] in, This represents a semantic fusion function;

[0073] Introducing a JEPA-style predictor into a world model for UAV navigation to predict obstacle and target semantic representations in the potential space;

[0074] For the semantic representation of the next time step, the JEPA style predictor predicts the continuous deterministic semantic representation of the next time step. :

[0075] ,

[0076] in This represents the JEPA style predictor;

[0077] During model training, predictive representations are used. With true representation The distance between them is used as the core predictor of loss. :

[0078] ,

[0079] Total Forecast Loss Represented as:

[0080] ,

[0081] in, and Let these represent the target semantic prediction loss and the obstacle semantic prediction loss, respectively. , , Indicates the weight of each loss;

[0082] Semantic representation of continuous determinism As the current state ,according to The target semantic representation and obstacle semantic representation are used to construct the reward function of the world model. :

[0083] ,

[0084] in, Obstacle semantic representation in semantic representation of continuity and determinism The target semantic representation in semantic representations that indicate continuous determinism. This represents the drone's actions based on an environment-semantic strategy at the current moment. A reward mapping function that represents the context semantics;

[0085] Finally, based on the constructed reward function, a value network and an action network are built using the dual-delay deep deterministic policy gradient TD3 offline reinforcement learning network to generate a navigation policy based on obstacle avoidance and target approach. This enables obstacle avoidance-based drone navigation.

[0086] Step S3 includes: the UAV acquires current environmental status information through visual sensors, distance sensors, depth sensors, lidar, inertial measurement units, and environmental perception modules, and constructs navigation decision evaluation indicators;

[0087] Constructed environmental assessment indicators for:

[0088] ,

[0089] in, Indicates the complexity of environmental obstacles; Indicates the degree of environmental risk; This indicates the distance or estimated distance between the drone and the target; This indicates the deviation between the target direction and the drone's current heading;

[0090] The target text-guided navigation strategy obtained in step S1 is output as follows: ;

[0091] The obstacle navigation strategy output obtained from step S2, based on environmental semantics and world model prediction, is as follows: ;

[0092] Let the target navigation strategy weight be... The obstacle navigation strategy weights are ,satisfy:

[0093] ,

[0094] The weights are generated based on environmental assessment indicators, ultimately resulting in the basic fusion navigation actions. :

[0095] ,

[0096] in, This represents the basic navigation action after weighted fusion of the target navigation strategy and the obstacle navigation strategy;

[0097] Let the physical state vector of the UAV at time t be... for:

[0098] ,

[0099] in: Indicates the roll angle; Indicates the pitch angle; Indicates the yaw angle; Indicates speed; Indicates acceleration; Indicates angular velocity;

[0100] Introducing a Long Short-Term Memory (LSTM) network to model historical flight states: Assume a sequence of states of the UAV over a past period. for:

[0101] ,

[0102] in, This represents the drone status within the g-th historical time window, where g ranges from 1 to k, and k is the total number of historical windows.

[0103] Extracting context state using an LSTM network:

[0104] ,

[0105] in, This indicates the current flight context state.

[0106] The current flight context state As input to the reinforcement learning reward function and policy network, the drone's actions are constrained to meet the actual dynamic characteristics to achieve trajectory optimization. The fine-grained action correction output by the trajectory optimization policy network is:

[0107] ,

[0108] in, Represents the trajectory optimization strategy network; This indicates the amount of fine-tuning for the basic movement;

[0109] Final optimized drone actions Represented as:

[0110] ,

[0111] Construct the following reinforcement learning reward function :

[0112] ,

[0113] in, Indicates a stable attitude reward; Indicates a smooth reward for the action; Indicates a dynamic constraint reward; , , The weights for each reward item.

[0114] The present invention also provides an autonomous obstacle avoidance and navigation system for unmanned aerial vehicles based on a visual language action model and a world model for implementing the method, including a target semantic active perception and action generation module, an environmental semantic world model obstacle avoidance and navigation module, and an adaptive strategy fusion and trajectory optimization module.

[0115] The target semantic active perception and action generation module is used to receive environmental images collected by a visual sensing device installed at the front end of the UAV, and input the environmental images and text instructions containing target semantics into a Transformer-based visual text action (VLA) network. The module also filters the importance of target objects, orientation information, action constraint information, and environmental cues in the text instructions to obtain filtered semantic text instructions, and extracts image region features and text features through a visual encoder and a text encoder, respectively. Furthermore, the module introduces an intermediate alignment state to establish a two-way collaborative training constraint between text semantics and image region semantics, and calculates the matching degree between each visual region and each text head based on a multi-head attention mechanism, selecting the visual region with the highest matching degree as the key visual region to generate target text-guided navigation actions. When the matching degree is lower than a preset threshold, the module predicts the possible direction of the target outside the field of view based on the visual text semantic fusion results within the current visual range, and controls the UAV to perform a rotational perception action to re-acquire environmental images and update the target text-guided navigation actions.

[0116] The environmental semantic world model obstacle avoidance and navigation module is used to receive environmental sensor data, visual image information, and target semantic information of the key visual region from the UAV, and to construct the world model input state. The module also maps the current world model input state to a latent state representation using a world model state encoder, and uses a joint embedding prediction architecture (JEPA-style predictor) to predict obstacle semantic representations and target semantic representations in the latent space. Furthermore, the module performs continuous vector encoding on the target semantic representation and obstacle semantic representation using a continuous representation encoder to obtain a continuous deterministic semantic representation at the current moment, and predicts the continuous deterministic semantic representation at the next moment. Finally, the module constructs a prediction loss based on the distance between the predicted representation and the true representation, constructs a world model reward function based on the obstacle semantic representation and the target semantic representation, and uses a dual-delay deep deterministic policy gradient (TD3) offline reinforcement learning network to generate an obstacle navigation policy output based on environmental semantic obstacle avoidance and target approach.

[0117] The adaptive strategy fusion and trajectory optimization module is used to construct navigation decision evaluation indicators based on the current environmental state information obtained by the visual sensor, distance sensor, depth sensor, lidar, inertial measurement unit, and environmental perception module. These indicators include the complexity of environmental obstacles, the degree of environmental risk, the distance or estimated distance between the UAV and the target, and the deviation between the target direction and the UAV's current heading. The module also dynamically generates target navigation strategy weights and obstacle navigation strategy weights based on the navigation decision evaluation indicators, and performs weighted fusion of the target text-guided navigation action and the obstacle navigation strategy output to obtain a basic fused navigation action. Furthermore, the module extracts the flight context state using a Long Short-Term Memory (LSTM) network based on the UAV's current physical state vector and historical flight state sequence. This flight context state is used as input to the reinforcement learning reward function and the trajectory optimization strategy network, outputting a fine-grained action correction amount for the basic fused navigation action to obtain the final optimized UAV flight control action.

[0118] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0119] Beneficial Effects: This invention can autonomously understand and integrate target textual and visual image information, identify environmental obstacle information, and optimize attitude and trajectory for action decisions, solving the problem of stable autonomous obstacle avoidance navigation for UAVs in complex environments. This method integrates reinforcement learning, visual language action models, and world models into a UAV autonomous obstacle avoidance navigation method and system, achieving an organic unity of cross-view target semantic guidance, environmental obstacle semantic modeling, and fine trajectory optimization based on UAV state constraints. This invention is significant for improving the training efficiency of UAV navigation models, enhancing the intelligence level and flight stability of long-distance obstacle avoidance navigation strategies, and can be widely applied to autonomous navigation scenarios for UAVs in complex environments. Attached Figure Description

[0120] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0121] Figure 1 This is a flowchart of the method of the present invention.

[0122] Figure 2 This is a schematic diagram of the structure for policy generation based on the dual-end joint training of the VLA model using Transformer in this invention.

[0123] Figure 3 This is a network structure diagram of the world model latent space state compression achieved by the JEPA-style predictor in this invention.

[0124] Figure 4 This is a detailed architecture diagram of the navigation strategy adaptive adjustment based on environmental feedback and the trajectory optimization based on physical constraints in this invention.

[0125] Figure 5 This is a schematic diagram of an indoor navigation scene with a constellation-patterned navigation carpet. Detailed Implementation

[0126] like Figure 1 As shown, this embodiment of the invention provides an autonomous obstacle avoidance and navigation method for unmanned aerial vehicles (UAVs) based on a visual language action model and a world model, comprising the following steps:

[0127] S1 uses a visual sensing device installed at the front of the drone to collect environmental images. :

[0128] ,

[0129] Where t represents the current time, Represents the real number space, where H, W, and C represent the height, width, and number of channels of the image, respectively;

[0130] After acquiring environmental images, the target text is filtered based on its importance. By performing hierarchical identification and weighting of target objects, location information, action constraints, and environmental cues in the input text, key semantic text that plays a dominant role in UAV navigation decisions is selected. This reduces the interference of irrelevant text on the model's decisions, ensuring that text information no longer participates in decision-making as a single vector, but rather participates in cross-modal alignment and action generation in a more logical and interpretable form. The filtered semantic text instructions... for:

[0131] ,

[0132] in, This represents the l-th word in the text, where l ranges from 1 to n, and n is the total number of words.

[0133] Features were extracted using both a visual encoder and a text encoder:

[0134] ,

[0135] ,

[0136] in, For visual feature extraction networks; For text semantic encoding networks; Image region features; d represents the text features; d represents the unified feature dimension after feature extraction and semantic encoding; M represents the number of visual image region features; and N represents the number of lexical units after text filtering.

[0137] To address the issues of large training oscillations, slow convergence, and strong data dependence that arise when visual and text features are directly mapped to control actions, a dual-end collaborative training mechanism including intermediate alignment states is introduced during the training of the visual and text encoders. This enables the visual language action model to simultaneously establish constraint relationships in two directions: from text semantics to intermediate alignment states and then to visual region features, and from visual region features to intermediate alignment states and then to text semantics.

[0138] Among them, the intermediate aligned state is used to represent the shared potential representation between the semantics of the text target and the semantics of the image region. On the one hand, it serves as a bridge between the text end and the visual end, and on the other hand, it serves as a transitional semantic layer before end-to-end action generation, which improves the stability and robustness of the model training process.

[0139] In the dual-end collaborative training approach that combines forward alignment training of text to visual region features with backward backtracking training of visual region features to text semantics, since forward training and backward training share the intermediate alignment state and part of the cross-modal representation space, the unidirectional training of text to visual region features and the backward training of vision to text can provide each other with clearer semantic boundaries and region discrimination basis, correcting semantic offset and region mismatch problems in training, thus forming a bidirectional complementary and joint enhancement training mechanism.

[0140] Based on dual-end collaborative training, cross-modal attention fusion of text features and visual region features after center alignment is performed using Transformer. Based on the multi-head attention structure of Transformer, ... The text feature vectors of n categories contained within are mapped to n text heads in multi-head attention, thus mapping the target text... It is decomposed into multiple text header representations with clearly defined semantic responsibilities:

[0141] ,

[0142] in This represents the i-th text header in the target text, where i can be 1 to n and n is the total number of text headers.

[0143] By using n text heads as query vectors in cross-modal attention and visual region feature vectors as key and value inputs, multi-head attention fusion of text semantics and visual region features is achieved.

[0144] Suppose the visual encoder extracts visual region features for:

[0145] ,

[0146] in This represents the j-th visual region in the visual input, where j ranges from 1 to m, and m is the total number of visual regions.

[0147] Using visual region features as query vectors and text header features as key and value inputs, multi-head attention is performed on all visual regions in the visual image with n text headers to obtain the degree of semantic matching of each visual region relative to each text header category.

[0148] For the j-th visual region and the i-th text header First, the query vector, key vector, and value vector are generated separately through linear mapping:

[0149] ,

[0150] ,

[0151] ,

[0152] in, This is the query mapping matrix corresponding to the visual region features; Let i be the key mapping matrix corresponding to the i-th text header; Let be the value mapping matrix corresponding to the i-th text header.

[0153] Then the matching score between the j-th visual region and the i-th text header for:

[0154] ,

[0155] in, This represents the dimension of the key vector.

[0156] Furthermore, the matching scores of the j-th visual region and all n text headers are normalized to obtain the attention weights of the j-th visual region for each text header:

[0157] ,

[0158] in, exp represents the degree of semantic matching between the j-th visual region and the i-th text header category, and exp represents the natural exponential function.

[0159] Based on the above attention weights, the text semantic fusion result corresponding to the j-th visual region can be obtained:

[0160] ,

[0161] in, This represents the fused text representation of the j-th visual region obtained by semantically weighting all text headers.

[0162] To characterize the overall visual-text matching degree of this visual region, the matching scores of each text header are summed to represent the comprehensive matching degree of the j-th visual region. :

[0163] ,

[0164] The above calculation process is performed on all visual regions in the visual image to obtain a comprehensive set of matching degrees for all visual regions. :

[0165] ,

[0166] This represents the overall matching degree of the i-th visual region.

[0167] Furthermore, the visual region with the highest overall matching degree is selected as the key visual region most relevant to the semantics of the target text, and its index is determined accordingly. Represented as:

[0168] ,

[0169] Thus, the visual region with the highest matching degree is obtained. and the corresponding text semantic fusion results The result of fusing the visual region with the highest matching degree with the text semantics is input into the action mapping network to obtain the action output of the UAV based on the visual-text action network. :

[0170] ,

[0171] To further achieve omnidirectional autonomous navigation for targets outside the field of view, an adaptive rotation perception mechanism driven by the correlation between text and image is proposed. The matching value of the visual region with the highest matching degree is calculated using the above methods. and the set threshold, when When the current viewpoint is below a threshold, it is determined that the current drone viewpoint cannot effectively support navigation decisions. In this case, the result of visual-text semantic fusion with the highest matching degree within the current visual range is used. The system uses a prediction algorithm to infer the possible direction of targets outside the field of view, predicts the next navigation direction of the drone, and controls the drone to perform rotational actions, adjust the angle of the captured images, re-acquire environmental images, and recalculate the degree of visual region matching.

[0172] After multiple rounds of rotational perception and association judgment, the system continuously matches image regions with target text from different perspectives until the matching degree meets a threshold requirement, and selects the visual region with the highest matching degree as the primary basis for subsequent navigation decisions. Through the adaptive rotational perception mechanism, the UAV transforms from a traditional passive perception mode that relies on a fixed current field of view to an active perception mode that can proactively adjust the observation angle around the target semantics, improving target search capabilities, field of view expansion capabilities, and omnidirectional autonomous navigation capabilities in complex environments. A detailed structural diagram is shown below. Figure 2 As shown.

[0173] S2, let the data obtained by the UAV from the environmental sensor at time t be... for:

[0174] ,

[0175] in, This represents distance sensor or depth sensor data. This indicates the distance distribution of lidar point clouds or obstacles.

[0176] Based on the target-related important visual region information obtained in step S1, the current world model input state is further constructed. :

[0177] ,

[0178] in, This represents the state fusion function, which is used to uniformly map environmental sensor data, UAV motion state, and semantic information of target-related visual regions into a state representation that can be processed by the world model.

[0179] To enhance the ability of UAVs to predict current and future environmental conditions, a JEPA-style predictor is introduced into the world model of UAV navigation to predict obstacle semantics and target semantic representations in potential space.

[0180] The core idea of ​​the JEPA style predictor is to predict the semantic representations of target and obstacle regions in the latent space instead of directly reconstructing the complete original image. This reduces the dependence of the reconstruction task on irrelevant details, shrinks the state space, and improves the model's ability to express environmental structure and semantic relationships, as well as its training efficiency.

[0181] Suppose that the current state is mapped to the potential state by the encoder:

[0182] ,

[0183] in, Represents the world model state encoder. This represents the potential state of the current environment.

[0184] Use a semantic decoupler to decouple the obstacle semantics and target semantics in the current state:

[0185] ,

[0186] ,

[0187] in, This represents the semantic representation of the predicted obstacles. This represents the predicted semantic representation of the target. This represents a semantic decoupler for obstacles. This represents the target semantic decoupler;

[0188] After obtaining the semantic representation of the current state, future states are predicted based on a world model. A continuous representation encoder is used to continuously vectorize the semantic representations of the target and obstacles.

[0189] ,

[0190] ,

[0191] in, This represents a continuous semantic representation of the target. Represents the semantic continuity of obstacles; and These are the target semantic encoder and the obstacle semantic encoder, respectively.

[0192] Furthermore, by fusing the two, we obtain a continuous deterministic representation of the current moment. :

[0193] ,

[0194] in, This represents a semantic fusion function.

[0195] For the semantic representation of the next time step, the JEPA style predictor directly predicts the continuous deterministic semantic representation of the next time step. :

[0196] ,

[0197] This represents the JEPA style predictor; Indicates the current actions of the drone;

[0198] During model training, predictive representations are used. With true representation The distance between them is used as the core predictor of loss. :

[0199] ,

[0200] Total Forecast Loss Represented as:

[0201] ,

[0202] in, and These represent the semantic prediction losses for targets and obstacles, respectively. , , This represents the weight of each loss.

[0203] Using the methods described above, the model can effectively address the problems of traditional reinforcement learning, such as its inability to fully utilize environmental semantic information and the low training efficiency caused by excessively large state spaces. This allows for the creation of continuous, deterministic semantic representations. As the current state ,according to Target semantic representation and obstacle semantic representation and Reward function for constructing a world model :

[0204] ,

[0205] in, Obstacle semantic representation in semantic representation of continuity and determinism The target semantic representation in semantic representations that indicate continuous determinism. This represents the drone's actions based on an environment-semantic strategy at the current moment. This represents a reward mapping function dominated by environmental semantics.

[0206] Finally, based on the constructed reward function, a value network and an action network are built using the TD3 offline reinforcement learning network to generate a navigation strategy based on obstacle avoidance and target approach. This enables obstacle avoidance-based drone navigation. The specific network structure is as follows: Figure 3 As shown.

[0207] S3. To enable the UAV to dynamically adjust its navigation strategy according to the actual environmental conditions, a strategy weight adaptive adjustment system based on sensor feedback evaluation indicators is constructed. The system architecture diagram is shown below. Figure 4 As shown.

[0208] Specifically, the UAV acquires current environmental status information through visual sensors, distance sensors, depth sensors, lidar, inertial measurement units, and environmental perception modules, and constructs navigation decision evaluation indicators;

[0209] The constructed environmental assessment indicators are :

[0210] ,

[0211] in, It indicates the complexity of environmental obstacles, reflecting the number, density, distribution complexity, and degree of occlusion of obstacles in the current environment; It indicates the degree of environmental risk and is used to reflect the impact of factors such as dense smoke, heavy fog, low light, and strong interference on the visual perception and navigation safety of drones; This indicates the distance or estimated distance between the drone and the target; This indicates the deviation between the target direction and the drone's current heading.

[0212] After obtaining the navigation decision evaluation index based on sensor feedback, this paper further studies an adaptive weight adjustment method between target navigation strategy and obstacle navigation strategy. The purpose of this method is to dynamically adjust the proportion of the two types of strategies in the final basic navigation decision according to the complexity, risk level and target proximity of the current environment.

[0213] The target text-guided navigation strategy output obtained in step S1 is: This action primarily reflects the guiding role of target text semantics, target-related visual regions, and target orientation on UAV navigation.

[0214] The obstacle navigation strategy output obtained from step S2, based on environmental semantics and world model prediction, is as follows: This action primarily reflects the constraints imposed on drone navigation by obstacle semantics, environmental risks, and path accessibility.

[0215] To dynamically integrate the two types of strategies mentioned above, let the target navigation strategy weight be... The obstacle navigation strategy weights are: Both conditions are met:

[0216] ,

[0217] The weights are generated based on environmental assessment indicators and are adaptively adjusted according to the feedback from the indicators. When the complexity of environmental obstacles increases, the weight of the obstacle navigation strategy is increased; when the degree of environmental risk increases, such as the presence of dense smoke, fog, low light, or increased perception uncertainty, the weight of the obstacle navigation strategy is increased; when the target distance is far, the target visibility is low, or the target offset angle is large, the weight of the target navigation strategy is increased.

[0218] The final basic fusion navigation action is obtained:

[0219] ,

[0220] in, This represents the basic navigation action after the weighted fusion of the target navigation strategy and the obstacle navigation strategy.

[0221] In addition, to ensure that the drone maintains a stable attitude during flight and to suppress drastic changes in action and attitude in a short period of time, reinforcement learning is used to optimize the trajectory based on the drone's own physical state.

[0222] Let the physical state vector of the UAV at time t be... for:

[0223] ,

[0224] in: Indicates the roll angle; Indicates the pitch angle; Indicates the yaw angle; Indicates speed; Indicates acceleration; Indicates angular velocity;

[0225] Since the flight actions of drones have obvious temporal continuity, the state at a single moment is difficult to fully reflect the recent trend of action changes. Therefore, LSTM network is introduced to model the historical flight state.

[0226] Suppose the state sequence of the drone over a past period of time. for:

[0227] ,

[0228] in, This represents the drone status within the g-th historical time window, where g ranges from 1 to k, and k is the total number of historical windows.

[0229] Extracting context state using an LSTM network:

[0230] ,

[0231] in, This indicates the current flight context state.

[0232] The LSTM context state can reflect the recent attitude change trend, velocity change trend and action continuity of the UAV, enabling reinforcement learning-based trajectory optimization to fully consider the impact of historical flight state on the current action.

[0233] Based on this, the LSTM context state As input to the reinforcement learning reward function and policy network, the drone's actions are constrained to meet the actual dynamic characteristics to achieve trajectory optimization. The fine-grained action correction output by the trajectory optimization policy network is:

[0234] ,

[0235] in, Represents the trajectory optimization strategy network; This represents the basic action obtained by fusing the goal-oriented strategy and the contextual semantic strategy. It indicates the amount of fine-tuning for the basic movement.

[0236] Final optimized drone actions Represented as:

[0237] ,

[0238] To ensure flight stability, the following reinforcement learning reward function is constructed. :

[0239] ,

[0240] in, Indicates a stable attitude reward; Indicates a smooth reward for the action; Indicates a dynamic constraint reward; , , The weights for each reward item.

[0241] This embodiment also provides an autonomous obstacle avoidance and navigation system for unmanned aerial vehicles based on a visual language action model and a world model, including a target semantic active perception and action generation module, an environmental semantic world model obstacle avoidance and navigation module, and an adaptive strategy fusion and trajectory optimization module.

[0242] The system is deployed on an onboard computing platform of the UAV or in an edge computing device that communicates with the UAV. The system communicates with the UAV's visual sensing devices, distance sensors, depth sensors, lidar, inertial measurement unit, and flight controller to receive environmental perception data, target text commands, and UAV flight status, and output UAV flight control actions.

[0243] Specifically, the target semantic active perception and action generation module first receives environmental images acquired by the UAV's front-end visual sensing device and target text commands input by the user. This module performs semantic filtering on the target text commands, extracting target objects, orientation information, action constraint information, and environmental cues. Subsequently, it extracts image region features through a visual encoder and text features through a text encoder, and establishes a shared latent representation between text semantics and visual region semantics using intermediate alignment. Based on this, the module calculates the matching degree between each visual region and the text head through a cross-modal multi-head attention mechanism, designating the visual region with the highest matching degree as the key visual region, and generating target text-guided navigation actions based on the fusion result of the key visual region and text semantics. When the matching degree is lower than a preset threshold, the module outputs a rotation perception control command to the flight controller, causing the UAV to adjust its course and reacquire images, thereby expanding the UAV's active search capability for targets outside its field of view.

[0244] The environmental semantic world model obstacle avoidance and navigation module receives key visual region information output by the target semantic active perception and action generation module, and simultaneously receives environmental obstacle information obtained from distance sensors, depth sensors, and LiDAR. This module fuses environmental obstacle information, visual image information, UAV motion state, and target-related visual region semantic information into a world model input state, and obtains a latent state representation through a world model state encoder. Subsequently, the module decouples obstacle semantics and target semantics through a semantic decoupler, encoding both into a continuous deterministic semantic representation. Utilizing a joint embedding prediction architecture (JEPA-style predictor), the module predicts the continuous deterministic semantic representation for the next time step. This allows the module to model changes in obstacle distribution and target approach trends in the latent space, and constructs a world model reward function based on the prediction results. Based on this reward function, the module generates obstacle navigation policy output through a dual-delay deep deterministic policy gradient (TD3) offline reinforcement learning network, enabling the UAV to explicitly respond to obstacle semantics and environmental risks during target approach.

[0245] The adaptive strategy fusion and trajectory optimization module receives target text-guided navigation actions from the target semantic active perception and action generation module, as well as obstacle navigation strategy outputs from the environmental semantic world model obstacle avoidance navigation module. This module dynamically calculates the target navigation strategy weights and obstacle navigation strategy weights based on the current environmental obstacle complexity, environmental risk level, distance or estimated distance between the UAV and the target, and the deviation between the target direction and the UAV's current heading. When the environmental obstacle complexity or environmental risk level increases, the obstacle navigation strategy weight is increased; when the target distance is far, target visibility is low, or the target direction deviation is large, the target navigation strategy weight is increased. Subsequently, this module performs weighted fusion of the two types of navigation strategy outputs to obtain the basic fused navigation actions. Further, this module receives the UAV's current roll angle, pitch angle, yaw angle, velocity, acceleration, and angular velocity, and combines this with historical flight state sequences. It extracts the flight context state using a Long Short-Term Memory (LSTM) network; then, it uses the trajectory optimization strategy network to output fine-grained action corrections to refine the basic fused navigation actions, ultimately generating UAV flight control actions that satisfy flight stability, action smoothness, and dynamic constraints.

[0246] Through the above system architecture, the target semantic active perception and action generation module solves the problems of target text understanding and active perception of targets outside the field of view; the environmental semantic world model obstacle avoidance and navigation module solves the problems of semantic modeling of obstacles in complex environments and generation of obstacle avoidance strategies; and the adaptive strategy fusion and trajectory optimization module solves the problems of dynamic weight adjustment between target navigation and obstacle avoidance navigation, as well as optimization of flight trajectory stability. These three modules are sequentially connected and work in a closed-loop manner, enabling the UAV to simultaneously achieve target semantic guidance, environmental semantic obstacle avoidance, and stable flight control based on physical state constraints in complex environments.

[0247] In a specific embodiment of this system, the method and system described herein are verified in an indoor physical drone verification environment. The verification environment is an indoor planar scene covered with a constellation-patterned navigation carpet, such as... Figure 5 As shown, the scene includes text-labeled areas for Aries, Taurus, Cancer, Gemini, Leo, Libra, Scorpio, and the Big Dipper, as well as star-shaped visual markers, constellation lines, and the carpet's boundaries. Multiple small drones with protective frames are distributed at different locations on the navigation carpet; some serve as target drones to be controlled, while others act as dynamic or static obstacles. The objective is to guide the target drones, based on user-inputted text commands, to autonomously avoid other drones and reach a safe location near a designated constellation area, recognizing constellation markers and star paths.

[0248] In this embodiment, the indoor space includes a gray floor, white walls, and a navigation carpet with a constellation pattern laid on the floor. The navigation carpet features high-contrast circular stars, constellation lines, and constellation text labels in both Chinese and English, serving as reference markers for visual positioning, target recognition, and path semantic understanding. The drone to be controlled initially positions itself near the Aries area in front of the navigation carpet, with the target location set as a safe hovering area near the Cancer or Taurus markers. The remaining small drones are distributed in the middle, right, and rear areas of the constellation pattern, forming close-range obstacles in the flight path. The user-inputted target text command is: "Take off from the Aries area, fly along the constellation pattern area, avoid other drones, fly to the vicinity of the Cancer marker and maintain a safe distance."

[0249] In this embodiment, the UAV is equipped with a forward-looking vision sensor, a distance sensor, a depth sensor, and an inertial measurement unit (IMU), and performs model inference through an edge computing device communicatively connected to the UAV. The forward-looking vision sensor is used to acquire images of a constellation-patterned carpet, text labels, star lines, carpet boundaries, and other UAVs; the distance and depth sensors are used to estimate the relative distances between the UAV under control and other UAVs, carpet boundaries, and walls; the IMU is used to obtain the UAV's attitude angles, velocity, acceleration, and angular velocity. The UAV outputs control quantities including forward velocity, lateral velocity, vertical velocity, and yaw rate.

[0250] like Figure 1 As shown, the drone first executes step S1. The target semantic active perception and action generation module receives the indoor constellation pattern carpet image collected by the forward-looking visual sensor and the target text command input by the user. This module filters the target text command based on importance, identifying "near the Cancer constellation" as the target object semantic, "take off from the Aries area" and "fly along the constellation pattern area" as the starting point and path prompt semantic, "avoid other drones" as the action constraint semantic, and "maintain a safe distance" as the environmental prompt semantic. After filtering, the system retains the above key semantics and removes modifiers that are not directly related to navigation decisions, thus obtaining the filtered semantic text command.

[0251] like Figure 2 As shown, the target semantic active perception and action generation module divides the current environment image into multiple visual regions. It extracts image region features such as star points, constellation lines, constellation text labels, carpet boundaries, and other drones from each visual region using a visual encoder, and simultaneously extracts filtered text features using a text encoder. Subsequently, this module establishes a shared latent representation between text semantics and image region semantics through intermediate alignment, and calculates the matching degree between each visual region and the target text semantics using a cross-modal multi-head attention mechanism. From the current perspective, the system identifies the Aries region, part of the star point path, and several other drones, but the target "near the Cancer label" has not yet fully entered the current field of view. The highest overall matching degree is 0.52, lower than the preset threshold of 0.65. Based on this, the system determines that the current image is insufficient to support stable target navigation decisions.

[0252] When the current highest matching degree is lower than the preset threshold, the target semantic active perception and action generation module predicts that the target area may be located to the right of the drone's current heading, based on the direction of the constellation line extension in the image, the relative position of the visible text markers, and the semantic meaning of "near the Cancer marker" in the text. The system outputs a rotation perception action, controlling the drone to rotate approximately 25 degrees to the right and re-acquire the environmental image. After rotation, the Cancer text marker and its surrounding star point area enter the drone's forward view image more completely. The system recalculates the matching degree between the visual area and the target text, obtaining a maximum comprehensive matching degree of 0.83, which is higher than the preset threshold of 0.65. At this point, the system identifies this visual area as the key visual area and generates a target text-guided navigation action. An example output of this target text-guided navigation action is: forward speed 0.80 m / s, lateral speed 0.24 m / s, vertical speed 0 m / s, and yaw rate 0.12 radians / s. This action indicates that the drone should move along the constellation pattern area towards the target constellation marker and adjust its heading to the right appropriately.

[0253] Subsequently, the drone executes step S2. The environmental semantic world model obstacle avoidance and navigation module receives environmental obstacle information obtained from the distance sensor, depth sensor, and visual sensor, and simultaneously receives key visual region semantic information output from step S1. For example... Figure 3 As shown, this module integrates the positions of other UAVs, the boundaries of the constellation pattern carpet, the motion state of the UAVs, and the semantic information of the target-related visual regions into the world model input state, and maps it into a potential state representation through the world model state encoder.

[0254] In this embodiment, the distance sensor detects that other drones are 1.8 meters ahead of the drone to be controlled, 1.4 meters to the left front, and approximately 3.2 meters of passable space in the star point path area to the right front. The depth map shows a small drone with a protective frame near the central constellation line, and no nearby obstacles are detected near the Cancer constellation marker area to the front right. The semantic decoupler decouples obstacle semantics and target semantics based on the above potential state representation. The obstacle semantics are described as "small drone ahead, small drone to the left front, carpet boundary constraint, passable star point path to the right front," and the target semantics are described as "safe area near the Cancer constellation marker to the front right."

[0255] The environmental semantic world model obstacle avoidance and navigation module further encodes obstacle and target semantic representations into continuous deterministic semantic representations and uses a JEPA-style predictor to predict the continuous deterministic semantic representation for the next moment. A world model reward function is constructed based on obstacle and target semantics, where positive rewards are given for actions that approach the Cancer constellation marker area along the star point path, negative rewards are given for approaching other drones at distances below a safe threshold, and positive rewards are given for actions that move away from the carpet boundary and remain within the constellation pattern area.

[0256] Based on the aforementioned reward function, the dual-delay deep deterministic policy gradient TD3 offline reinforcement learning network generates an obstacle navigation policy output based on environmental semantic obstacle avoidance and target approach. An example of this obstacle navigation policy output is: forward velocity 0.46 m / s, lateral velocity 0.31 m / s, vertical velocity 0 m / s, and yaw rate 0.06 radians / s. This action indicates that the UAV should reduce its forward speed and laterally avoid small UAVs in front and to the left of the accessible planetary path area.

[0257] Subsequently, the drone executes step S3. For example... Figure 4 As shown, the adaptive strategy fusion and trajectory optimization module constructs navigation decision evaluation indicators based on the current environmental state information obtained from the visual sensor, distance sensor, depth sensor, and inertial measurement unit. In this embodiment, the system calculates the environmental obstacle complexity as 0.68 based on the number of other drones, the distance to the nearest obstacle, the passable width of the star point path, and the distance to the carpet boundary; calculates the environmental risk level as 0.24 based on the indoor lighting conditions, depth map confidence, and target text label occlusion; calculates the distance between the drone and the target as 4.6 meters based on the estimated position of the Cancer constellation marker area; and calculates the deviation between the target direction and the drone's current heading as 18 degrees based on the angle between the target direction and the drone's current heading.

[0258] The adaptive strategy fusion and trajectory optimization module dynamically generates target navigation strategy weights and obstacle navigation strategy weights based on the aforementioned navigation decision evaluation indicators. Since multiple small drones are present near the current path and the nearest obstacle is relatively close, the system increases the obstacle navigation strategy weight; since the Cancer marker has been identified and the target direction deviation is small, the target navigation strategy weight is relatively decreased. In this embodiment, the target navigation strategy weight is 0.42, and the obstacle navigation strategy weight is 0.58. The system performs a weighted fusion of the target text-guided navigation action obtained in step S1 and the obstacle navigation strategy output obtained in step S2 to obtain a basic fused navigation action. An example of this basic fused navigation action is: forward velocity 0.60 m / s, lateral velocity 0.28 m / s, vertical velocity 0 m / s, and yaw rate 0.09 radians / s.

[0259] To further ensure the flight stability of the UAV, the adaptive strategy fusion and trajectory optimization module receives the UAV's current physical state and historical flight state sequence. In this embodiment, the UAV's current roll angle is 1.8 degrees, pitch angle is 2.6 degrees, yaw angle is 28 degrees, speed is 0.61 m / s, acceleration is 0.16 m / s², and angular velocity is 0.08 radians / s. The system selects the states of the past 10 control cycles as the historical flight state sequence and extracts the flight context state through a Long Short-Term Memory (LSTM) network. Based on the flight context state, the trajectory optimization strategy network determines that the UAV's recent lateral avoidance maneuvers are significant. If the basic fusion navigation maneuvers are executed directly, it may cause a short-term increase in the roll angle. Therefore, it outputs a fine-grained action correction. An example of this fine-grained action correction is: forward speed correction -0.03 m / s, lateral speed correction -0.04 m / s, vertical speed correction 0 m / s, and yaw rate correction 0.01 radians / s.

[0260] After trajectory optimization, the system generates the final optimized UAV flight control maneuvers: forward speed 0.57 m / s, lateral speed 0.24 m / s, vertical speed 0 m / s, and yaw rate 0.10 radians / s. These control maneuvers are sent to the UAV flight controller, enabling the UAV to move towards the vicinity of the Cancer constellation along the right-hand star-shaped path area while avoiding other small UAVs in front and to the left.

[0261] In a complete navigation process of the above embodiment, the UAV sequentially completes active rotation perception, constellation target area identification, semantic prediction of obstacles from other UAVs, weighted fusion of target navigation and obstacle avoidance navigation, and trajectory optimization based on physical state constraints. Ultimately, the UAV is able to bypass other small UAVs on the constellation pattern carpet, reach a safe area near the Cancer sign, and maintain stable hovering near the target area.

[0262] Compared to solutions that only use visual language action models, this embodiment does not simply generate actions directly based on target text and the current image. Instead, it explicitly introduces obstacle semantic prediction into the world model, enabling the UAV to identify and avoid nearby obstacles during target approach. Compared to solutions that only use reinforcement learning for local obstacle avoidance, this embodiment can utilize key visual region information to determine the target direction across the field of view, reducing ineffective exploration and local wandering in indoor patterned scenes. Compared to solutions that only use world model prediction, this embodiment further introduces target text guidance and UAV physical state constraints, ensuring that the final action not only avoids obstacles and approaches the target but also maintains stable flight attitude and continuous action changes.

[0263] Furthermore, based on the aforementioned indoor physical verification, the system can also be deployed on actual drone platforms of different specifications. As a specific implementation, the electronic device can be an onboard computing device equipped with a graphics processor or neural network computing unit, or an edge computing device wirelessly connected to the drone; the drone platform can be a small multi-rotor drone with a protective frame or an industrial-grade multi-rotor drone. In actual deployment, the onboard computing device or edge computing device receives actual visual images, depth data, distance sensor data, and inertial measurement unit data, and executes the target semantic active perception and action generation, environmental semantic world model obstacle avoidance navigation, adaptive strategy fusion, and trajectory optimization processes described in this invention, ultimately outputting speed control, heading control, or trajectory control quantities to the drone flight controller.

[0264] As can be seen from the above specific embodiments, the practical technical problem solved by the present invention is that, when the target area is not entirely within the current field of view of the UAV, there are nearby obstacles in the flight path, and the UAV needs to maintain attitude stability, traditional single-vision navigation, single-reinforcement learning obstacle avoidance, or single-world model prediction methods are difficult to simultaneously take into account target semantic understanding, safe obstacle avoidance, and smooth continuous flight maneuvers. The present invention achieves active matching of target text semantics and key visual regions through step S1, explicitly models obstacle semantics and target semantics in the latent space through step S2, and dynamically fuses and optimizes actions based on environmental complexity and the physical state of the UAV through step S3, enabling the UAV to achieve safer, more stable, and target-semantic-oriented autonomous obstacle avoidance navigation in complex environments.

[0265] This invention provides a method and system for autonomous obstacle avoidance and navigation of unmanned aerial vehicles (UAVs) based on a visual language action model and a world model. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for autonomous obstacle avoidance and navigation of unmanned aerial vehicles (UAVs) based on a visual language action model and a world model, characterized in that, Includes the following steps: Step S1: Use the visual sensing device installed at the front of the drone to collect environmental images, and input the environmental images and text containing target semantics into the visual text action network (VLA). Based on the correlation between the text and the image region, predict the direction of the target outside the field of view, adaptively adjust the drone's orientation, and plan the drone's actions guided by the target text. Step S2: Based on the target information outside the field of view, combined with the environmental obstacle information obtained by the sensor and visual sensing device, the latent space of the world model is extracted and semantic continuity is represented. The semantic representation of obstacles and targets in the future state is predicted using the joint embedding prediction architecture JEPA style predictor. Through reinforcement learning based on the world model, an obstacle avoidance-based drone navigation strategy is generated. Step S3: The obstacle avoidance-based UAV navigation strategy generated in step S2 is combined with the UAV actions guided by the target text. The dominant weights of the UAV navigation strategy and the UAV actions guided by the target text are dynamically adjusted according to the specific environmental conditions to achieve UAV flight control. By using reinforcement learning to take the current and historical states of the UAV as input to the reward function, trajectory optimization based on the physical characteristics of the UAV is achieved.

2. The method according to claim 1, characterized in that, Step S1 includes: acquiring environmental images using a visual sensing device installed at the front of the drone. : ， Where t represents the current time, Represents the real number space, where H, W, and C represent the height, width, and number of channels of the image, respectively; After acquiring environmental images, the target text is filtered for importance. By performing hierarchical identification and weight allocation of target objects, location information, action constraint information, and environmental cues in the input text, key semantic text that plays a dominant role in UAV navigation decisions is selected. The resulting semantic text command T c for: ， in, This represents the l-th word element in the text, where l ranges from 1 to n, and n is the total number of words. Features were extracted using both a visual encoder and a text encoder: ， ， in, For visual feature extraction networks; For text semantic encoding networks; Image region features; d represents the text features; d represents the unified feature dimension after feature extraction and semantic encoding; M represents the number of visual image region features; and N represents the number of lexical units after text filtering.

3. The method according to claim 2, characterized in that, Step S1 also includes: introducing a dual-end collaborative training mechanism containing intermediate alignment states during the training of visual and text encoders, so that the visual language action model can simultaneously establish constraint relationships in two directions: from text semantics to intermediate alignment states and then to visual region features, and from visual region features to intermediate alignment states and then to text semantics. Among them, the intermediate alignment state is used to characterize the shared potential representation between the semantics of the text target and the semantics of the image region; Based on dual-end collaborative training, cross-modal attention fusion of text features and visual region features after center alignment is performed using Transformer. Based on the multi-head attention structure of Transformer, The text feature vectors of n categories contained within are mapped to n text heads in multi-head attention, thus mapping the target text... Decompose into two or more text headers with clearly defined semantic responsibilities: ， in This represents the i-th text header in the target text, where i can be 1 to n, and n is the total number of text headers. The n text heads are used as query vectors in cross-modal attention, and the visual region feature vectors are used as key and value inputs, thereby realizing multi-head attention fusion of text semantics to visual region features; Suppose the visual encoder extracts visual region features for: ， in This represents the j-th visual region in the visual input, where j ranges from 1 to m, and m is the total number of visual regions. Using visual region features as query vectors and text header features as key and value inputs, multi-head attention is performed on all visual regions in the visual image with n text headers to obtain the degree of semantic matching of each visual region relative to each text header category.

4. The method according to claim 3, characterized in that, Step S1 also includes: for the j-th visual region and the i-th text header First, query vectors are generated through linear mapping. Key vector Sum value vector : ， ， ， in, This is the query mapping matrix corresponding to the visual region features; Let i be the key mapping matrix corresponding to the i-th text header; This is the value mapping matrix corresponding to the i-th text header; Matching score between the j-th visual region and the i-th text header for: ， in, Represents the dimension of the key vector. This indicates transpose.

5. The method according to claim 4, characterized in that, Step S1 further includes: normalizing the matching scores of the j-th visual region with all n text headers to obtain the attention weights of the j-th visual region for each text header: ， in, exp represents the degree of semantic matching between the j-th visual region and the i-th text header category; The text semantic fusion result corresponding to the j-th visual region is obtained as follows: ， in, This represents the fused text representation of the j-th visual region obtained by semantically weighting all text headers.

6. The method according to claim 5, characterized in that, Step S1 further includes: summing the matching scores of each text header to obtain the overall matching degree of the j-th visual region. : ， Obtain the comprehensive matching degree set of all visual regions : ， in This indicates the overall matching degree of the m-th visual region; The visual region with the highest overall matching degree is selected as the key visual region most relevant to the semantics of the target text, and indexed. Represented as: ， Ultimately, the visual region with the highest matching degree is obtained. and the corresponding text semantic fusion results The result of fusing the visual region with the highest matching degree with the text semantics is input into the action mapping network. The action output of the drone based on the visual text action network is obtained. : ， The matching value of the visual region with the highest matching degree Compared with the set threshold, when When the current viewpoint is below a threshold, it is determined that the current drone viewpoint cannot effectively support navigation decisions. In this case, the result of visual-text semantic fusion with the highest matching degree within the current visual range is used. The system uses a prediction algorithm to infer the possible direction of targets outside the field of view, predicts the next navigation direction of the drone, and controls the drone to perform rotational actions, adjust the angle of the captured images, re-acquire environmental images, and recalculate the degree of visual region matching.

7. The method according to claim 6, characterized in that, Step S2 includes: assuming the data obtained by the UAV from the environmental sensor at time t for: ， in, This represents distance sensor or depth sensor data. This indicates the distance distribution of lidar point clouds or obstacles. Further construct the current world model input state : ， in, Represents the state fusion function; Suppose that the current state is mapped to the potential state by the encoder. : ， in, Represents the world model state encoder. This represents the potential state representation of the current environment; Use a semantic decoupler to decouple the obstacle semantics and target semantics in the current state: ， ， in, This represents the semantic representation of the predicted obstacles. This represents the predicted semantic representation of the target. This represents a semantic decoupler for obstacles. This represents the target semantic decoupler; The semantic representations of the target and obstacles are continuously vector-encoded using a continuous representation encoder. ， ， in, This represents a continuous semantic representation of the target. Represents the semantic continuity of obstacles; and These are the target semantic encoder and the obstacle semantic encoder, respectively. Obtain the continuous deterministic representation of the current moment. : ， in, This represents a semantic fusion function; Introducing a JEPA-style predictor into a world model for UAV navigation to predict obstacle and target semantic representations in the potential space; For the semantic representation of the next time step, the JEPA style predictor predicts the continuous deterministic semantic representation of the next time step. : ， in This represents the JEPA style predictor; During model training, predictive representations are used. With true representation The distance between them is used as the core predictor of loss. : ， Total Forecast Loss Represented as: ， in, and Let these represent the target semantic prediction loss and the obstacle semantic prediction loss, respectively. , , Indicates the weight of each loss; Semantic representation of continuous determinism As the current state ,according to The target semantic representation and obstacle semantic representation are used to construct the reward function of the world model. : ， in, Obstacle semantic representation in semantic representation of continuity and determinism The target semantic representation in semantic representations that indicate continuous determinism. This represents the drone's actions based on an environment-semantic strategy at the current moment. A reward mapping function that represents the context semantics; Finally, based on the constructed reward function, a value network and an action network are built using the dual-delay deep deterministic policy gradient TD3 offline reinforcement learning network to generate a navigation policy based on obstacle avoidance and target approach. This enables obstacle avoidance-based drone navigation.

8. The method according to claim 7, characterized in that, Step S3 includes: the UAV acquires current environmental status information through visual sensors, distance sensors, depth sensors, lidar, inertial measurement units, and environmental perception modules, and constructs navigation decision evaluation indicators; Constructed environmental assessment indicators for: ， in, Indicates the complexity of environmental obstacles; Indicates the degree of environmental risk; This indicates the distance or estimated distance between the drone and the target; This indicates the deviation between the target direction and the drone's current heading; The target text-guided navigation strategy obtained in step S1 is output as follows: ; The obstacle navigation strategy output obtained from step S2, based on environmental semantics and world model prediction, is as follows: ; Let the target navigation strategy weight be... The obstacle navigation strategy weights are ,satisfy: ， The weights are generated based on environmental assessment indicators, ultimately resulting in the basic fusion navigation actions. : ， in, This represents the basic navigation action after weighted fusion of the target navigation strategy and the obstacle navigation strategy; Let the physical state vector of the UAV at time t be... for: ， in: Indicates the roll angle; Indicates the pitch angle; Indicates the yaw angle; Indicates speed; Indicates acceleration; Indicates angular velocity; Introducing a Long Short-Term Memory (LSTM) network to model historical flight states: Assume a sequence of states of the UAV over a past period. for: ， in, This represents the drone status within the g-th historical time window, where g ranges from 1 to k, and k is the total number of historical windows. Extracting context state using an LSTM network: ， in, This indicates the current flight context state; The current flight context state As input to the reinforcement learning reward function and policy network, the drone's actions are constrained to meet the actual dynamic characteristics to achieve trajectory optimization. The fine-grained action correction output by the trajectory optimization policy network is: ， in, Represents the trajectory optimization strategy network; This indicates the amount of fine-tuning for the basic movement; Final optimized drone actions Represented as: ， Construct the following reinforcement learning reward function : ， in, Indicates a stable attitude reward; Indicates a smooth reward for the action; Indicates a dynamic constraint reward; , , The weights for each reward item.

9. An autonomous obstacle avoidance and navigation system for unmanned aerial vehicles (UAVs) based on a visual language action model and a world model for implementing the method as described in any one of claims 1 to 8, characterized in that, It includes a target semantic active perception and action generation module, an environmental semantic world model obstacle avoidance and navigation module, and an adaptive strategy fusion and trajectory optimization module; The target semantic active perception and action generation module is used to receive environmental images collected by a visual sensing device installed at the front end of the UAV, and input the environmental images and text instructions containing target semantics into a Transformer-based visual text action (VLA) network. The module also filters the importance of target objects, orientation information, action constraint information, and environmental cues in the text instructions to obtain filtered semantic text instructions, and extracts image region features and text features through a visual encoder and a text encoder, respectively. Furthermore, the module introduces an intermediate alignment state to establish a two-way collaborative training constraint between text semantics and image region semantics, and calculates the matching degree between each visual region and each text head based on a multi-head attention mechanism, selecting the visual region with the highest matching degree as the key visual region to generate target text-guided navigation actions. When the matching degree is lower than a preset threshold, the module predicts the possible direction of the target outside the field of view based on the visual text semantic fusion results within the current visual range, and controls the UAV to perform a rotational perception action to re-acquire environmental images and update the target text-guided navigation actions. The environmental semantic world model obstacle avoidance and navigation module is used to receive environmental sensor data, visual image information, and target semantic information of the key visual region from the UAV, and to construct the world model input state. The module also maps the current world model input state to a latent state representation using a world model state encoder, and uses a joint embedding prediction architecture (JEPA-style predictor) to predict obstacle semantic representations and target semantic representations in the latent space. Furthermore, the module performs continuous vector encoding on the target semantic representation and obstacle semantic representation using a continuous representation encoder to obtain a continuous deterministic semantic representation at the current moment, and predicts the continuous deterministic semantic representation at the next moment. Finally, the module constructs a prediction loss based on the distance between the predicted representation and the true representation, constructs a world model reward function based on the obstacle semantic representation and the target semantic representation, and uses a dual-delay deep deterministic policy gradient (TD3) offline reinforcement learning network to generate an obstacle navigation policy output based on environmental semantic obstacle avoidance and target approach. The adaptive strategy fusion and trajectory optimization module is used to construct navigation decision evaluation indicators based on the current environmental state information obtained by the visual sensor, distance sensor, depth sensor, lidar, inertial measurement unit, and environmental perception module. These indicators include the complexity of environmental obstacles, the degree of environmental risk, the distance or estimated distance between the UAV and the target, and the deviation between the target direction and the UAV's current heading. The module also dynamically generates target navigation strategy weights and obstacle navigation strategy weights based on the navigation decision evaluation indicators, and performs weighted fusion of the target text-guided navigation action and the obstacle navigation strategy output to obtain a basic fused navigation action. Furthermore, the module extracts the flight context state using a Long Short-Term Memory (LSTM) network based on the UAV's current physical state vector and historical flight state sequence. This flight context state is used as input to the reinforcement learning reward function and the trajectory optimization strategy network, outputting a fine-grained action correction amount for the basic fused navigation action to obtain the final optimized UAV flight control action.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images