Unmanned aerial vehicle vision navigation method and system based on world modeling

Abstract

The application provides a kind of unmanned aerial vehicle vision navigation method and system based on world modeling, comprising: obtaining the current observation data of unmanned aerial vehicle, and based on the future scene generation model and the current observation data of pre-construction, the future scene prediction is carried out, and the future scene feature of unmanned aerial vehicle is obtained;The current observation data is encoded, and the feature of the current observation data is obtained, and based on the feature of the current observation data and the future scene feature, a joint feature set is obtained;The feature in the joint feature set is pruned, and a target feature set is obtained;The target feature in the target feature set is decoded, and the unmanned aerial vehicle control signal is obtained.The application reduces the delay of unmanned aerial vehicle navigation and improves the robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle navigation, in particular to an unmanned aerial vehicle visual navigation method and system based on world modeling. BACKGROUND

[0002] At present, the unmanned aerial vehicle navigation mainly adopts the following ways:

[0003] (1) Traditional GNSS / INS integrated navigation: In open airspace, the unmanned aerial vehicle fuses the received GPS / Beidou satellite signal with the on-board inertial measurement unit (IMU) to position, and estimates the position, velocity and attitude by using Kalman filter. This method depends on satellite signals, and is prone to drift or loss of lock in high-rise valley, indoor or electromagnetic interference environment.

[0004] (2) Visual-inertial navigation (VIO / SLAM): Single, binocular or RGB-D cameras are used to obtain continuous image frames, and ORB, SIFT and other feature points are extracted or optical flow is directly used, and the unmanned aerial vehicle pose is estimated in real time under the filtering (MSCKF, ESKF) or optimization (BA, factor graph) framework combined with IMU data. The representative frameworks are OKVIS, VINS-Mono and ORB-SLAM3. The advantage is that no external beacon is needed, and the disadvantage is that it is sensitive to light changes, texture loss and high-speed motion, and there is a cumulative error in long-time scene.

[0005] (3) Light radar-inertial navigation (LIO): 3D LiDAR scans the environment point cloud, and uses ICP, NDT or LOAM series algorithms to tightly couple with IMU to achieve centimeter-level positioning. This method is resistant to light interference, but the device is heavy and expensive, and it is prone to degradation in scenes with few geometric features (such as long corridors and open squares).

[0006] (4) End-to-end navigation based on deep learning: Convolutional neural network (CNN) or Transformer is used to directly map the original image to control instructions (linear velocity, angular velocity or quaternion throttle). The network is usually trained offline on large-scale simulation or artificially collected data, and online inference does not require explicit mapping. The advantage is simple deployment and fast response, but the generalization is limited to the distribution of training data, and it is difficult to cope with extreme scenes that have never been seen.

[0007] (5) Map-based planning-control separation framework: First, an accurate 2.5D / 3D map is constructed by offline SLAM or mapping means, and then path planning (A*, RRT*, MPC) and trajectory tracking are performed online. This framework has been commercialized in structured environments (warehouses, parks), but the map update cost is high and the response to new obstacles is lagging.

[0008] (6) Hybrid solution: cascade the above methods: GNSS provides global reference, VIO / LIO provides local refinement, and deep learning provides semantic understanding and obstacle avoidance decision. Although the advantages of each are integrated, the system link is complex, the computing resource occupation is large, and the real-time performance decreases with the increase of modules. SUMMARY

[0009] Therefore, the present application aims to provide a UAV visual navigation method and system based on world modeling, which reduces the delay of UAV navigation and improves the robustness.

[0010] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0011] In a first aspect, the embodiments of the present application provide a UAV visual navigation method based on world modeling, comprising: obtaining current observation data of a UAV, and predicting a future scene based on a pre-constructed future scene generation model and the current observation data to obtain future scene features of the UAV; encoding the current observation data to obtain features of the current observation data, and obtaining a joint feature set based on the features of the current observation data and the future scene features; pruning the features in the joint feature set to obtain a target feature set; and decoding the target features in the target feature set to obtain a UAV control signal.

[0012] Optionally, before obtaining the current observation data of the UAV, it further comprises: encoding dynamic targets and static targets in a space-time scene where the UAV is located to obtain a scene model of the space-time scene where the UAV is located; adding an independent noise representation to each future time step of each dynamic target encoding based on the scene model; and training the future scene generation model based on the dynamic target encoding and the static target encoding with added noise representation, with minimizing noise prediction error as the training target, to obtain a trained future scene generation model.

[0013] Optionally, predicting the future scene based on the pre-constructed future scene generation model and the current observation data to obtain the future scene features of the UAV comprises: encoding the current observation data into an initial dynamic target tensor; generating a UAV state sequence for a future preset time period based on the future scene generation model and the initial dynamic target tensor using a DDIM inverse diffusion 32-step method, and discretizing the UAV state sequence to obtain the future scene features of the UAV.

[0014] Optionally, the current observation data comprises: an RGB image, a depth map, pose information, a task language instruction, and a global map; the current observation data is encoded to obtain a feature of the current observation data, comprising: the RGB image is encoded based on a visual encoder to obtain a visual representation; the depth map is subjected to convolution calculation based on a depth encoder to obtain a depth representation, and the depth representation and the visual representation are added to obtain a fused visual feature; a language instruction representation of the task language instruction is obtained based on a language instruction encoder, and the language instruction representation is interacted using a self-attention mechanism to obtain a language instruction feature; the pose information is encoded based on an IMU encoder to obtain an IMU tensor, and the IMU tensor is copied and spliced to each fused visual feature; the global map is divided into a plurality of non-overlapping blocks based on a map encoder, and each non-overlapping block is subjected to convolution operation and then flattened to obtain a global map feature.

[0015] Optionally, based on the feature of the current observation data and the future scene feature, a joint feature set is obtained, comprising: the future scene feature is subjected to linear mapping to obtain a target future scene feature; the joint feature set is constructed based on the fused visual feature, the language instruction feature, the global map feature, and the target future scene feature; the significance weight of each feature in the joint feature set is calculated to obtain a semantic significance vector.

[0016] Optionally, the features in the joint feature set are pruned to obtain a target feature set, comprising: the features in the joint feature set are sorted according to the significance weight to obtain an ordered list; a current feature is taken from the head of the ordered list in turn, and the effective similarity between the current feature and each target feature in the target feature set is calculated; if the effective similarity is greater than a preset similarity threshold, the current feature is deleted, otherwise the current feature is saved to the target feature set, until the number of target features in the target feature set reaches a preset value.

[0017] Optionally, the target features in the target feature set are decoded to obtain a UAV control signal, comprising: the target features in the target feature set are decoded using a two-layer cross-attention network and an MLP lightweight decoder to obtain a UAV control signal; wherein the UAV control signal comprises: a roll angle, a pitch angle, a yaw angle, and a throttle of the UAV.

[0018] In a second aspect, an embodiment of the present application provides a UAV visual navigation system based on world modeling, comprising: a world modeling module configured to obtain current observation data of a UAV, and perform future scene prediction based on a pre-constructed future scene generation model and the current observation data to obtain future scene features of the UAV; a multi-modal encoding module configured to encode the current observation data to obtain features of the current observation data, and obtain a joint feature set based on the features of the current observation data and the future scene features; a pruning module configured to prune features in the joint feature set to obtain a target feature set; and an action decoding module configured to decode target features in the target feature set to obtain a UAV control signal.

[0019] In a third aspect, an embodiment of the present application provides a UAV, comprising a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement steps of the method of any one of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement steps of the method of any one of the first aspect.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] The UAV visual navigation method and system based on world modeling provided by the present application first obtain current observation data of a UAV, and perform future scene prediction based on a pre-constructed future scene generation model and the current observation data to obtain future scene features of the UAV; then encode the current observation data to obtain features of the current observation data, and obtain a joint feature set based on the features of the current observation data and the future scene features; then prune features in the joint feature set to obtain a target feature set; and finally decode target features in the target feature set to obtain a UAV control signal. In the above method, potential representations of a future scene, i.e. future scene features, can be obtained based on a future scene generation model; then a joint pruning strategy is used to prune a joint feature set of the current observation data and the future scene features; and finally the pruned target features are input into an action decoder to obtain a UAV control signal, thereby realizing low-delay and high-robustness UAV navigation.

[0023] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by means of the structures particularly pointed out in the description and the appended claims.

[0024] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for a detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0026] Figure 1 A flow chart of a world modeling-based unmanned aerial vehicle visual navigation method provided by an embodiment of the present application;

[0027] Figure 2 A world modeling-based unmanned aerial vehicle visual navigation flow chart provided by an embodiment of the present application;

[0028] Figure 3 A structural schematic diagram of a world modeling-based unmanned aerial vehicle visual navigation system provided by an embodiment of the present application;

[0029] Figure 4 A structural schematic diagram of an unmanned aerial vehicle provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be described clearly and completely below in combination with the drawings. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0031] At present, the common unmanned aerial vehicle navigation mode has the following problems:

[0032] (1) Strong dependence on environment: GNSS loses lock in high-rise / indoor / electromagnetic interference environment, VIO drifts in weak texture or strong light scene, and LIO fails in open geometry degradation area.

[0033] (2) Reaction lag to dynamic obstacles: traditional SLAM or map-planning framework only processes static environment; end-to-end network often only depends on single-frame image and lacks foresight for suddenly appearing targets.

[0034] (3)Computational resource bottleneck: VIO / LIO needs real-time BA or ICP, end-to-end Transformer needs to process hundreds of tokens; it is difficult to simultaneously meet low latency and high accuracy on onboard Jetson / FPGA.

[0035] (4)No effective correction mechanism for accumulated error: pure vision / INS error diverges after long navigation time; offline map cannot be updated online.

[0036] (5)Weak generalization ability: deep end-to-end method is poor in robustness to scenes outside the training distribution (new weather, new obstacle category).

[0037] (6)Complex system link and parameter coupling: traditional hybrid scheme needs to tune GNSS, IMU, VIO, obstacle avoidance, and planning respectively, resulting in long debugging period.

[0038] Based on this, the embodiment of the application provides a kind of unmanned aerial vehicle vision navigation method and system based on world modeling, to reduce the delay of unmanned aerial vehicle navigation, improve robustness.

[0039] In order to facilitate the understanding of the present embodiment, first of all, a kind of based on world modeling unmanned aerial vehicle vision navigation method disclosed in the embodiment of the application is introduced in detail, which is applied to the unmanned aerial vehicle vision navigation system based on world modeling, can be executed by unmanned aerial vehicle, the system includes: world modeling device, multi-modal encoder, Token pruner, action decoder.

[0040] The world modeling device aims to predict the dynamic evolution of the space-time where the unmanned aerial vehicle is located after taking different actions in the future period of time based on the observation data of the unmanned aerial vehicle at the current time, and needs to meet three key requirements, which are the diversity of generated scenes (combined with different actions), the controllability of other space-time participants (such as possible collision), and the dynamic interaction ability with space-time domain (interaction with other agent actions and static environment).

[0041] The multi-modal encoder encodes the current observation into a set of latent vectors that can be seamlessly spliced with future scene tokens.

[0042] The Token pruner removes redundant tokens after the multi-modal encoder outputs tokens and before the action decoder, with zero training overhead, to achieve effective compression of tokens, improve the running efficiency of subsequent modules, and almost not lose navigation accuracy.

[0043] The action decoder receives the key tokens output by the Token pruner and decodes them to output 4-DoF unmanned aerial vehicle control signals.

[0044] Referring to Figure 1A flowchart of a world modeling based unmanned aerial vehicle visual navigation method is shown, which shows that the method mainly includes the following steps S101 to S104:

[0045] Step S101: Obtain the current observation data of the unmanned aerial vehicle, and perform future scene prediction based on the pre-constructed future scene generation model and the current observation data to obtain the future scene features of the unmanned aerial vehicle.

[0046] In an embodiment, the world modeler first encodes dynamic targets and static targets in the space-time scene where the unmanned aerial vehicle is located to obtain a scene model of the space-time scene where the unmanned aerial vehicle is located. Specifically, first, the space-time scene where the unmanned aerial vehicle is located is encoded as a structure scene. For dynamic objects, they are encoded as , wherein, represents the maximum number of dynamic targets in the predefined scene, represents the time step (including the historical time step and the future time step), represents the attribute dimension, and the dynamic target attributes include: the position of the dynamic target in the coordinate system centered on the unmanned aerial vehicle , heading angle, speed, size; for static objects, they are encoded as , wherein, represents the maximum number of static objects in the space-time domain where the unmanned aerial vehicle is located, and respectively represent the length and width of the static object in the overhead coordinate system, represents the static object attribute dimension.

[0047] Then, based on the scene model, an independent noise representation is added for each future time step of each dynamic target. Specifically, based on the scene modeling, the future scene generation task is to predict the future target coding based on the historical dynamic target coding and the static object coding. In the embodiment of the present application, in order to realize fine-grained control of dynamic targets, an independent noise representation is allocated for each future time step of each dynamic target , a noise matrix is constructed, for low noise regions, i.e. the historical information of the dynamic target is retained, so as to guide the scene generation trend to be close to this, for high noise regions, i.e. then give the model greater freedom to dynamically adjust, so as to better respond to environmental changes.

[0048] Finally, based on the dynamic target coding and the static target coding with added noise representation, the future scene generation model is trained with the training target of minimizing the noise prediction error to obtain the trained future scene generation model.

[0049] Specifically, the training target of the future scene generation model is to reconstruct the future sequence from the partially noisy sequence, therefore, in the embodiment of the present application, the future scene generation model is trained by minimizing the noise prediction error, that is:

[0050]

[0051] wherein, represents the original noise, is a denoising model, represents the initial data is added with a noise matrix , and is the static object code and other condition information.

[0052] Based on this, in the embodiment of the present application, after obtaining the current observation data of the unmanned aerial vehicle, the future scene can be sampled and predicted based on the future scene generation model, and the future scene feature of the unmanned aerial vehicle, that is, the future scene token.

[0053] Step S102: encoding the current observation data to obtain the feature of the current observation data, and obtaining a joint feature set based on the feature of the current observation data and the future scene feature.

[0054] In one embodiment, the multi-modal encoder encodes the current observation data to obtain the feature of the current observation data, and splices the feature of the current observation data and the future scene feature to obtain the joint feature set.

[0055] Step S103: pruning the features in the joint feature set to obtain a target feature set.

[0056] In one embodiment, the Token pruner prunes the features in the joint feature set to select the target feature set, which maximizes the semantic saliency while maintaining the token diversity.

[0057] Step S104: decoding the target features in the target feature set to obtain the unmanned aerial vehicle control signal.

[0058] In one embodiment, the action decoder decodes the target features in the target feature set output by the Token pruner to obtain the unmanned aerial vehicle control signal.

[0059] The unmanned aerial vehicle vision navigation method based on world modeling provided by the embodiment of the application can obtain potential representation of a future scene, i.e., future scene features, based on a future scene generation model, jointly prune a joint feature set of current observation data and the future scene features by using a pruning strategy, and finally send the pruned target features to an action decoder to obtain an unmanned aerial vehicle control signal by decoding, so as to realize low-delay and high-robustness unmanned aerial vehicle navigation.

[0060] In an embodiment, for the foregoing step S101, i.e., when the future scene features of the unmanned aerial vehicle are obtained by predicting the future scene based on the pre-constructed future scene generation model and the current observation data, the following methods can be used, but are not limited to the following methods: first, the current observation data is encoded into an initial dynamic target tensor; then, based on the future scene generation model and the initial dynamic target tensor, the DDIM inverse diffusion 32-step method is used to generate the unmanned aerial vehicle state sequence in the future preset time period, and the unmanned aerial vehicle state sequence is discretized to obtain the future scene features of the unmanned aerial vehicle.

[0061] In specific implementation, for the current observation data of the unmanned aerial vehicle , it is encoded into an initial dynamic target tensor , then the DDIM inverse diffusion 32-step method is used to obtain the unmanned aerial vehicle state sequence in the future frame , and the unmanned aerial vehicle state sequence is discretized into future scene representation tokens:

[0062]

[0063] , wherein, is the number of discrete tokens for each frame, is a VQ-GAN encoder.

[0064] In an embodiment, the current observation data includes: an RGB image , a depth map , attitude information (i.e., IMU six-axis data) , task language instructions , and a global map . For the foregoing step S102, i.e., when the current observation data is encoded to obtain the features of the current observation data, the following methods can be used, but are not limited to the following methods:

[0065] (1) The RGB image is encoded based on a visual encoder to obtain visual representation. Specifically, the visual encoder uses two parallel ways to extract 256 patch visual tokens using CLIP-VIT-L, and uses EfficientNet-B3 combined The convolution process yields 64 visual tokens, which are then fused using a cross-attention mechanism to obtain a visual representation. in The answer is 256 + 64 = 320.

[0066] (2) Convolution calculation is performed on the depth map based on the depth encoder to obtain the depth representation, and the depth representation and visual representation are added together to obtain the fused visual features. Specifically, the depth encoder... Depth map Perform convolution calculations to obtain the depth representation. Furthermore, the visual token and the depth token are added element-wise to obtain the fused visual features. .

[0067] (3) Based on the language instruction encoder, language instruction representations of task language instructions are obtained, and a self-attention mechanism is used to interact with these representations to obtain language instruction features. Specifically, the language instruction encoder... Sixty-four language instruction tokens were obtained using the phi-1.5 sampler. Furthermore, a self-attention mechanism was used to interact with these language instruction tokens to obtain language instruction features. ,in It is 64.

[0068] (4) The pose information is encoded based on the IMU encoder to obtain the IMU tensor, and the IMU tensor is copied and concatenated to each fused visual feature. Specifically, the IMU encoder... First, the IMU data is input into a 3-layer 1D convolutional neural network and a global average pooling layer to obtain the IMU tensor. Then copy it. The images are then stitched together to each path to fuse visual features, thereby injecting posture information.

[0069] (5) Based on the map encoder, the global map is divided into multiple non-overlapping blocks, and each non-overlapping block is convolved and flattened to obtain the global map features. Specifically, the map encoder... First, divide the global map into... Non-overlapping blocks, each block is processed separately After convolution, flatten the image to obtain global map features. ,in It is 64.

[0070] Furthermore, when obtaining a joint feature set based on the features of current observation data and future scene features, the following methods can be used, including but not limited to:

[0071] First, linearly map the future scene features to obtain target future scene features. Specifically, for future scene tokens, to ensure that they are the same distribution as visual tokens, input them into a linear mapping layer to obtain target future scene features: .

[0072] Then, based on the fusion of visual features, language instruction features, global map features and target future scene features, a joint feature set is constructed. Specifically, the above encoding outputs a unified token set (that is, the joint feature set):

[0073]

[0074] Finally, the significance weight of each feature in the joint feature set is calculated to obtain a semantic significance vector.

[0075] Specifically, based on the above encoding, a joint shallow space constraint is established:

[0076]

[0077] The significance weight of each token is calculated :

[0078]

[0079] wherein is the prediction confidence output by the world modeler. Based on the weight of each token, a semantic significance vector is obtained. The token set and the semantic significance vector are input to the Token Pruner.

[0080] In one embodiment, after the multi-modal encoder outputs 516 tokens (current observation 452 + future scene 64), for the joint token set , the goal of the Token Pruner is to select a token subset , while maximizing semantic significance and maintaining token diversity:

[0081]

[0082] wherein is the semantic significance weight, is the semantic similarity threshold.

[0083] Based on this, for the foregoing step S103, that is, when the features in the joint feature set are pruned to obtain the target feature set, the following methods can be used, including but not limited to: first, the features in the joint feature set are sorted according to the semantic saliency weight to obtain an ordered list; then, the current feature is taken out from the head of the ordered list in turn, and the effective similarity between the current feature and each target feature in the target feature set is calculated; if the effective similarity is greater than a preset similarity threshold (i.e., a semantic similarity threshold), the current feature is deleted, otherwise the current feature is saved to the target feature set, until the number of target features in the target feature set reaches a preset value.

[0084] In a specific implementation, first, the 516 tokens in the joint feature set are sorted from high to low according to the semantic saliency weight to obtain an ordered list U, and an empty set S is prepared as a target feature set for storing the final retained key token (i.e., target feature). As long as the size of S has not reached the budget (i.e., a preset value, such as 64 / 128 / 192), and there are still tokens in the list U, the following steps are continued:

[0085] (1) the current "most important" token p (i.e. the largest feature) is taken out from the head of the ordered list U;

[0086] (2) the effective similarity between the token p and each token s in the target feature set is calculated; when the effective similarity between any token s and the token p exceeds the similarity threshold , the token p is marked as a redundant feature, discarded directly and the next round is continued; otherwise, the token p is added to the target feature set S.

[0087] When the loop ends, the target feature set S retains important and mutually dissimilar tokens; at the same time, a Boolean mask mask is generated to mark which original tokens are retained and which are discarded, based on which the token set S and the mask mask are obtained for the next action decoder to use directly.

[0088] In an embodiment, for the foregoing step S104, that is, when the target features in the target feature set are decoded to obtain the UAV control signal, the following methods can be used, including but not limited to: a two-layer cross-attention network and an MLP lightweight decoder are used to decode the target features in the target feature set to obtain the UAV control signal; wherein the UAV control signal includes: the roll angle, the pitch angle, the yaw angle, and the throttle of the UAV.

[0089] In specific implementation, the action decoder receives the key tokens output by the Token Pruner , and decodes using a two-layer cross-attention mechanism and an MLP lightweight decoder:

[0090]

[0091] Output 4-DoF drone control signals: roll angle, pitch angle, yaw angle, and throttle.

[0092] In the embodiment of the application, two-stage training is adopted for training, the first stage is to pre-train the world modeler, and the second stage is to jointly train the complete navigation model.

[0093] For ease of understanding, the embodiment of the application further provides a world modeling-based unmanned aerial vehicle visual navigation flowchart, as shown in Figure 2 , mainly comprising:

[0094] World modeler: based on the current observation of the unmanned aerial vehicle, the dynamic evolution of the space-time in which the unmanned aerial vehicle is located after the unmanned aerial vehicle takes different actions in the future period of time is speculated, and the future scene representation token is obtained.

[0095] Multimodal encoder: unify the "current observation" into a set of latent vectors that can be seamlessly spliced with the future scene token, and calculate the semantic saliency weight.

[0096] Token Pruner: the multimodal encoder outputs 516 tokens (after the action decoder, the redundant tokens are removed in a zero training overhead manner, the token is effectively compressed, and the subsequent module operation efficiency is improved while the navigation accuracy is almost not lost.

[0097] Action decoder: receives the key tokens output by the Token Pruner, and uses a two-layer cross-attention + MLP lightweight decoder to output 4-DoF unmanned aerial vehicle control signals.

[0098] The world modeling-based unmanned aerial vehicle visual navigation method provided by the embodiment of the application has the following technical effects:

[0099] (1) Improved environmental robustness

[0100] The world modeler of the application explicitly models the "future 8 unmanned aerial vehicle and obstacle state" as AgentTensor, and completes the prediction in the latent space rather than the original pixel space. In the reasoning stage, only 32 steps of DDIM are required to obtain 16 frames of future state. Since the latent space dimension is fixed and not affected by low-level noise such as illumination and texture, even if the satellite loses lock or the features are sparse, the average positioning error can still be significantly lower than the VIO paradigm.

[0101] (2) Dynamic obstacle anticipation

[0102] The application uses the diffusion model to predict future collision risks (collision probability channels of dynamic target tokens) in the latent space, and then retains high salient tokens through a pruner, and the action decoder makes evasive maneuvers 0.5-2.0s in advance accordingly.

[0103] (3) Dual optimization of computation and power consumption

[0104] The application adopts a joint criterion of "semantic saliency + cosine diversity" in the pruning stage, and compresses 516 tokens to 64-192 tokens with zero training overhead. After pruning, the video memory is reduced to 1.1 GB, the measured delay of Flash-Attention is 4.3 ms, the power consumption is <8W, and it can run in real time on Jetson Orin NX 16 GB.

[0105] (4) Long-endurance error suppression

[0106] The world modeler reinitializes the future sequence every 0.5s with the latest observation, which is equivalent to real-time loop in the latent space without additional feature matching.

[0107] (5) Enhanced generalization ability

[0108] The application uses the scene synthesized in the diffusion stage, and the pruner forcibly retains cross-scene high salient tokens, so that the task success rate of the model is improved under 5 unseen weather conditions.

[0109] (6) Reduced system complexity and debugging period

[0110] The application uses a unified latent space token as an intermediate representation, and all modules share the same loss function, so the total number of parameters is greatly reduced compared with the "VIO+MPC" scheme.

[0111] In summary, the application couples "predicting the future" and "real-time compression" into an end-to-end framework through "world modeling + token pruning", and simultaneously improves the positioning accuracy, dynamic obstacle avoidance, computation efficiency, long-endurance robustness, and cross-scene generalization without relying on external signals or increasing hardware.

[0112] For the world modeling-based unmanned aerial vehicle visual navigation method provided by the foregoing embodiments, an embodiment of the application further provides a world modeling-based unmanned aerial vehicle visual navigation system, which refers to a structure schematic diagram of a world modeling-based unmanned aerial vehicle visual navigation system as shown in Figure 3 The system can include the following parts:

[0113] The world modeling module 301 is configured to acquire current observation data of the UAV, and perform future scene prediction based on a pre-constructed future scene generation model and the current observation data to obtain future scene features of the UAV.

[0114] The multi-modal encoding module 302 is configured to encode the current observation data to obtain feature of the current observation data, and obtain a joint feature set based on the feature of the current observation data and the future scene features.

[0115] The pruning module 303 is configured to prune the features in the joint feature set to obtain a target feature set.

[0116] The action decoding module 304 is configured to decode the target features in the target feature set to obtain the UAV control signal.

[0117] The UAV visual navigation system based on world modeling provided by the embodiment of the present application can obtain potential representation of the future scene, i.e., the future scene features, based on the future scene generation model, jointly prune the joint feature set of the current observation data and the future scene features by using a pruning strategy, and finally send the pruned target features into the action decoder to obtain the UAV control signal by decoding, so as to realize low-delay and high-robust UAV navigation.

[0118] The system provided by the embodiment of the present application has the same implementation principle and technical effects as the foregoing method embodiment, and for brevity of description, the part not mentioned in the system embodiment can be referred to the corresponding content in the foregoing method embodiment.

[0119] The embodiment of the present application further provides a UAV, specifically, the UAV comprises a processor and a storage device; the storage device stores a computer program, and the computer program performs the method described in any one of the above embodiments when executed by the processor.

[0120] Figure 4 A structural schematic diagram of a UAV provided by the embodiment of the present application is shown in the figure, the UAV 100 comprises a processor 40, a memory 41, a bus 42 and a communication interface 43, the processor 40, the communication interface 43 and the memory 41 are connected through the bus 42; the processor 40 is configured to execute the executable modules stored in the memory 41, such as a computer program.

[0121] The memory 41 can include a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0122] The bus 42 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0123] The memory 41 is used to store programs, and the processor 40 executes the programs after receiving execution instructions. The method executed by the device defined by the flow process disclosed in any of the embodiments of the application can be applied to the processor 40 or implemented by the processor 40.

[0124] The processor 40 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 40 or the instruction in the form of software. The processor 40 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and combines the hardware to complete the steps of the above method.

[0125] The computer program product of the readable storage medium provided by the embodiment of the present application comprises a computer readable storage medium storing program codes, and the program codes comprise instructions for executing the method described in the foregoing method embodiments. The specific implementation can be referred to the foregoing method embodiments, and will not be described here.

[0126] The function, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art or the part of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0127] Finally, it should be noted that: the above-described embodiments are merely specific implementations of the present application, used to illustrate the technical solutions of the present application, and not to limit the same. The protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical range disclosed by the present application, or replace some technical features with equivalent replacements; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for visual navigation of a UAV based on world modeling, the method comprising: The method comprises the following steps: obtaining current observation data of a UAV, and performing future scene prediction based on a pre-constructed future scene generation model and the current observation data to obtain future scene features of the UAV; wherein the current observation data comprises an RGB image, a depth map, attitude information, a task language instruction and a global map; encoding the current observation data to obtain features of the current observation data, and obtaining a joint feature set based on the features of the current observation data and the future scene features; wherein the joint feature set comprises fused visual features, language instruction features, global map features and target future scene features; obtaining the joint feature set based on the features of the current observation data and the future scene features comprises: performing linear mapping on the future scene features to obtain target future scene features; constructing the joint feature set based on the fused visual features, the language instruction features, the global map features and the target future scene features; calculating the significance weight of each feature in the joint feature set to obtain a semantic significance vector; wherein a joint shallow space constraint is established based on the fused visual features and the target future scene features: calculating the significance weight of each feature, and obtaining a semantic significance vector based on the weight of each feature: wherein, represents a fused visual feature, represents a linguistic instruction feature, represents a global map feature, represents a target future scenario feature, is a prediction confidence output by the world modeler, is a saliency weight for the fused visual feature, is a saliency weight for the global map feature, is a saliency weight for the target future scenario feature; pruning the features in the joint feature set to obtain a target feature set; decoding the target features in the target feature set to obtain a UAV control signal; wherein the UAV control signal comprises a roll angle, a pitch angle, a yaw angle and a throttle of the UAV; pruning the features in the joint feature set to obtain a target feature set, comprising: sorting the features in the joint feature set according to the significance weight to obtain an ordered list; sequentially taking a current feature from the head of the ordered list, and calculating the effective similarity between the current feature and each target feature in the target feature set; if the effective similarity is greater than a preset similarity threshold, the current feature is deleted, otherwise the current feature is saved to the target feature set, until the number of target features in the target feature set reaches a preset value.

2. The method of claim 1, wherein, Before obtaining the current observation data of the UAV, the method further comprises the following steps: encoding dynamic targets and static targets in a space-time scene where the UAV is located to obtain a scene model of the space-time scene where the UAV is located; adding an independent noise representation to each future time step of each dynamic target encoding based on the scene model; training the future scene generation model based on the dynamic target encoding and the static target encoding with added noise representations, with minimizing noise prediction error as a training target, to obtain a trained future scene generation model.

3. The method of claim 1, wherein, performing future scene prediction based on the pre-constructed future scene generation model and the current observation data to obtain the future scene features of the UAV, comprising: encoding the current observation data into an initial dynamic target tensor; Based on the future scene generation model and the initial dynamic target tensor, a DDIM reverse diffusion 32-step method is used to generate a future preset time period UAV state sequence, and the UAV state sequence is discretely processed to obtain the future scene feature of the UAV.

4. The method of claim 1, wherein, The current observation data includes: an RGB image, a depth map, attitude information, a task language instruction, and a global map. The current observation data is encoded to obtain the features of the current observation data, including: The RGB image is encoded based on a visual encoder to obtain a visual representation; The depth map is convoluted based on a depth encoder to obtain a depth representation, and the depth representation and the visual representation are added to obtain a fused visual feature; The language instruction representation of the task language instruction is obtained based on a language instruction encoder, and the language instruction representation is interacted using a self-attention mechanism to obtain a language instruction feature; The attitude information is encoded based on an IMU encoder to obtain an IMU tensor, and the IMU tensor is copied and spliced to each fused visual feature; The global map is divided into multiple non-overlapping blocks based on a map encoder, and each non-overlapping block is convoluted and flattened to obtain a global map feature.

5. The method of claim 1, wherein, The target features in the target feature set are decoded to obtain a UAV control signal, including: A two-layer cross-attention network and an MLP lightweight decoder are used to decode the target features in the target feature set to obtain a UAV control signal.

6. A world modeling based unmanned aerial vehicle vision navigation system, characterized in that, It includes: A world modeling module is configured to obtain current observation data of a UAV, and predict a future scene based on a pre-constructed future scene generation model and the current observation data to obtain a future scene feature of the UAV; wherein the current observation data includes: an RGB image, a depth map, attitude information, a task language instruction, and a global map; A multi-modal encoding module is configured to encode the current observation data to obtain features of the current observation data, and obtain a joint feature set based on the features of the current observation data and the future scene feature; wherein the joint feature set includes: a fused visual feature, a language instruction feature, a global map feature, and a target future scene feature; obtaining a joint feature set based on the features of the current observation data and the future scene feature includes: linearly mapping the future scene feature to obtain a target future scene feature; constructing a joint feature set based on the fused visual feature, the language instruction feature, the global map feature, and the target future scene feature; calculating the significance weight of each feature in the joint feature set to obtain a semantic significance vector; wherein a joint shallow space constraint is established based on the fused visual feature and the target future scene feature: The significance weight of each feature is calculated, and a semantic significance vector is obtained based on the weight of each feature: wherein, represents a fused visual feature, represents a linguistic instruction feature, represents a global map feature, represents a target future scenario feature, is a prediction confidence output by the world modeler, is a saliency weight for the fused visual feature, is a saliency weight for the global map feature, is a saliency weight for the target future scenario feature; A pruning module is configured to prune the features in the joint feature set to obtain a target feature set. An action decoding module is configured to decode the target features in the target feature set to obtain a UAV control signal, wherein the UAV control signal comprises a roll angle, a pitch angle, a yaw angle and a throttle of the UAV. The pruning module is specifically configured to: sort the features in the joint feature set according to the significance weights to obtain an ordered list; sequentially take out a current feature from a head of the ordered list, and calculate an effective similarity between the current feature and each target feature in the target feature set; if the effective similarity is greater than a preset similarity threshold, delete the current feature, otherwise save the current feature to the target feature set, until the number of target features in the target feature set reaches a preset value.

7. A drone, characterized in that, A processor and a memory are included, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is run by the processor to execute the steps of the method in any one of claims 1 to 5.

Images