Sparse representation based unmanned aerial vehicle visual language action navigation method and system

Abstract

The application provides a kind of unmanned aerial vehicle visual language action navigation method and system based on sparsification representation, by converting the obtained image into multiple visual processing units, and the key visual processing unit in multiple visual processing units is determined by sparsification processing.Remaining other visual processing units are subjected to context merging processing, to obtain merged visual processing units.Combining key visual processing units and merged visual processing units obtains visual processing unit set.Based on the obtained language instruction and visual processing unit set, a predicted action sequence is generated, and the navigation control of the unmanned aerial vehicle is executed based on the predicted action sequence.In this scheme, by sparsification processing of visual processing unit, the processing amount can be reduced, and navigation can be more accurate and specific by combining the language instruction of the user.

Description

Technical Field

[0001] This invention relates to the field of navigation technology, and more specifically, to a visual language action navigation method and system for unmanned aerial vehicles (UAVs) based on sparse representation. Background Technology

[0002] With the widespread application of unmanned aerial vehicles (UAVs) in both civilian and military fields, improving their autonomous navigation capabilities has become a research hotspot. Traditional UAV navigation relies on GPS signals or pre-defined path planning; however, these methods perform poorly in complex environments, lacking flexibility and real-time performance. In recent years, Visual Language Models (VLMs) have combined image recognition and natural language processing capabilities, providing new navigation or action methods for intelligent agents, autonomous vehicles, and other applications.

[0003] Most existing Visual Models (VLMs) require a large number of visual processing units (tokens) to achieve high-performance scene understanding or navigation, which not only consumes significant computational costs but may also lead to unnecessary redundant information. Drones need to process large amounts of visual data, resulting in wasted computational resources and reduced response speed. Furthermore, the injection of redundant information may cause the model to fail to focus on key regions in the image, thus hindering accurate navigation. Summary of the Invention

[0004] The purpose of this invention is to provide a UAV visual language action navigation method and system based on sparse representation, so as to achieve more accurate and specific navigation while reducing the processing workload.

[0005] In a first aspect, the present invention provides a visual language action navigation method for unmanned aerial vehicles (UAVs) based on sparse representation, the method comprising:

[0006] Acquire images captured by the acquisition device on the drone;

[0007] The image is converted into multiple visual processing units, and the multiple visual processing units are sparsified to determine the key visual processing units among them.

[0008] The other visual processing units, excluding the key visual processing unit, are subjected to context merging processing to obtain a merged visual processing unit.

[0009] The key visual processing unit and the merged visual processing unit are combined to obtain a set of visual processing units;

[0010] Obtain the user's language instructions, and generate a predicted action sequence based on the language instructions and the set of visual processing units;

[0011] The navigation control of the UAV is performed based on the predicted action sequence.

[0012] In an optional implementation, the step of converting the image into multiple visual processing units and performing sparsification on the multiple visual processing units to determine the key visual processing units includes:

[0013] The image is processed using a pre-trained visual transformer to obtain multiple feature maps of different scales, and each feature map is converted into multiple visual processing units.

[0014] For each of the aforementioned feature maps, calculate the attention score of each visual processing unit in the feature map within the multi-head attention mechanism;

[0015] The visual processing unit with the highest attention score of a set number is identified as the key visual processing unit.

[0016] In an optional implementation, the value of the set number is determined in the following way:

[0017] The current environmental state is determined based on the collected images, and the task requirements are determined based on the acquired user's language instructions.

[0018] The set number is dynamically adjusted based on the current environmental state and the task requirements.

[0019] In an optional implementation, the step of performing context merging processing on the other visual processing units besides the key visual processing unit among the plurality of visual processing units to obtain merged visual processing units includes:

[0020] For any two other visual processing units besides the key visual processing unit among the plurality of visual processing units, calculate the similarity between the two other visual processing units;

[0021] Other visual processing units with similarity higher than a preset threshold are grouped into the same cluster;

[0022] Context merging is performed on other visual processing units in the same cluster to obtain the corresponding merged visual processing unit.

[0023] In an optional implementation, the step of generating a predicted action sequence based on the language instructions and the set of visual processing units includes:

[0024] The key regions in the image are determined according to the language instructions;

[0025] The weights of each visual processing unit in the visual processing unit set are determined based on the key regions.

[0026] Based on the language instructions and the set of visual processing units carrying weights, a predicted action sequence is generated.

[0027] In an optional implementation, the step of determining the weights of each visual processing unit in the visual processing unit set based on the key region includes:

[0028] Detect whether each visual processing unit in the visual processing unit cluster is located in the critical area;

[0029] For the visual processing units located within the critical region, a weight increment is added based on the initial weight;

[0030] For visual processing units that are not located within the critical region, their initial weights are maintained.

[0031] In an optional implementation, the step of generating a predicted action sequence based on the language instructions and the set of visual processing units includes:

[0032] The language instructions and the set of visual processing units are converted into embedding vectors respectively, and the two embedding vectors are fused to obtain a fused vector;

[0033] An initial action sequence is obtained based on the fusion vector;

[0034] The task scenario is determined based on the language instructions, and an optimized action sequence is generated based on the fusion vector and the task scenario;

[0035] The initial action sequence and the optimized action sequence are combined to generate a predicted action sequence.

[0036] In an optional implementation, the step of generating a predicted action sequence based on the language instructions and the set of visual processing units includes:

[0037] Obtain historical instruction sequences and the current environment state;

[0038] The context-enhanced representation is obtained by combining the historical instruction sequence with the current environment state;

[0039] The enhanced language instruction is obtained by combining the context-enhanced representation and the language instruction;

[0040] Based on the enhanced language instructions and the set of visual processing units, a predicted action sequence is generated.

[0041] In an optional implementation, the predicted action sequence includes a desired posture and a desired velocity;

[0042] The step of performing navigation control of the UAV based on the predicted action sequence includes:

[0043] Attitude control is performed based on the current attitude of the UAV and the desired attitude, and speed control is performed based on the current speed of the UAV and the desired speed.

[0044] During speed control, the speed is dynamically adjusted based on the detected relative positional relationship between the drone and obstacles, and the flight path is adjusted to achieve obstacle avoidance.

[0045] Secondly, the present invention provides a UAV visual language action navigation system based on sparse representation, the system comprising:

[0046] The image acquisition module is used to acquire images captured by the acquisition device on the drone;

[0047] A sparsification processing module is used to convert the image into multiple visual processing units and perform sparsification processing on the multiple visual processing units to determine the key visual processing units among them.

[0048] The merging processing module is used to perform context merging processing on the other visual processing units among the plurality of visual processing units, excluding the key visual processing unit, to obtain a merged visual processing unit.

[0049] The module is used to combine the key visual processing unit and the merged visual processing unit to obtain a set of visual processing units.

[0050] The generation module is used to acquire the user's language instructions and generate a predicted action sequence based on the language instructions and the set of visual processing units;

[0051] A navigation control module is used to perform navigation control of the UAV based on the predicted action sequence.

[0052] This invention provides a UAV visual-language-action navigation method and system based on sparse representation. The method involves converting an acquired image into multiple visual processing units (VRAMs), and then performing sparsification on these VRAMs to identify key VRAMs. The remaining VRAMs are then merged using context merging to obtain merged VRAMs. A set of VRAMs is obtained by combining the key VRAMs and the merged VRAMs. Based on the acquired language commands and the set of VRAMs, a predicted action sequence is generated, and navigation control of the UAV is executed based on this predicted action sequence. In this solution, the sparsification of the visual processing units reduces the processing load, and combining navigation with user language commands enables more precise and specific navigation. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of a UAV visual language action navigation method based on sparse representation provided in an embodiment of the present invention;

[0055] Figure 2 for Figure 1 A flowchart of the sub-steps included in S12;

[0056] Figure 3 for Figure 1 A flowchart of the sub-steps included in S13;

[0057] Figure 4 for Figure 1 One of the flowcharts for the sub-steps included in S15;

[0058] Figure 5 for Figure 1 The second flowchart of the sub-steps included in S15;

[0059] Figure 6 for Figure 1 The third flowchart of the sub-steps included in S15;

[0060] Figure 7 for Figure 1 A flowchart of the sub-steps included in S16;

[0061] Figure 8 A functional block diagram of a UAV visual language action navigation system based on sparse representation provided in an embodiment of the present invention;

[0062] Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0063] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0064] Please see Figure 1 This is a flowchart illustrating a UAV visual language action navigation method based on sparse representation provided in an embodiment of the present invention. This UAV visual language action navigation method based on sparse representation can be executed by electronic devices on the UAV or electronic devices communicating with the UAV.

[0065] It should be understood that in other embodiments, the order of some steps in the UAV visual language action navigation method based on sparse representation in this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of the UAV visual language action navigation method based on sparse representation are described below.

[0066] S11, acquire images collected by the acquisition device on the drone.

[0067] S12, the image is converted into multiple visual processing units, and the multiple visual processing units are sparsified to determine the key visual processing units among them.

[0068] S13, perform context merging processing on the other visual processing units besides the key visual processing unit among the plurality of visual processing units to obtain a merged visual processing unit.

[0069] S14, combine the key visual processing unit and the merged visual processing unit to obtain a visual processing unit set.

[0070] S15, Obtain the user's language instructions, and generate a predicted action sequence based on the language instructions and the visual processing unit set.

[0071] S16, Perform navigation control of the UAV based on the predicted action sequence.

[0072] In this embodiment, the drone is equipped with a data acquisition device, which can be a camera, a video camera, or other similar device. The data acquisition device can acquire images and transmit the acquired images to electronic devices in real time for processing.

[0073] First, images can be converted into multiple visual processing units (tokens). Visual tokens refer to the conversion of visual information (such as images and videos) into discrete, processable units or symbols in computer vision and multimodal AI systems, similar to text tokens in natural language processing.

[0074] By segmenting an image into small pieces, and then using an encoder to convert these pieces into vector representations, these vectors become visual processing units.

[0075] Since there are often a large number of visual processing units in an image, and some of these units may not contain important information, this embodiment performs sparsification on the multiple visual processing units to identify the key visual processing units and avoid excessive processing work. Key visual processing units can be understood as visual processing units in the image that provide important information for navigation control.

[0076] Furthermore, to preserve potentially important information, the remaining visual processing units can be merged. In this way, by extracting key visual processing units and merging others, the amount of data can be reduced while preserving relatively complete image information.

[0077] In this embodiment, during navigation control, user language commands, such as "fly over the red building," can also be obtained. These language commands often instruct the execution of specific flight tasks during normal flight. Therefore, a predicted action sequence is ultimately generated based on the language commands and the visual processing unit in the visual processing unit cluster, and then the navigation control of the UAV is executed according to the predicted action sequence.

[0078] The navigation method provided in this embodiment extracts key visual processing units from the visual processing units and merges the remaining visual processing units to obtain a set of visual processing units. This reduces the amount of data that needs to be processed while preserving relatively complete image information. Furthermore, by combining the obtained language commands, specific flight tasks can be executed during flight, achieving more precise and targeted navigation.

[0079] The following provides a detailed explanation of the specific implementation methods for each of the above steps.

[0080] Please see Figure 2 The steps described above, which convert an image into multiple visual processing units and then perform sparsification on these units to determine the key visual processing units, can be implemented in the following way:

[0081] S121, the image is processed using a pre-trained visual transformer to obtain multiple feature maps of different scales, and each feature map is converted into multiple visual processing units.

[0082] S122, For each of the feature maps, calculate the attention score of each of the visual processing units in the feature map in the multi-head attention mechanism.

[0083] S123, determine the visual processing unit with the highest attention score of a set number as the key visual processing unit.

[0084] In this embodiment, a visual encoder can be used to process the image. First, features are extracted from the image using a pre-trained visual encoder (ViT). Let the input image be... The feature map is obtained after passing through the visual encoder. :

[0085]

[0086] in, This is a multi-scale feature representation extracted from the input image. Furthermore, to capture dependencies between different locations, a multi-head self-attention mechanism (MHSA) is applied. Given a feature map... Calculate its attention score for each head. :

[0087]

[0088] in, and These are the results of linear transformations of the query and the key, respectively. This represents the head dimension. Then, the attention scores of all heads are averaged to obtain the final attention matrix. :

[0089]

[0090] Here, H represents the number of attention heads, and the elements in the attention matrix are the attention scores of each visual processing unit in feature maps of multiple different scales.

[0091] By combining multi-scale feature maps and attention matrices, the feature representation is further enhanced. For each scale of feature map... Combine it with the attention matrix Multiply and add to obtain the final image features:

[0092]

[0093] in, This indicates the number of scales in the feature map.

[0094] To avoid the influence of redundant information, a method for selecting key visual tokens is adopted. For each visual token... Calculate the attention score it receives. Suppose that the attention matrix obtained after the input feature map undergoes a multi-head self-attention mechanism is... The rows correspond to feature maps, and the columns correspond to visual processing units within those feature maps. The formula for calculating the attention score of the j-th visual processing unit in the i-th feature map is as follows:

[0095]

[0096] in, It is the sequence length. Represents the first element in the attention matrix. Line number The elements of the column. Based on the calculated attention scores, select the visual processing units with the highest attention scores (e.g., K) as the key visual processing units:

[0097]

[0098] in, It is a list of attention scores for all tokens. This indicates selecting the top K tokens with the highest attention scores.

[0099] To adapt to different scenarios and task requirements, the number of visual processing units selected as key visual processing units can be dynamically adjusted according to the actual situation. Specifically, the number can be determined in the following way:

[0100] The current environmental state is determined based on the acquired images, and the task requirements are determined based on the acquired user's language instructions; the value of the set number is dynamically adjusted in combination with the current environmental state and the task requirements.

[0101] In this embodiment, the current environmental state can be determined by analyzing the image. The environmental state can be characterized by information such as the number of objects in the environment and the clarity of the objects. For example, if there are many objects in the environment, it indicates that the environmental state is more complex; if the clarity of the objects in the environment is low (i.e., the flight altitude is high), it also indicates that the environmental state is more complex.

[0102] Furthermore, analyzing the user's language commands can determine the task requirements, and based on these requirements, the accuracy requirements of the task can be determined.

[0103] When adjusting the number of elements based on the current environment and task requirements, if the current environment is complex, the number of elements can be increased to retain more image details. Conversely, if the current environment is simple, the number of elements can be decreased, meaning that only a small number of key visual processing units need to be extracted to avoid a large processing workload.

[0104] In addition, if the task requires high precision, the number of parameters can be increased to retain more image details, while if the task requires low precision, the number of parameters can be decreased to avoid wasting computing resources.

[0105] The dynamic adjustment of the set number can be represented as follows:

[0106]

[0107] in, Indicates the current environmental state. This represents the dynamic threshold adjustment function. This indicates the task requirements.

[0108] After selecting the key visual processing units using the above method, in order to preserve potentially important information in the image, the remaining visual processing units can be merged to obtain merged visual processing units. For details, please refer to [link to relevant documentation]. Figure 3 This step can be achieved in the following way:

[0109] S131, For any two other visual processing units other than the key visual processing unit among the plurality of visual processing units, calculate the similarity between the two other visual processing units.

[0110] S132, group other visual processing units with similarity higher than a preset threshold into the same cluster.

[0111] S133, perform context merging processing on other visual processing units in the same cluster to obtain the corresponding merged visual processing unit.

[0112] In this embodiment, a similarity metric is used to identify visual processing units containing similar information, and these visual processing units are then merged. Assuming two visual processing units... and The vector features are respectively and Then the similarity between them It can be calculated using the following formula:

[0113]

[0114] in, Represents the vector dot product. and Let represent the norms of the vectors. Based on the similarity metric, tokens with high similarity are grouped into the same cluster, and the tokens within each cluster are merged. Let the set of tokens in a cluster be . The merged tokens are then represented as:

[0115]

[0116] in, It is a collection of tokens within a cluster. This is the result of a similarity measurement. This indicates a merge operation.

[0117] After selecting key visual processing units using the above method and merging them, these visual processing units form a visual processing unit set.

[0118] In the flight control process of drones, in addition to conventional flight control such as flying from the starting point to the destination and obstacle avoidance during the process, users can also issue voice commands based on their needs to instruct the drone to perform specific flight tasks.

[0119] Based on this, in this embodiment, a predicted action sequence is generated based on the obtained language instructions and visual processing unit set.

[0120] Because language commands often relate to specific areas in an image—for example, the command "fly over the red building"—this command relates to the area associated with the "red building" in the image. To enhance control precision during processing, it's crucial to focus on this specific area. Therefore, please refer to [link to relevant documentation]. Figure 4 In one possible implementation, the step of generating the predicted action sequence based on language instructions and a set of visual processing units can be achieved in the following way:

[0121] S151A, determine the key regions in the image according to the language instructions.

[0122] S152A, determine the weight of each visual processing unit in the visual processing unit set based on the key region.

[0123] S153A, based on the language instructions and the set of visual processing units carrying weights, a predicted action sequence is generated.

[0124] In this embodiment, key regions in an image are determined based on language commands. For example, if the language command is "fly over the red building", the image can be analyzed and processed to determine the region in the image containing "red building" as the key region.

[0125] As described above, each visual processing unit is obtained by segmenting the image, and each visual processing unit corresponds to a certain region in the image. Therefore, the weights of each visual processing unit can be determined based on the identified key regions, and subsequently, a predicted action can be generated based on the language instructions and the visual processing units carrying the weights.

[0126] Specifically, determining the weights of each visual processing unit in the visual processing unit set based on key regions can be achieved in the following way:

[0127] The system detects whether each visual processing unit in the visual processing unit cluster is located in the key region; for visual processing units located in the key region, a weight increment is added to their initial weight; for visual processing units not located in the key region, their initial weight is maintained.

[0128] In order to enhance the focus on key areas during subsequent processing and thus complete specific flight missions, the weight of visual processing units located in key areas can be increased, while the weight of visual processing units not located in key areas can remain unchanged from the initial weight, for example, the initial weight can be 1.

[0129] The final weights of each visual processing unit can be represented as follows:

[0130]

[0131] in, Indicates the key area. This represents the weight increment. The above means that if the visual processing unit is located within a critical region, the weight increment is added to the initial weight; otherwise, the initial weight remains unchanged.

[0132] This embodiment considers a specific flight mission based on user requirements. To enable the model to better understand this specific flight mission, an optimization layer is added to parse the flight mission, thereby ensuring its successful completion. Based on this, please refer to... Figure 5 In one possible implementation, the steps described above for generating predicted action sequences based on language instructions and a set of visual processing units can be achieved in the following way:

[0133] S151B, the language instructions and the visual processing unit set are converted into embedding vectors respectively, and the two embedding vectors are fused to obtain a fused vector.

[0134] S152B, an initial action sequence is obtained based on the fusion vector.

[0135] S153B, determine the task scenario based on the language instructions, and generate an optimized action sequence based on the fusion vector and the task scenario.

[0136] S154B, combining the initial action sequence and the optimized action sequence, a predicted action sequence is generated.

[0137] In this embodiment, the language decoder receives a sparsed set of visual processing units and the user's language commands, converts them into vector form through an embedding layer, and utilizes the Transformer architecture for feature fusion and understanding. To ensure that the model can accurately understand and respond to diverse flight requests, a custom command tuning layer is introduced.

[0138] First, the input set of visual processing units and the user's language commands are converted into vector representations respectively. Let the set of visual processing units after sparsification be... The user's language commands are The resulting embedding vector is represented as follows:

[0139]

[0140] in, Represents the embedding vector of the visual processing unit. This represents the embedding vector of the language instruction. The embedding vectors of the visual processing unit and the language instruction are then input into the Transformer decoder. Feature fusion is performed through multi-layer self-attention and cross-attention mechanisms to obtain a fused vector, represented as follows:

[0141]

[0142] in, The fusion vector represents the hidden state output by the decoder. This represents the Transformer decoder. It further outputs the initial action sequence through a fully connected layer, represented as follows:

[0143]

[0144] in, Indicates the initial action sequence. This indicates a fully connected layer.

[0145] To enhance the language decoder's comprehension capabilities, an additional instruction tuning layer is introduced during training to fine-tune the code for specific task scenarios. For example, when a user issues the instruction "fly over the red building," the tuning layer helps the model better understand the concept of "red building" and generate a corresponding tuned action sequence.

[0146]

[0147] in, It is the sequence of tuning actions output by the instruction tuning layer. This indicates the instruction tuning layer function. This represents language instructions. Specifically, the instruction tuning layer function first uses a linear transformation on the embedding vectors of visual tokens. and embedding vectors of language instructions The processing is as follows:

[0148]

[0149] in, , , These represent the weight matrices used to generate the query, key, and value, respectively. , K h , These are the query matrix, key matrix, and value matrix (these matrices are determined during pre-training of the model under different task scenarios). The attention score is calculated based on this matrix.

[0150]

[0151] Based on the calculated attention score, the user's verbal commands can be incorporated into the hidden state. Specifically, this is achieved by combining a weighted sum with a value matrix. and the original hidden state The optimized action sequence is obtained as follows:

[0152]

[0153] Finally, the initial action sequence and the optimized action sequence can be combined to obtain the predicted action sequence.

[0154] To enhance the language decoder's understanding capabilities, a context-enhancing mechanism can be introduced. For further information, please refer to [link / reference needed]. Figure 6 When generating predicted action sequences based on language instructions and a set of visual processing units, one possible implementation is as follows:

[0155] S151C obtains the historical instruction sequence and the current environment status.

[0156] S152C, combining the historical instruction sequence and the current environment state to obtain a context-enhanced representation.

[0157] S153C, combining the context-enhanced representation and the language instruction to obtain the enhanced language instruction.

[0158] S154C, based on the enhanced language instructions and the set of visual processing units, generates a predicted action sequence.

[0159] In this embodiment, the understanding of user language commands is enhanced by combining historical instruction sequences and the current environment state. Assume the current environment state is... The historical instruction sequence is The context-enhanced representation can then be as follows:

[0160]

[0161] in, It is a context-enhanced representation. This represents the context enhancement function. Specifically, the context enhancement function first constructs an embedded sequence of historical instructions, as shown below:

[0162]

[0163] in, It is a historical instruction sequence. It is its set of embedding vectors. The context attention score is calculated based on this:

[0164]

[0165] Then, by combining the current environment state with the historical instruction sequence, the enhanced language instructions can be obtained:

[0166]

[0167] In this way, predicted action sequences can be generated based on the enhanced language instructions and the set of visual processing units.

[0168] To adapt to different task requirements and environmental changes, language decoders need to have the ability to dynamically adjust. Specifically, the decoder parameters can be dynamically adjusted according to the current environmental state and task requirements. For example, in complex environments, the number of attention heads can be increased to improve model performance; when computational resources are limited, the number of attention heads can be reduced to lower computational costs.

[0169]

[0170] in, This represents a dynamically adjusted function. Indicates the current environmental state. This indicates the task requirements.

[0171] To improve system robustness, an error detection and correction mechanism can be added to the language decoder. Specifically, when the action sequence generated by the decoder contains unreasonable elements, the system will automatically detect and correct these errors. Assume the generated action sequence is... The error detection result is The corrected formula is as follows:

[0172]

[0173] in, It is the corrected sequence of actions. This represents the error correction function.

[0174] The final predicted action sequence is obtained through the above method. The predicted action sequence includes at least the desired attitude and desired velocity, which will be used for subsequent navigation and control of the UAV.

[0175] Based on this, please refer to Figure 7 The steps for performing navigation control of a UAV based on predicted action sequences can be achieved in the following ways:

[0176] S161, attitude control is performed based on the current attitude of the UAV and the desired attitude, and speed control is performed based on the current speed of the UAV and the desired speed.

[0177] S162, during the speed control process, dynamically adjusts the speed based on the detected relative positional relationship between the UAV and obstacles, and adjusts the flight path to achieve obstacle avoidance.

[0178] In this embodiment, a motion actuator is used to perform navigation control. The motion actuator is one of the core components of the entire system, responsible for adjusting the UAV's attitude and speed in real time based on the predicted motion sequence generated by the language decoder, ensuring that it reaches its destination safely and efficiently along the predetermined route. To achieve this goal, the motion actuator needs to have high flexibility and precision to cope with various complex flight environments.

[0179] Attitude control is one of the important functions of motion actuators, mainly including the adjustment of pitch, roll, and yaw angles. These parameters directly affect the flight direction and stability of the UAV.

[0180] Specifically, the drone's forward or backward tilt angle is adjusted to change its forward direction. Let the current pitch angle be... The desired pitch angle is The adjustment formula is as follows:

[0181]

[0182] Roll angle control is used for lateral movement, allowing the drone to tilt left and right. Assume the current roll angle is... The expected roll angle is The adjustment formula is as follows:

[0183]

[0184] Yaw angle control is used to change the drone's orientation. Assume the current yaw angle is... The desired yaw angle is The adjustment formula is as follows:

[0185]

[0186] Combining the adjustments from the three angles mentioned above, the total attitude adjustment can be obtained:

[0187]

[0188] The attitude control of the UAV is performed based on the total attitude adjustment.

[0189] Besides attitude control, speed regulation is also a key factor in ensuring stable drone flight. Depending on different mission requirements and environmental changes, the actuators need to dynamically adjust the drone's speed.

[0190] Specifically, in most cases, the drone maintains a constant speed during flight. Let the desired speed be... The current speed is The adjustment formula is as follows:

[0191]

[0192] When encountering obstacles or other emergencies, it may be necessary to temporarily adjust speed. For example, slowing down as you approach an obstacle and resuming your original speed after moving away from it. Let the acceleration be... The time interval is The new speed calculation formula is as follows:

[0193]

[0194] In addition, intelligent speed planning can be considered, which is based on environmental perception and path planning. The motion actuator can adaptively adjust its speed according to the terrain and obstacle distribution ahead. That is, it uses a visual encoder to obtain information in the field of vision ahead and predicts the optimal speed based on the perceived information.

[0195] Obstacle avoidance mechanisms are crucial for ensuring the safe flight of drones. In this embodiment, the motion actuator integrates advanced obstacle avoidance algorithms, enabling the drone to adjust its flight path promptly when encountering obstacles to avoid collisions. Specific strategies include:

[0196] Distance Detection and Early Warning: Utilizing sensors on the drone to monitor the surrounding environment in real time, assuming the distance to an obstacle ahead is... The set security threshold is The obstacle avoidance judgment formula is as follows:

[0197]

[0198] Path replanning: Once an obstacle is detected ahead, the action actuator immediately initiates a path replanning algorithm to find a new feasible path. Let the current path be... The new path is The path update formula is as follows:

[0199]

[0200] Dynamic obstacle avoidance strategy: In complex environments, obstacles may be dynamically changing. Therefore, the motion actuator employs a dynamic obstacle avoidance strategy, continuously adjusting its flight trajectory based on real-time environmental information. Let the current environmental state be... Based on the following dynamic obstacle avoidance strategy, a new flight trajectory is determined:

[0201]

[0202] in, This represents the dynamic obstacle avoidance function.

[0203] The UAV visual language navigation method based on sparse representation provided in this embodiment filters visual tokens using a sparse processor, retaining the most representative parts, which significantly reduces computational resource consumption. This achieves a significant reduction in memory usage and computation time while minimizing redundant information injection under the same visual token limit, thus improving performance.

[0204] Furthermore, by utilizing sparsity representation technology, drones can flexibly respond to various challenges in complex and ever-changing environments. Through intelligent filtering of visual tokens, the system can focus on the most critical information, thereby improving its sensitivity to environmental changes and response speed.

[0205] Furthermore, this solution integrates advanced obstacle avoidance algorithms, enabling the UAV to adjust its flight path promptly when encountering obstacles to avoid collisions. Dynamic obstacle avoidance strategies and local area enhancement mechanisms further enhance the system's robustness and safety.

[0206] Based on the same inventive concept, please refer to Figure 8 This invention also provides a functional module diagram of a UAV visual language action navigation system based on sparse representation. This embodiment can divide the UAV visual language action navigation system based on sparse representation into functional modules according to the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0207] For example, when dividing functional modules according to their respective functions, Figure 8 The UAV visual language action navigation system shown is only a schematic diagram of a device. This system may include an image acquisition module, a sparsity processing module, a merging processing module, a combination module, a generation module, and a navigation control module. The functions of each module in this sparse representation-based UAV visual language action navigation system will be described in detail below.

[0208] The image acquisition module is used to acquire images captured by the acquisition device on the drone;

[0209] A sparsification processing module is used to convert the image into multiple visual processing units and perform sparsification processing on the multiple visual processing units to determine the key visual processing units among them.

[0210] The merging processing module is used to perform context merging processing on the other visual processing units among the plurality of visual processing units, excluding the key visual processing unit, to obtain a merged visual processing unit.

[0211] The module is used to combine the key visual processing unit and the merged visual processing unit to obtain a set of visual processing units.

[0212] The generation module is used to acquire the user's language instructions and generate a predicted action sequence based on the language instructions and the set of visual processing units;

[0213] A navigation control module is used to perform navigation control of the UAV based on the predicted action sequence.

[0214] The UAV visual language action navigation system based on sparse representation provided in this embodiment can be used to execute the UAV visual language action navigation method based on sparse representation under any of the above embodiments. For details not covered in this embodiment, please refer to the corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0215] Please see Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be a control device on a drone, or a control device communicating with a drone, etc. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0216] The memory is used to store computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0217] The processor is used to read / write data or programs stored in the memory and execute the UAV visual language action navigation method based on sparse representation provided in any embodiment of the present invention.

[0218] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0219] It should be understood that, Figure 9 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0220] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the UAV visual language action navigation method based on sparse representation provided in the above embodiments.

[0221] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is executed, it can perform the aforementioned UAV visual language action navigation method based on sparse representation. The processes involved in the execution of the executable instructions on the computer-readable storage medium can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0222] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0223] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0225] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0226] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual-language-action navigation method for unmanned aerial vehicles (UAVs) based on sparse representation, characterized in that, The method includes: Acquire images captured by the acquisition device on the drone; The image is converted into multiple visual processing units, and the multiple visual processing units are sparsified to determine the key visual processing units among them. The other visual processing units, excluding the key visual processing unit, are subjected to context merging processing to obtain a merged visual processing unit. The key visual processing unit and the merged visual processing unit are combined to obtain a set of visual processing units; Obtain the user's language instructions, and generate a predicted action sequence based on the language instructions and the set of visual processing units; The navigation control of the UAV is executed based on the predicted action sequence; The step of converting the image into multiple visual processing units and performing sparsification on the multiple visual processing units to determine the key visual processing units includes: The image is processed using a pre-trained visual transformer to obtain multiple feature maps of different scales, and each feature map is converted into multiple visual processing units. For each feature map, the attention score of each visual processing unit in the feature map is calculated in the multi-head attention mechanism. The visual processing unit with the highest attention score of a set number is determined as the key visual processing unit. The step of performing context merging processing on the other visual processing units besides the key visual processing unit among the plurality of visual processing units to obtain merged visual processing units includes: For any two other visual processing units besides the key visual processing unit among the plurality of visual processing units, calculate the similarity between the two other visual processing units; group other visual processing units with similarity higher than a preset threshold into the same cluster; perform context merging processing on other visual processing units in the same cluster to obtain the corresponding merged visual processing unit.

2. The UAV visual language action navigation method based on sparse representation according to claim 1, characterized in that, The set number is determined in the following way: The current environmental state is determined based on the collected images, and the task requirements are determined based on the acquired user's language instructions. The set number is dynamically adjusted based on the current environmental state and the task requirements.

3. The UAV visual language action navigation method based on sparse representation according to claim 1, characterized in that, The step of generating a predicted action sequence based on the language instructions and the set of visual processing units includes: The key regions in the image are determined according to the language instructions; The weights of each visual processing unit in the visual processing unit set are determined based on the key regions. Based on the language instructions and the set of visual processing units carrying weights, a predicted action sequence is generated.

4. The UAV visual language action navigation method based on sparse representation according to claim 3, characterized in that, The step of determining the weights of each visual processing unit in the visual processing unit set based on the key region includes: Detect whether each visual processing unit in the visual processing unit cluster is located in the critical area; For the visual processing units located within the critical region, a weight increment is added based on the initial weight; For visual processing units that are not located within the critical region, their initial weights are maintained.

5. The UAV visual language action navigation method based on sparse representation according to claim 1, characterized in that, The step of generating a predicted action sequence based on the language instructions and the set of visual processing units includes: The language instructions and the set of visual processing units are converted into embedding vectors respectively, and the two embedding vectors are fused to obtain a fused vector; An initial action sequence is obtained based on the fusion vector; The task scenario is determined based on the language instructions, and an optimized action sequence is generated based on the fusion vector and the task scenario; The initial action sequence and the optimized action sequence are combined to generate a predicted action sequence.

6. The UAV visual language action navigation method based on sparse representation according to claim 1, characterized in that, The step of generating a predicted action sequence based on the language instructions and the set of visual processing units includes: Obtain historical instruction sequences and the current environment state; The context-enhanced representation is obtained by combining the historical instruction sequence with the current environment state; The enhanced language instruction is obtained by combining the context-enhanced representation and the language instruction; Based on the enhanced language instructions and the set of visual processing units, a predicted action sequence is generated.

7. The UAV visual language action navigation method based on sparse representation according to claim 1, characterized in that, The predicted action sequence includes the desired posture and the desired velocity; The step of performing navigation control of the UAV based on the predicted action sequence includes: Attitude control is performed based on the current attitude of the UAV and the desired attitude, and speed control is performed based on the current speed of the UAV and the desired speed. During speed control, the speed is dynamically adjusted based on the detected relative positional relationship between the drone and obstacles, and the flight path is adjusted to achieve obstacle avoidance.

8. A UAV visual-language-action navigation system based on sparse representation, characterized in that, The system is used to implement the UAV visual language action navigation method based on sparse representation as described in any one of claims 1-7, the system comprising: The image acquisition module is used to acquire images captured by the acquisition device on the drone; A sparsification processing module is used to convert the image into multiple visual processing units and perform sparsification processing on the multiple visual processing units to determine the key visual processing units among them. The merging processing module is used to perform context merging processing on the other visual processing units among the plurality of visual processing units, excluding the key visual processing unit, to obtain a merged visual processing unit. The module is used to combine the key visual processing unit and the merged visual processing unit to obtain a set of visual processing units. The generation module is used to acquire the user's language instructions and generate a predicted action sequence based on the language instructions and the set of visual processing units; A navigation control module is used to perform navigation control of the UAV based on the predicted action sequence.

Images