Flight control method and device of unmanned aerial vehicle, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a flight control method and device of an unmanned aerial vehicle, electronic equipment and a storage medium, and belongs to the technical field of visual language navigation processing. The method comprises the following steps: inputting a visual image and a corresponding instruction text at each moment into a visual language sub-model of a pre-trained flight prediction model, and outputting navigation intention information corresponding to an unmanned aerial vehicle at each moment; acquiring current state information and current action information of the unmanned aerial vehicle at the current moment, and taking the navigation intention information at the current moment or the navigation intention information at the moment closest to the current moment as target navigation intention information of the unmanned aerial vehicle; and inputting the current state information, the current action information and the target navigation intention information into an action prediction sub-model of a pre-trained flight prediction model, and outputting information for guiding the unmanned aerial vehicle to adjust subsequent predicted actions. The flight continuity of the unmanned aerial vehicle can be improved, so that the flight stability of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] This application relates to the field of visual language navigation processing technology, specifically to a flight control method, device, electronic device, and storage medium for an unmanned aerial vehicle (UAV). Background Technology

[0002] Currently, Vision-and-Language Navigation for Unmanned Aerial Vehicles (VLN) has become a popular research area. UAVs require onboard models to predict flight maneuvers in real time to ensure smooth flight. In related technologies, UAVs process real-time visual information, control command text, and flight data during flight to predict their next flight maneuver.

[0003] However, because the visual information from drones requires complex image processing, real-time reasoning is difficult to achieve in some complex scenarios, resulting in delays in drone flight maneuvers. This manifests as the drone having to stop and wait before predicting its next flight action, and the resulting flight delays increase the risk of instability and crashes. In other words, the poor flight continuity of drones in related technologies leads to a decrease in drone flight stability. Summary of the Invention

[0004] This application provides a flight control method, device, electronic device, and storage medium for a drone, which can improve the flight continuity of the drone and thus improve its flight stability.

[0005] To achieve the above objectives, one embodiment of this application provides a flight control method for an unmanned aerial vehicle (UAV), comprising: Acquire visual images captured by the drone at multiple times and generate command text corresponding to the visual images at each time point; The visual image and corresponding command text at each time step are input into the visual language sub-model of the pre-trained flight prediction model, and the navigation intent information of the UAV at each time step is output. Obtain the current state and action information of the drone at the current moment, and use the navigation intention information at the current moment or the navigation intention information of the moment closest to the current moment as the target navigation intention information of the drone; The current state information, current action information, and target navigation intent information are input into the action prediction sub-model of the pre-trained flight prediction model, and the predicted action information corresponding to the UAV is output. The predicted action information is used to guide the UAV to adjust its subsequent flight actions. Among them, the visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image and sample command text corresponding to the UAV at multiple sample times into the flight prediction model for prediction.

[0006] In some embodiments, the navigation intent information at the current moment or the navigation intent information at the moment closest to the current moment is used as the target navigation intent information of the UAV, including: If the navigation intent information output by the visual language sub-model of the pre-trained flight prediction model includes the navigation intent information at the current moment, determine the first image sharpness value of the visual image corresponding to the current moment, and determine the second image sharpness value of the visual image corresponding to the moment closest to the current moment. When the first image sharpness value is equal to or greater than the second image sharpness value, the navigation intent information at the current moment is used as the target navigation intent information of the UAV. When the first image sharpness value is less than the second image sharpness value, or when the navigation intent information output by the visual language sub-model of the pre-trained flight prediction model does not include the navigation intent information at the current moment, the navigation intent information at the moment closest to the current moment will be used as the target navigation intent information of the UAV.

[0007] In some embodiments, after outputting the predicted motion information corresponding to the drone, the method further includes: Obtain the instruction action reference table; The flight command is obtained by parsing the predicted action information. The corresponding flight action is determined from the command action lookup table based on the flight command, and the drone is controlled to fly according to the flight action.

[0008] In some embodiments, before inputting the visual image and corresponding instruction text at each time step into the visual language sub-model of the pre-trained flight prediction model, the method further includes: Acquire sample visual images collected by the drone at multiple sample times, and generate sample command text corresponding to the sample visual image at each sample time; The visual image and corresponding command text of the sample at each sample time are input into the visual language sub-model of the flight prediction model, and the sample navigation intent information of the UAV at each sample time is output. Acquire the sample state information and sample action information of the UAV at the target sample time, and take the sample navigation intention information at the target sample time or the sample navigation intention information at the closest sample time to the target sample time as the sample target navigation intention information of the UAV. The sample state information, sample action information, and sample target navigation intent information are input into the action prediction sub-model of the flight prediction model, and the corresponding sample prediction action information of the UAV is output. Obtain the label action information corresponding to the sample predicted action information, calculate the action difference between the sample predicted action information and the label action information, update the flight prediction model based on the action difference, and obtain the updated flight prediction model. When the action difference does not meet the preset action difference condition, return to the step of obtaining the sample state information and sample action information of the UAV at the target sample time, until the action difference meets the preset action difference condition, and obtain the pre-trained flight prediction model.

[0009] In some embodiments, the flight prediction model is updated based on the action differences to obtain an updated flight prediction model, including: Obtain the drone's tag navigation intent information; Calculate the difference in navigation intent between sample target navigation intent information and corresponding label navigation intent information; When the navigation intent difference does not meet the preset navigation intent conditions, the visual language sub-model of the flight prediction model is updated according to the navigation intent difference, and the action prediction sub-model of the flight prediction model is updated according to the action difference, so as to obtain the updated flight prediction model. When the difference in navigation intent meets the preset navigation intent conditions, the action prediction sub-model of the flight prediction model is updated according to the difference in action to obtain the updated flight prediction model.

[0010] In some embodiments, the process includes obtaining labeled action information corresponding to the sample predicted action information, calculating the action difference between the sample predicted action information and the labeled action information, updating the flight prediction model based on the action difference, and obtaining an updated flight prediction model, including: Obtain prediction action information for multiple samples and the corresponding label action information for each sample's prediction action information; Calculate the action difference between the predicted action information and the corresponding labeled action information for each sample, and update the flight prediction model based on the action difference to obtain the updated flight prediction model.

[0011] In some embodiments, the action difference between the predicted action information and the corresponding labeled action information for each sample is calculated, and the flight prediction model is updated based on the action difference to obtain the updated flight prediction model, including: Obtain the desired termination time and preset base; For each sample predicted action information, the time difference between the expected termination time and the sample time corresponding to the sample predicted action information is calculated, and the time difference is used as the exponent of the preset base at the corresponding sample time. Based on the preset base and the index corresponding to the preset base at different sample times, the time weight corresponding to the predicted action information of each sample is determined; Based on the time weight corresponding to the predicted action information of each sample, the action difference between the predicted action information of each sample and the corresponding labeled action information is calculated. The flight prediction model is then updated based on the action difference to obtain the updated flight prediction model.

[0012] To achieve the above objectives, one embodiment of this application provides a flight control device for an unmanned aerial vehicle (UAV), comprising: The first acquisition module is used to acquire visual images collected by the UAV at multiple times and generate instruction text corresponding to the visual images at each time. The navigation intent information determination module is used to input the visual image and corresponding command text at each time moment into the visual language sub-model of the pre-trained flight prediction model, and output the navigation intent information of the UAV at each time moment. The second acquisition module is used to acquire the current state information and current action information of the UAV at the current moment, and to use the navigation intention information at the current moment or the navigation intention information of the moment closest to the current moment as the target navigation intention information of the UAV. The target prediction module is used to input current state information, current action information, and target navigation intent information into the action prediction sub-model of the pre-trained flight prediction model, and output the predicted action information corresponding to the UAV. The predicted action information is used to guide the UAV to adjust subsequent flight actions. Among them, the visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image, and sample command text corresponding to the sample visual image of the UAV at multiple sample times into the flight prediction model for prediction.

[0013] To achieve the above objectives, one aspect of this application provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the steps in the flight control method for a drone provided in this application.

[0014] To achieve the above objectives, one aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the steps in the flight control method for a drone provided in this application.

[0015] To achieve the above objectives, one aspect of this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the flight control method for a drone provided in this application.

[0016] The UAV flight control method, device, electronic equipment, and storage medium proposed in this application acquire visual images collected by the UAV at multiple times and generate command text corresponding to the visual images at each time. The visual images and corresponding command text at each time are input into the visual language sub-model of a pre-trained flight prediction model, outputting navigation intent information corresponding to the UAV at each time. The current state information and current action information of the UAV at the current time are acquired, and the navigation intent information at the current time or the navigation intent information at the closest time to the current time is used as the target navigation intent information of the UAV. The current state information, current action information, and target navigation intent information are input into the action prediction sub-model of the pre-trained flight prediction model, outputting predicted action information corresponding to the UAV. This predicted action information is used to guide the UAV to adjust subsequent flight actions. The pre-trained flight prediction model is trained based on the difference between predicted sample action information and labeled action information. The sample predicted action information is obtained by inputting sample state information, sample action information, sample visual images, and sample command text corresponding to the sample visual images at multiple sample times into the flight prediction model.

[0017] This application separates the visual language sub-model for processing visual images and corresponding command text from the action prediction sub-model for processing the current state and action information output by the UAV. This allows the action prediction sub-model for simple data to avoid flight discontinuity at each data processing moment due to waiting for navigation intent information at that moment. Instead, it allows the visual language sub-model for complex visual information to dynamically select, based on the actual situation, whether to acquire the current navigation intent information or the navigation intent information of the closest moment to the current moment as the target navigation intent information. This ensures that, while acquiring the target navigation intent information in real time, the predicted action information for adjusting subsequent flight can be continuously predicted and output at each moment. Thus, this application improves the flight continuity of the UAV, thereby enhancing its flight stability.

[0018] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the system framework corresponding to the flight control method for the UAV provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the flight control method for an unmanned aerial vehicle (UAV) provided in an embodiment of this application. Figure 3 This is a schematic diagram of the data processing flow of the UAV flight control method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the module structure of the flight control device for an unmanned aerial vehicle provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that in all specific embodiments of this application, when it is necessary to acquire visual images collected by the drone, permission or consent from the relevant personnel managing the drone is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to acquire sensitive personal information of relevant personnel, separate permission or consent from the relevant personnel is obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the separate permission or consent of the relevant personnel is the necessary visual image for the normal operation of this application embodiment acquired. Other data acquired in this application embodiment are all authorized and legal data, and will not be elaborated upon here.

[0023] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer computer devices, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0024] The technical problems existing in the related technologies are as follows: Currently, VLN has become a popular research field, as drones require onboard models to predict flight maneuvers in real time to ensure smooth flight. In related technologies, drones process real-time visual information, control command text, and flight data during flight to predict their flight maneuvers in the next moment.

[0025] However, because the visual information from drones requires complex image processing, real-time reasoning is difficult to achieve in some complex scenarios, resulting in delays in drone flight maneuvers. This manifests as the drone having to stop and wait before predicting its next flight action, and the resulting flight delays increase the risk of instability and crashes. In other words, the poor flight continuity of drones in related technologies leads to a decrease in drone flight stability.

[0026] For example, users can utilize drones equipped with VLN systems to perform tasks such as autonomous indoor and outdoor inspections, logistics delivery, or emergency search and rescue. The flight control system needs to dynamically plan flight paths based on real-time environmental images and natural language commands (such as "fly to the building with the red roof" or "turn left after bypassing the obstacle ahead"). However, because VLN models typically rely on complex multimodal fusion networks to perform joint inference on high-resolution images and text, on computationally limited airborne platforms, especially in complex scenarios like dense urban areas, they struggle to complete action prediction within milliseconds. This results in frequent drone hovering while waiting for inference results. Such flight interruptions caused by inference delays not only disrupt mission continuity but can also induce attitude instability or even collisions and crashes during high-speed flight or in confined spaces.

[0027] The flight control method, device, electronic equipment, and storage medium for unmanned aerial vehicles (UAVs) proposed in this application acquire visual images collected by the UAV at multiple moments and generate command text corresponding to the visual images at each moment; input the visual images and corresponding command text at each moment into the visual language sub-model of a pre-trained flight prediction model, and output the navigation intent information of the UAV at each moment; acquire the current state information and current action information of the UAV at the current moment, and use the navigation intent information at the current moment or the navigation intent information at the moment closest to the current moment as the target navigation intent information of the UAV; and combine the current state information, current action information, and target navigation intent information... The flight intention information is input into the action prediction sub-model of the pre-trained flight prediction model, which outputs the predicted action information of the UAV. The predicted action information is used to guide the UAV to adjust its subsequent flight actions. The visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image and sample command text corresponding to the sample visual image of the UAV at multiple sample times into the flight prediction model for prediction.

[0028] This application separates the visual language sub-model for processing visual images and corresponding command text from the action prediction sub-model for processing the current state and action information output by the UAV. This allows the action prediction sub-model for simple data to avoid flight discontinuity at each data processing moment due to waiting for navigation intent information at that moment. Instead, it allows the visual language sub-model for complex visual information to dynamically select, based on the actual situation, whether to acquire the current navigation intent information or the navigation intent information of the closest moment to the current moment as the target navigation intent information. This ensures that, while acquiring the target navigation intent information in real time, the predicted action information for adjusting subsequent flight can be continuously predicted and output at each moment. Thus, this application improves the flight continuity of the UAV, thereby enhancing its flight stability.

[0029] The specific details regarding the flight control method, apparatus, electronic equipment, and storage medium for the unmanned aerial vehicle provided in the embodiments of this application will be described in detail below.

[0030] Please see Figure 1 , Figure 1 This is a schematic diagram of the system framework corresponding to the UAV flight control method provided in this application embodiment. The UAV flight control method provided in this application embodiment can be applied to this system framework.

[0031] It includes terminal 140, Internet 130, gateway 120, server 110, etc.

[0032] Terminal 140 or server 110 may be a device that performs flight control methods for the drone.

[0033] Terminal 140 includes, but is not limited to, mobile phones, tablets, computers, and intelligent computing centers. Terminal 140 can be a single device or a collection of multiple devices. For example, multiple computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, thus forming a terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0034] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0035] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0036] The embodiments of this application can be applied to various scenarios, such as drone intelligent logistics delivery, drone power line inspection, and drone emergency search and rescue. This is only an example and does not mean that the embodiments of this application limit the scenarios in which the drone flight control method is applied.

[0037] Next, we will describe it from the perspective of the drone's flight control device, such as... Figure 2 As shown, Figure 2 This is a schematic flowchart of a flight control method for a drone provided in an embodiment of this application. The flight control method for a drone is applied to a flight control device for a drone. Figure 2 The method may include, but is not limited to, the following steps 210 to 240. When the UAV's flight control device executes the UAV's flight control method, the specific process is as follows. It should be noted that this embodiment... Figure 2 The order of steps 210 to 240 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0038] Step 210: Acquire visual images captured by the drone at multiple times and generate instruction text corresponding to the visual images at each time. Step 220: Input the visual image and corresponding command text at each time moment into the visual language sub-model of the pre-trained flight prediction model, and output the navigation intent information of the UAV at each time moment; Step 230: Obtain the current state information and current action information of the UAV at the current moment, and use the navigation intention information at the current moment or the navigation intention information of the moment closest to the current moment as the target navigation intention information of the UAV. Step 240: Input the current state information, current action information and target navigation intent information into the action prediction sub-model of the pre-trained flight prediction model, and output the predicted action information corresponding to the UAV. The predicted action information is used to guide the UAV to adjust subsequent flight actions.

[0039] Steps 210 to 240 are described in detail below.

[0040] In step 210, visual images collected by the UAV at multiple times are acquired, and instruction text corresponding to the visual image at each time time is generated.

[0041] Visual images refer to environmental image data collected by a drone at multiple moments during its flight using onboard cameras or other visual sensors. Visual images can be acquired in real-time by the drone or input into the drone's visual image processor from external devices.

[0042] In this context, command text refers to the natural language description or navigation command corresponding to each visual image. It expresses the expected operation or objective that the drone should perform at the given moment corresponding to the visual image. Command text can be manually input, output by a language model trained on pre-defined rules and manually labeled data, or inferred from the content of the visual image using other models. The role of command text is to transform visual perception into understandable and actionable semantic instructions. For example, command text could be "Obstacle ahead, please turn right and detour" or "Target building is located to the left front, maintain close proximity."

[0043] Furthermore, after acquiring the input data, the flight prediction model first performs data preprocessing, converting the collected data into a standardized format that the model can read: the input to the flight prediction model is dictionary data, namely Input_dict:{['image_tensor':,'state.pos':,'state.yaw':,'action':,'task_description':]}, where image_tensor is the visual image, state.pos and state.yaw are the current state information, action is the current action information, and task_description is the instruction text.

[0044] In step 220, the visual image and corresponding command text at each time point are input into the visual language sub-model of the pre-trained flight prediction model, and the navigation intent information of the UAV at each time point is output.

[0045] Among them, the visual language sub-model of the pre-trained flight prediction model and the action prediction sub-model of the pre-trained flight pre-model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image and sample command text corresponding to the UAV at multiple sample times into the flight prediction model for prediction.

[0046] In some embodiments, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the data processing flow of the UAV flight control method provided in the embodiments of this application. The pre-trained flight prediction model includes two sub-models, namely the visual language sub-model and the action prediction sub-model. The visual language sub-model further includes a visual encoder, a text tokenizer, and a visual language module (VLM).

[0047] Furthermore, after inputting the visual image and corresponding command text at each time step into the visual language sub-model of the pre-trained flight prediction model, the visual encoder first performs visual encoding processing on each visual image to obtain visual encoding information; and the text encoder performs text encoding processing on the corresponding command text of each visual image to obtain text encoding information; then, the visual encoding information and the corresponding text encoding information at each time step are input into the VLM, and the VLM outputs the navigation intent information corresponding to each time step.

[0048] In this embodiment, a multilingual visual-language encoder (SigLIP-2) is used as the Vision Encoder; the VLM module uses a pre-trained model (Large Language Model Meta AI, version 2, with 7 billion parameters, Llama-27B); and the Text Tokenizer uses the tokenizer corresponding to the VLM model. This is merely an example; specific settings can be implemented according to actual circumstances, and this embodiment does not impose any limitations.

[0049] Navigation intent information refers to the high-level navigation target or behavioral guidance that the UAV understands based on its perceived visual environment and corresponding natural language commands at a specific moment. It instructs the UAV "which direction and how it should move," thus fulfilling the semantic intent of the mission. Navigation intent information is a key intermediate representation connecting environmental perception and action decision-making, used to guide subsequent action prediction sub-models in generating predicted flight actions that conform to the mission objectives.

[0050] Furthermore, such as Figure 3 As shown, the visual language sub-model processes the visual image and corresponding instruction text at each time step, and outputs the navigation intent information obtained at each time step to the storage module (Visual feature) of the action prediction sub-model for storage, so that the action prediction sub-model can determine the target navigation intent information from the storage module according to actual needs.

[0051] It should be noted that the tasks handled by VLM involve deep semantic understanding and cross-modal alignment of high-dimensional visual images and natural language instructions. The computational complexity is high, resulting in a large inference delay. Therefore, it is a slow module in flight prediction models.

[0052] In step 230, the current state information and current action information of the UAV at the current moment are obtained, and the navigation intention information at the current moment or the navigation intention information of the moment closest to the current moment is used as the target navigation intention information of the UAV.

[0053] The current moment refers to the point in time when the UAV is making flight decisions or control actions. At this point in time, the UAV needs to integrate the latest current status information, current action information, and navigation intent information to generate the next stage of flight actions to ensure the continuity of the UAV's flight.

[0054] The current state information includes the UAV's current position (pos) and yaw information. pos represents the UAV's coordinates in space, usually expressed as a three-dimensional vector (x, y, z), typically in meters (m). yaw refers to the angle of rotation of the UAV around its vertical axis (usually the z-axis), indicating the horizontal direction the UAV's nose is facing, typically in radians (rad) or degrees (°). For example, if the motion prediction sub-model obtains current state information of pos = (0, 0, 2) and yaw = 45°, this means the UAV is currently 2 meters directly above the origin, with its nose facing northeast (45° east of north).

[0055] The current action information refers to the flight action performed by the drone at the current moment. The flight action includes the action state and the action value. For example, the current action information can be: fly forward 3 meters, turn left 30 degrees, turn right 30 degrees, and ascend vertically 2 meters.

[0056] It should be noted that the data input of the action prediction sub-model is low-dimensional variable information, so its processing output frequency is relatively high, making it a fast module.

[0057] Furthermore, the visual-language sub-model and the action prediction sub-model process input data in parallel within the flight prediction model, simultaneously processing visual-language input and state-action input to efficiently generate navigation intentions and predict flight actions. Compared to the sequential operation mode in traditional methods, the embodiments of this application significantly improve the overall response speed and computational efficiency of the flight prediction model, avoiding the latency caused by sequential processing. This enables the UAV to adjust its flight actions more quickly, thereby enhancing its real-time decision-making and adaptability in dynamic environments.

[0058] Furthermore, if the visual language sub-model has completed processing the information at the current moment and output the navigation intent information at the current moment, the action prediction sub-module will use the navigation intent information at the current moment as the target navigation intent information to calculate subsequent flight actions. Conversely, if the visual language sub-model has not yet completed processing due to computational delay, the action prediction sub-module will not wait, but will directly use the navigation intent information at the moment closest to the current moment (i.e., the navigation intent information generated at the previous moment) as the target navigation intent information to ensure that flight actions can be output without interruption at multiple moments, thereby ensuring the flight continuity of the UAV.

[0059] In some embodiments, the navigation intent information at the current moment or the navigation intent information at the moment closest to the current moment is used as the target navigation intent information of the UAV, including: (1.1) If the navigation intent information output by the visual language sub-model of the pre-trained flight prediction model includes the navigation intent information at the current moment, determine the first image sharpness value of the visual image corresponding to the current moment, and determine the second image sharpness value of the visual image corresponding to the moment closest to the current moment. (1.2) When the first image sharpness value is equal to or greater than the second image sharpness value, the navigation intent information at the current moment shall be used as the target navigation intent information of the UAV; (1.3) When the first image sharpness value is less than the second image sharpness value, or when the navigation intent information output by the visual language sub-model of the pre-trained flight prediction model does not include the navigation intent information at the current moment, the navigation intent information at the moment closest to the current moment shall be used as the target navigation intent information of the UAV.

[0060] In some embodiments, after the visual language sub-model has successfully output the navigation intent information for the current moment, an additional pre-decision step is introduced. Specifically, the flight prediction model actively quantifies and evaluates the quality of the visual information input to the visual language sub-model, that is, it determines the first image sharpness value of the visual image corresponding to the current moment, and simultaneously determines the second image sharpness value of the visual image corresponding to the nearest moment to the current moment. The image sharpness value can be calculated using an image quality assessment algorithm, such as the Laplacian operator, gradient function, etc., which can be set according to the actual situation. Generally, a higher image sharpness value usually indicates that the corresponding visual image is sharper and more information-rich, while a lower value may indicate that the visual image is blurry, noisy, or distorted.

[0061] Furthermore, the first image sharpness value and the second image sharpness value are compared. If the first image sharpness value is equal to or greater than the second image sharpness value, the quality of the visual image acquired at the current moment is determined to be better than that at the previous moment, and the navigation intent information generated based on the visual image at the current moment is considered reliable. In this case, the navigation intent information at the current moment is determined as the target navigation intent information of the UAV and is passed to the action prediction sub-model to generate subsequent flight actions.

[0062] Furthermore, when the first image clarity value is less than the second image clarity value, it indicates that although the visual language sub-model outputs the navigation intent information corresponding to the current moment in a timely manner, the corresponding visual image is unclear. For example, if the drone needs to traverse a jungle, and at the current moment, the drone's camera is obscured by a large number of leaves, then the available information in the corresponding visual image is limited, and the reliability of the navigation intent information generated using this visual image is low. In this case, to avoid making incorrect flight decisions based on low-quality perception information, the current moment's navigation intent information is abandoned, and instead, the navigation intent information of the moment closest to the current moment is used as the drone's target navigation intent information. Alternatively, when the visual language sub-model fails to output the current moment's navigation intent information in a timely manner, the navigation intent information of the moment closest to the current moment is also used as the drone's target navigation intent information.

[0063] It is understandable that in real-world flight environments, drones inevitably encounter instantaneous image quality degradation due to their own rapid maneuvers or changes in the external environment. The embodiments of this application avoid the motion prediction sub-model relying on unreliable perception data for inference, effectively enhancing the drone's flight stability and mission success rate in complex and changing environments, thereby making the drone more adaptable to sensor noise and interference in the real world.

[0064] In step 240, the current state information, current action information, and target navigation intent information are input into the action prediction sub-model of the pre-trained flight prediction model, and the predicted action information corresponding to the UAV is output. The predicted action information is used to guide the UAV to adjust subsequent flight actions.

[0065] In some embodiments, such as Figure 3 As shown, the flight prediction model's action prediction sub-model further includes a state encoder, an action encoder, and action prediction modules (DiffusionTransformer, DiT Blocks). In this embodiment, different multilayer perceptrons (MLPs) are used as the state encoder and action encoder, respectively. This is only an example; the specific configuration can be adjusted according to actual conditions.

[0066] Furthermore, the action prediction sub-model corresponds to Figure 3 Subsystem 2, the core of which is the DiT module, has relatively low computational cost and fast inference speed. The action prediction sub-model inputs current state information, current action information, and target navigation intent information into the pre-trained flight prediction model's action prediction sub-model, outputting the predicted action information for the UAV. This predicted action information is then encoded by the discrete encoding module (Action Decoder) to obtain data recognizable by the UAV processing module. The predicted action information is similar to the current action information, but it is used to guide the UAV's flight at the next moment or in the next time period, and its duration of action differs from that of the current action information.

[0067] In some embodiments, after outputting the predicted motion information corresponding to the drone, the method further includes: (2.1) Obtain the instruction action reference table; (2.2) Analyze the predicted action information to obtain flight commands, determine the corresponding flight actions from the command action lookup table according to the flight commands, and control the UAV to fly according to the flight actions.

[0068] The instruction-action lookup table is a predefined data structure that establishes a deterministic mapping between abstract instructions and specific, executable UAV physical actions. First, the predicted action information is parsed to obtain flight instructions. For example, flight instructions with values ​​[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] correspond to their respective flight actions: 0 represents a stop instruction, 1 represents flying forward 3 meters, 2 represents turning left 30 degrees, 3 represents turning right 30 degrees, 4 represents ascending vertically 2 meters, 5 represents descending vertically 2 meters, 6 represents turning left 5 degrees, 7 represents turning right 5 degrees, 8 represents flying forward 6 meters, and 9 represents flying forward 9 meters. This is just an example; the specific settings can be adjusted according to actual conditions.

[0069] Furthermore, the drone's flight is controlled based on the corresponding mapped flight maneuvers. Thus, by introducing a command-action lookup table, the training objective of the flight prediction model is simplified and standardized. It only needs to focus on outputting correct discrete flight commands, without needing to concern itself with the complex details of how those commands are physically executed, thereby significantly reducing the complexity of model design. Simultaneously, this standardized interface makes system maintenance and upgrades exceptionally flexible: when flight maneuvers need adjustment, only the corresponding entries in the command-action lookup table need to be modified, without retraining the massive flight prediction model. This significantly improves development efficiency, reduces maintenance costs, and enhances the adaptability of the flight prediction model to different drone platforms or specific mission requirements.

[0070] In some embodiments, before inputting the visual image and corresponding instruction text at each time step into the visual language sub-model of the pre-trained flight prediction model, the method further includes: (3.1) Acquire sample visual images collected by the UAV at multiple sample times, and generate sample command text corresponding to the sample visual image at each sample time; (3.2) Input the visual image of the sample at each sample time and the corresponding sample command text into the visual language sub-model of the flight prediction model, and output the sample navigation intent information of the UAV at each sample time. (3.3) Obtain the sample state information and sample action information of the UAV at the target sample time, and take the sample navigation intention information at the target sample time or the sample navigation intention information of the sample time closest to the target sample time as the sample target navigation intention information of the UAV. (3.4) Input the sample state information, sample action information and sample target navigation intent information into the action prediction sub-model of the flight prediction model, and output the sample prediction action information corresponding to the UAV; (3.5) Obtain the label action information corresponding to the sample predicted action information, calculate the action difference between the sample predicted action information and the label action information, update the flight prediction model according to the action difference, and obtain the updated flight prediction model. (3.6) When the action difference does not meet the preset action difference condition, return to the step of obtaining the sample state information and sample action information of the UAV at the target sample time, until the action difference meets the preset action difference condition, and obtain the pre-trained flight prediction model.

[0071] In some embodiments, the flight prediction model needs to be trained before it is formally applied. Specifically, sample visual images collected by the UAV at multiple sample times are acquired, and sample command text corresponding to the sample visual image at each sample time is generated. The sample visual images and sample command text are consistent in data type with the visual images and command text used during inference. In this embodiment, 30,000 flight path data collected in AirSim are used as the training set.

[0072] Furthermore, the visual image and corresponding command text for each sample time step are input into the visual language sub-model of the initial flight prediction model, outputting the sample navigation intent information of the UAV for each sample time step. The definition of the sample navigation intent information is similar to that of the navigation intent information.

[0073] Furthermore, during the training of the flight prediction model, the sample state information and sample action information corresponding to the target sample time are acquired, and the sample navigation intent information at the target sample time or the sample navigation intent information at the closest sample time to the target sample time is selected as the UAV's sample target navigation intent information. The target sample time corresponds to the current time in the inference process.

[0074] Furthermore, similar to the inference process, the acquired sample state information, sample action information, and determined sample target navigation intent information are input together into the action prediction sub-model of the flight prediction model, thereby outputting the sample predicted action information corresponding to the UAV. The sample predicted action information is similar in definition to the predicted action information; it represents the prediction result made by the model in the current training state for a given sample input. It is used to compare with the actual predicted flight results to achieve the purpose of training the model.

[0075] Furthermore, labeled action information is acquired, which represents the action that the UAV is expected to perform at the target sample time. The action difference between the sample predicted action information and the labeled action information is calculated, and the flight prediction model is updated based on the action difference to obtain the updated flight prediction model for the current round. The action difference is a loss value that quantifies the degree of prediction error, which can be calculated using the mean squared error function, cross-entropy loss function, or other loss functions. This application embodiment does not limit the method for quantifying the action difference.

[0076] Furthermore, it is determined whether the updated flight prediction model in the current round meets the preset action difference condition. If it does, the updated flight prediction model in the current round is used as the pre-trained flight prediction model. If it does not meet the condition, the process returns to the step of obtaining the sample state information and sample action information of the UAV at the target sample time, and iteratively trains the model.

[0077] It is understood that the embodiments of this application ensure the robustness and stability of the flight prediction model in a real deployment environment by training the information synchronization processing capability of the fast and slow modules. This makes the design of the entire dual-system visual language navigation scheme form a closed loop in theory and practice, and has extremely high practical value.

[0078] In some embodiments, the flight prediction model is updated based on the action differences to obtain an updated flight prediction model, including: (4.1) Obtain the UAV's tag navigation intent information; (4.2) Calculate the difference in navigation intent between the target navigation intent information and the corresponding label navigation intent information of the sample; (4.3) When the difference in navigation intent does not meet the preset navigation intent conditions, the visual language sub-model of the flight prediction model is updated according to the difference in navigation intent, and the action prediction sub-model of the flight prediction model is updated according to the difference in action, so as to obtain the updated flight prediction model. (4.4) When the difference in navigation intent meets the preset navigation intent conditions, the action prediction sub-model of the flight prediction model is updated according to the difference in action to obtain the updated flight prediction model.

[0079] In some embodiments, the update of the flight prediction model can be further divided into the update of the visual language sub-model and the update of the action prediction sub-model. The label navigation intent information corresponding to the target navigation intent information is obtained, and the difference in navigation intent between the two is quantified.

[0080] Furthermore, when the difference in navigation intent does not meet the preset navigation intent conditions (i.e., the difference is large), it is determined that the output quality of the slow module needs to be improved. At this time, the device will perform synchronous update operations on the two modules: update the visual language sub-model of the flight prediction model according to the difference in navigation intent to directly optimize its semantic understanding and intent generation capabilities, and update the action prediction sub-model of the flight prediction model according to the difference in action to obtain the updated flight prediction model.

[0081] Furthermore, when the difference in navigation intent meets the preset navigation intent conditions (i.e., the difference is small), it is determined that the slow module has been able to stably output high-quality navigation intent. In this case, the device no longer updates the visual language sub-model, but freezes the parameters of the visual language sub-model and updates the action prediction sub-model of the flight prediction model only based on the action difference. This concentrates the training resources on optimizing the fast module, enabling it to better learn how to convert accurate navigation intent into precise flight actions, thus obtaining the updated flight prediction model.

[0082] It is understood that in the early stages of training, the two sub-models are jointly optimized to improve overall performance; while once the visual language sub-model has been trained effectively, its parameters can be frozen, and only the action prediction sub-model is trained. This strategy not only greatly improves the flexibility and efficiency of the training process and reduces the demand for computing resources, but also effectively avoids knowledge forgetting or performance degradation of the visual language sub-model during training, ultimately resulting in a more robust and efficient pre-trained flight prediction model.

[0083] In some embodiments, the process includes obtaining labeled action information corresponding to the sample predicted action information, calculating the action difference between the sample predicted action information and the labeled action information, updating the flight prediction model based on the action difference, and obtaining an updated flight prediction model, including: (5.1) Obtain the predicted action information of multiple samples and the corresponding label action information of each sample's predicted action information; (5.2) Calculate the action difference between the predicted action information and the corresponding label action information for each sample, and update the flight prediction model based on the action difference to obtain the updated flight prediction model.

[0084] In some embodiments, the label action information corresponding to the sample predicted action information can be obtained each time the flight prediction model outputs sample predicted action information, so that the flight prediction model can be updated based on the action difference after each calculation, and the updated flight prediction model can be obtained.

[0085] In some embodiments, after the flight prediction model continuously outputs multiple sample predicted action information, it collects multiple sample predicted action information in batches to form a training batch. Then, it uniformly calculates the action differences corresponding to the predicted action information of all samples in the batch and performs a joint update of the model parameters. This batch processing method helps to improve training stability, reduce gradient noise, and make full use of the hardware's parallel computing capabilities, thereby optimizing the overall performance of the flight prediction model more efficiently.

[0086] In some embodiments, the action difference between the predicted action information and the corresponding labeled action information for each sample is calculated, and the flight prediction model is updated based on the action difference to obtain the updated flight prediction model, including: (6.1) Obtain the desired termination time and preset base; (6.2) For each sample predicted action information, calculate the time difference between the expected termination time and the sample time corresponding to the sample predicted action information, and use the time difference as the exponent of the preset base at the corresponding sample time; (6.3) Determine the time weight corresponding to the predicted action information of each sample based on the preset base and the index corresponding to the preset base at different sample times; (6.4) Calculate the action difference between the predicted action information and the corresponding label action information of each sample according to the time weight of the predicted action information of each sample, and update the flight prediction model according to the action difference to obtain the updated flight prediction model.

[0087] In some embodiments, the action difference is determined by the following formula: .

[0088] in, , A; This indicates the sample's predicted action information. Indicates tag action information; A mask representing the validity of time steps, used to support training of variable-length trajectories; These are the weighting coefficients for the corresponding time steps; This indicates the batch size, which is the number of trajectory samples processed in parallel during one forward propagation. This represents the time step length of action prediction (Action Horizon), which corresponds to the length of the future action sequence predicted by the model in a single inference or training iteration. This represents the action dimension at a single time step. An exponential time weighting strategy is used during training. , Indicates the expected termination time. This represents the preset base; as the time step progresses, the loss weights gradually increase, thereby guiding the model to pay more attention to the accurate modeling of later actions.

[0089] It is understood that the embodiments of this application introduce an exponential time weighting mechanism, enabling the model to learn the differences in importance of different stages of the task during the training phase. Specifically, by assigning higher time weights to actions approaching the expected termination time, the embodiments of this application make the model focus more on optimizing the action prediction accuracy in the latter half of the flight path during training. This effectively shifts learning resources towards the key stages of the task, thereby significantly improving the navigation accuracy and success rate of the UAV at the mission endpoint, making the trained flight prediction model more goal-oriented and practical.

[0090] like Figure 4 As shown, Figure 4 This is a schematic diagram of the module structure of a flight control device for a drone provided in an embodiment of this application. The flight control device 300 for the drone may include the following modules 310 to 340: The first acquisition module 310 is used to acquire visual images collected by the UAV at multiple times and generate instruction text corresponding to the visual images at each time. The navigation intent information determination module 320 is used to input the visual image and corresponding command text at each time moment into the visual language sub-model of the pre-trained flight prediction model, and output the navigation intent information of the UAV at each time moment. The second acquisition module 330 is used to acquire the current state information and current action information of the UAV at the current moment, and use the navigation intention information at the current moment or the navigation intention information of the moment closest to the current moment as the target navigation intention information of the UAV. The target prediction module 340 is used to input the current state information, current action information, and target navigation intent information into the action prediction sub-model of the pre-trained flight prediction model, and output the predicted action information corresponding to the UAV. The predicted action information is used to guide the UAV to adjust subsequent flight actions. The visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image, and sample command text corresponding to the sample visual image of the UAV at multiple sample times into the flight prediction model for prediction.

[0091] In some embodiments, the second acquisition module 330 is used for: If the navigation intent information output by the visual language sub-model of the pre-trained flight prediction model includes the navigation intent information at the current moment, determine the first image sharpness value of the visual image corresponding to the current moment, and determine the second image sharpness value of the visual image corresponding to the moment closest to the current moment. When the first image sharpness value is equal to or greater than the second image sharpness value, the navigation intent information at the current moment is used as the target navigation intent information of the UAV. When the first image sharpness value is less than the second image sharpness value, or when the navigation intent information output by the visual language sub-model of the pre-trained flight prediction model does not include the navigation intent information at the current moment, the navigation intent information at the moment closest to the current moment will be used as the target navigation intent information of the UAV.

[0092] In some embodiments, the target prediction module 340 is used for: Obtain the instruction action reference table; The flight command is obtained by parsing the predicted action information. The corresponding flight action is determined from the command action lookup table based on the flight command, and the drone is controlled to fly according to the flight action.

[0093] In some embodiments, the navigation intent information determination module 320 is used for: Acquire sample visual images collected by the drone at multiple sample times, and generate sample command text corresponding to the sample visual image at each sample time; The visual image and corresponding command text of the sample at each sample time are input into the visual language sub-model of the flight prediction model, and the sample navigation intent information of the UAV at each sample time is output. Acquire the sample state information and sample action information of the UAV at the target sample time, and take the sample navigation intention information at the target sample time or the sample navigation intention information at the closest sample time to the target sample time as the sample target navigation intention information of the UAV. The sample state information, sample action information, and sample target navigation intent information are input into the action prediction sub-model of the flight prediction model, and the corresponding sample prediction action information of the UAV is output. Obtain the label action information corresponding to the sample predicted action information, calculate the action difference between the sample predicted action information and the label action information, update the flight prediction model based on the action difference, and obtain the updated flight prediction model. When the action difference does not meet the preset action difference condition, return to the step of obtaining the sample state information and sample action information of the UAV at the target sample time, until the action difference meets the preset action difference condition, and obtain the pre-trained flight prediction model.

[0094] In some embodiments, the navigation intent information determination module 320 is further configured to: Obtain the drone's tag navigation intent information; Calculate the difference in navigation intent between sample target navigation intent information and corresponding label navigation intent information; When the navigation intent difference does not meet the preset navigation intent conditions, the visual language sub-model of the flight prediction model is updated according to the navigation intent difference, and the action prediction sub-model of the flight prediction model is updated according to the action difference, so as to obtain the updated flight prediction model. When the difference in navigation intent meets the preset navigation intent conditions, the action prediction sub-model of the flight prediction model is updated according to the difference in action to obtain the updated flight prediction model.

[0095] In some embodiments, the navigation intent information determination module 320 is used for: Obtain prediction action information for multiple samples and the corresponding label action information for each sample's prediction action information; Calculate the action difference between the predicted action information and the corresponding labeled action information for each sample, and update the flight prediction model based on the action difference to obtain the updated flight prediction model.

[0096] In some embodiments, the navigation intent information determination module 320 is used for: Obtain the desired termination time and preset base; For each sample predicted action information, the time difference between the expected termination time and the sample time corresponding to the sample predicted action information is calculated, and the time difference is used as the exponent of the preset base at the corresponding sample time. Based on the preset base and the index corresponding to the preset base at different sample times, the time weight corresponding to the predicted action information of each sample is determined; Based on the time weight corresponding to the predicted action information of each sample, the action difference between the predicted action information of each sample and the corresponding labeled action information is calculated. The flight prediction model is then updated based on the action difference to obtain the updated flight prediction model.

[0097] The flight control method, device, electronic equipment, and storage medium for unmanned aerial vehicles (UAVs) proposed in this application acquire visual images collected by the UAV at multiple moments and generate command text corresponding to the visual images at each moment; input the visual images and corresponding command text at each moment into the visual language sub-model of a pre-trained flight prediction model, and output the navigation intent information of the UAV at each moment; acquire the current state information and current action information of the UAV at the current moment, and use the navigation intent information at the current moment or the navigation intent information at the moment closest to the current moment as the target navigation intent information of the UAV; and combine the current state information, current action information, and target navigation intent information... The flight intention information is input into the action prediction sub-model of the pre-trained flight prediction model, which outputs the predicted action information of the UAV. The predicted action information is used to guide the UAV to adjust its subsequent flight actions. The visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image and sample command text corresponding to the sample visual image of the UAV at multiple sample times into the flight prediction model for prediction.

[0098] This application separates the visual language sub-model for processing visual images and corresponding command text from the action prediction sub-model for processing the current state and action information output by the UAV. This allows the action prediction sub-model for simple data to avoid flight discontinuity at each data processing moment due to waiting for navigation intent information at that moment. Instead, it allows the visual language sub-model for complex visual information to dynamically select, based on the actual situation, whether to acquire the current navigation intent information or the navigation intent information of the closest moment to the current moment as the target navigation intent information. This ensures that, while acquiring the target navigation intent information in real time, the predicted action information for adjusting subsequent flight can be continuously predicted and output at each moment. Thus, this application improves the flight continuity of the UAV, thereby enhancing its flight stability.

[0099] like Figure 5 As shown, Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes: The processor 401 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called by the processor 401 to execute the flight control method of the UAV according to the embodiments of this application. Input / output interface 403 is used to implement information input and output; The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404); The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0100] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the flight control method for the aforementioned unmanned aerial vehicle.

[0101] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0102] The flight control method, device, electronic equipment, and storage medium for unmanned aerial vehicles (UAVs) proposed in this application acquire visual images collected by the UAV at multiple moments and generate command text corresponding to the visual images at each moment; input the visual images and corresponding command text at each moment into the visual language sub-model of a pre-trained flight prediction model, and output the navigation intent information of the UAV at each moment; acquire the current state information and current action information of the UAV at the current moment, and use the navigation intent information at the current moment or the navigation intent information at the moment closest to the current moment as the target navigation intent information of the UAV; and combine the current state information, current action information, and target navigation intent information... The flight intention information is input into the action prediction sub-model of the pre-trained flight prediction model, which outputs the predicted action information of the UAV. The predicted action information is used to guide the UAV to adjust its subsequent flight actions. The visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image and sample command text corresponding to the sample visual image of the UAV at multiple sample times into the flight prediction model for prediction.

[0103] This application separates the visual language sub-model for processing visual images and corresponding command text from the action prediction sub-model for processing the current state and action information output by the UAV. This allows the action prediction sub-model for simple data to avoid flight discontinuity at each data processing moment due to waiting for navigation intent information at that moment. Instead, it allows the visual language sub-model for complex visual information to dynamically select, based on the actual situation, whether to acquire the current navigation intent information or the navigation intent information of the closest moment to the current moment as the target navigation intent information. This ensures that, while acquiring the target navigation intent information in real time, the predicted action information for adjusting subsequent flight can be continuously predicted and output at each moment. Thus, this application improves the flight continuity of the UAV, thereby enhancing its flight stability.

[0104] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0105] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0108] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0109] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0111] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A flight control method for an unmanned aerial vehicle (UAV), characterized in that, include: The system acquires visual images captured by the drone at multiple times and generates instruction text corresponding to the visual images at each time point. The visual image and the corresponding instruction text at each time moment are input into the visual language sub-model of the pre-trained flight prediction model, and the navigation intent information of the UAV at each time moment is output. Obtain the current state information and current action information of the UAV at the current moment, and use the navigation intention information at the current moment or the navigation intention information of the moment closest to the current moment as the target navigation intention information of the UAV; The current state information, the current action information, and the target navigation intent information are input into the action prediction sub-model of the pre-trained flight prediction model, and the predicted action information corresponding to the UAV is output. The predicted action information is used to guide the UAV to adjust its subsequent flight actions. The visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image and the sample command text corresponding to the UAV at multiple sample times into the flight prediction model for prediction.

2. The flight control method for an unmanned aerial vehicle according to claim 1, characterized in that, The step of using the navigation intent information at the current moment or the navigation intent information at the moment closest to the current moment as the target navigation intent information of the UAV includes: If the navigation intent information output by the visual language sub-model of the pre-trained flight prediction model includes the navigation intent information at the current moment, determine the first image sharpness value of the visual image corresponding to the current moment, and determine the second image sharpness value of the visual image corresponding to the moment closest to the current moment. When the first image clarity value is equal to or greater than the second image clarity value, the navigation intent information at the current moment is used as the target navigation intent information of the UAV; When the first image sharpness value is less than the second image sharpness value, or when the navigation intent information at the current moment is not included in the multiple navigation intent information output by the visual language sub-model of the pre-trained flight prediction model, the navigation intent information at the moment closest to the current moment will be used as the target navigation intent information of the UAV.

3. The flight control method for an unmanned aerial vehicle according to claim 1, characterized in that, After outputting the predicted action information corresponding to the drone, the method further includes: Obtain the instruction action reference table; The predicted action information is analyzed to obtain flight commands. The corresponding flight actions are determined from the command action lookup table based on the flight commands, and the UAV is controlled to fly according to the flight actions.

4. The flight control method for an unmanned aerial vehicle according to claim 1, characterized in that, Before inputting the visual image and corresponding instruction text at each time moment into the visual language sub-model of the pre-trained flight prediction model, the method further includes: Acquire sample visual images collected by the UAV at multiple sample times, and generate sample command text corresponding to the sample visual image at each sample time; The visual image of the sample at each sample time and the corresponding sample command text are input into the visual language sub-model of the flight prediction model, and the sample navigation intent information of the UAV at each sample time is output. The sample state information and sample action information of the UAV at the target sample time are obtained, and the sample navigation intention information at the target sample time or the sample navigation intention information of the sample time closest to the target sample time is used as the sample target navigation intention information of the UAV. The sample state information, the sample action information, and the sample target navigation intent information are input into the action prediction sub-model of the flight prediction model, and the sample prediction action information corresponding to the UAV is output. Obtain the label action information corresponding to the sample predicted action information, calculate the action difference between the sample predicted action information and the label action information, update the flight prediction model according to the action difference, and obtain the updated flight prediction model; When the action difference does not meet the preset action difference condition, return to the step of obtaining the sample state information and sample action information of the UAV at the target sample time, until the action difference meets the preset action difference condition, and obtain the pre-trained flight prediction model.

5. The flight control method for an unmanned aerial vehicle according to claim 4, characterized in that, The step of updating the flight prediction model based on the action differences to obtain the updated flight prediction model includes: Obtain the tag navigation intent information of the drone; Calculate the navigation intent difference between the sample target navigation intent information and the corresponding label navigation intent information; When the navigation intent difference does not meet the preset navigation intent condition, the visual language sub-model of the flight prediction model is updated according to the navigation intent difference, and the action prediction sub-model of the flight prediction model is updated according to the action difference, so as to obtain the updated flight prediction model. When the difference in navigation intent meets the preset navigation intent conditions, the action prediction sub-model of the flight prediction model is updated according to the difference in action to obtain the updated flight prediction model.

6. The flight control method for an unmanned aerial vehicle according to claim 4, characterized in that, The steps include: obtaining the label action information corresponding to the sample predicted action information, calculating the action difference between the sample predicted action information and the label action information, and updating the flight prediction model based on the action difference to obtain the updated flight prediction model. Acquire multiple sample prediction action information and the label action information corresponding to each sample prediction action information; Calculate the action difference between the predicted action information of each sample and the corresponding label action information, and update the flight prediction model based on the action difference to obtain the updated flight prediction model.

7. The flight control method for an unmanned aerial vehicle according to claim 6, characterized in that, The step of calculating the action difference between the predicted action information of each sample and the corresponding label action information, and updating the flight prediction model based on the action difference to obtain the updated flight prediction model includes: Obtain the desired termination time and preset base; For each sample predicted action information, the time difference between the expected termination time and the sample time corresponding to the sample predicted action information is calculated, and the time difference is used as the exponent of the preset base at the corresponding sample time. Based on the preset base and the index corresponding to the preset base at different sample times, the time weight corresponding to the predicted action information of each sample is determined; Based on the time weight corresponding to the predicted action information of each sample, the action difference between the predicted action information of each sample and the corresponding label action information is calculated, and the flight prediction model is updated based on the action difference to obtain the updated flight prediction model.

8. A flight control device for an unmanned aerial vehicle (UAV), characterized in that, include: The first acquisition module is used to acquire visual images collected by the UAV at multiple times and generate instruction text corresponding to the visual image at each time. The navigation intent information determination module is used to input the visual image and the corresponding instruction text at each time moment into the visual language sub-model of the pre-trained flight prediction model, and output the navigation intent information of the UAV at each time moment; The second acquisition module is used to acquire the current state information and current action information of the UAV at the current moment, and to use the navigation intention information at the current moment or the navigation intention information at the moment closest to the current moment as the target navigation intention information of the UAV. The target prediction module is used to input the current state information, the current action information, and the target navigation intent information into the action prediction sub-model of the pre-trained flight prediction model, and output the predicted action information corresponding to the UAV. The predicted action information is used to guide the UAV to adjust subsequent flight actions. The visual language sub-model and the action prediction sub-model of the pre-trained flight prediction model process the input data in parallel. The pre-trained flight prediction model is trained based on the difference between the predicted sample action information and the labeled action information. The sample predicted action information is obtained by inputting the sample state information, sample action information, sample visual image, and sample command text corresponding to the sample visual image of the UAV at multiple sample times into the flight prediction model for prediction.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the flight control method of the UAV according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the flight control method of the UAV according to any one of claims 1 to 7.

Images