Unmanned aerial vehicle endurance intelligent optimization method and system based on multi-modal data fusion

Abstract

The invention discloses an unmanned aerial vehicle endurance intelligent optimization method and system based on multi-modal data fusion, belongs to the technical field of unmanned aerial vehicles, generates low-energy-consumption candidate instructions by fusing multi-modal data, intelligently screens an optimal scheme, effectively reduces energy consumption on the premise of keeping a flight state, and improves the unmanned aerial vehicle endurance intelligent optimization efficiency. The technical effects of improving the energy utilization efficiency and prolonging the effective endurance time of the unmanned aerial vehicle are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a UAV endurance intelligent optimization method and system based on multi-modal data fusion. BACKGROUND

[0002] The UAV technology is widely used in aerial photography, surveying and mapping, logistics and other fields, and its endurance capability directly affects the task execution efficiency. Currently, the UAV mainly improves the endurance time by optimizing the battery capacity, reducing the weight of the body or improving the power system, etc. These hardware level improvements have limitations such as high cost and obvious technical bottleneck.

[0003] At the software control level, the existing solutions focus on single-dimensional optimization strategies. For example, by presetting the flight path to reduce the flight distance, or dynamically adjusting the flight height according to the environmental wind speed. Some systems will trigger the return mechanism based on real-time power information, but such strategies usually rely on single sensor data (such as remaining power or positioning information), and do not fully consider the synergistic relationship between flight state, environmental variables and task goals.

[0004] Further, the traditional control instruction generation mechanism has an optimization blind area. After the user sends the control instruction, the UAV can only passively execute the preset action process. Even if there is another low-energy instruction path that can achieve the same flight state, the system lacks real-time fusion analysis capability of multi-modal data (such as environmental airflow, body attitude, and task intention), and cannot generate equivalent alternative instructions, resulting in that the energy utilization efficiency cannot be maximized. SUMMARY

[0005] The embodiments of the present application provide a UAV endurance intelligent optimization method and system based on multi-modal data fusion, which can improve the energy utilization efficiency, prolong the effective endurance time of the UAV while maintaining the flight state, and the technical solutions are as follows: On the one hand, a UAV endurance intelligent optimization method based on multi-modal data fusion is provided, and the method comprises: In the case that the UAV is in an endurance optimization mode to automatically execute a target task, in response to an initial control instruction of the UAV sent by a user terminal, the instruction intention of the initial control instruction, the state information of the UAV, the environmental information of the environment where the UAV is located, and the task information of the target task are obtained, the initial control instruction is used to control the UAV to change the state, the user terminal is in communication connection with a control terminal, and the control terminal is in communication connection with the UAV; based on the initial control instruction, the state information and the environmental information, a plurality of candidate control instructions are determined, the state of the UAV after executing the candidate control instruction and the initial control instruction is the same, and the energy consumed by the UAV executing the candidate control instruction is less than that of the initial control instruction; based on the instruction intention and the task information, a target control instruction of the UAV is determined from the plurality of candidate control instructions; and the UAV is controlled to execute the target control instruction.

[0006] Further, the application also proposes that, based on the initial control instruction, the state information and the environment information, the multiple candidate control instructions are determined, including: based on the initial control instruction, the state information and the environment information, the predicted state of the UAV and the predicted energy consumption corresponding to the initial control instruction are determined; based on the state information and the predicted state, the multiple candidate state transformation modes of the UAV transforming to the predicted state are determined; based on the state information, the environment information and the multiple candidate state transformation modes, the energy consumption corresponding to each candidate state transformation mode is determined; based on the energy consumption corresponding to each candidate state transformation mode, the multiple reference state transformation modes are determined from the multiple candidate state transformation modes, and the multiple reference state transformation modes are the candidate state transformation modes with the corresponding energy consumption less than the predicted energy consumption; based on the multiple reference state transformation modes, the multiple candidate control instructions are generated.

[0007] Further, the application also proposes that the state information includes the initial pose, the initial motion information and the UAV information, based on the initial control instruction, the state information and the environment information, the predicted state of the UAV and the predicted energy consumption corresponding to the initial control instruction are determined, including: based on the initial pose, the initial motion information and the initial control instruction, the predicted state of the UAV is determined, and the initial motion information includes the initial speed, the initial motion direction and the initial acceleration; based on the initial control instruction, the UAV information, the predicted energy consumption corresponding to the initial control instruction is determined.

[0008] Further, the application also proposes that, based on the initial pose, the initial motion information and the initial control instruction, the predicted state of the UAV is determined, including: based on the initial pose, the initial motion information and the initial control instruction, the predicted pose and the predicted motion information of the UAV are determined; based on the predicted pose and the predicted motion information, the predicted state is determined; the UAV information includes the power component information, the battery state information and the UAV weight, based on the initial control instruction, the UAV information, the predicted energy consumption corresponding to the initial control instruction is determined, including: based on the initial control instruction, the power component information and the UAV weight, the state transformation demand power of the UAV is determined; based on the state transformation demand power and the battery state information, the predicted energy consumption corresponding to the initial control instruction is determined.

[0009] Further, the application further proposes that, based on the initial control instruction, the power component information and the weight of the unmanned aerial vehicle, the state transformation required power of the unmanned aerial vehicle is determined, including: determining the pose transformation mode and the motion transformation mode indicated by the initial control instruction; based on the pose transformation mode, the motion transformation mode and the weight of the unmanned aerial vehicle, the state transformation basic power of the unmanned aerial vehicle is determined; based on the power component information, the state transformation basic power is corrected to obtain the state transformation required power of the unmanned aerial vehicle; the battery state information includes the residual capacity, the battery temperature and the battery internal resistance, based on the state transformation required power and the battery state information, the predicted energy consumption corresponding to the initial control instruction is determined, including: based on the residual capacity, the battery temperature and the battery internal resistance, the current battery state is determined; based on the current battery state and the state transformation required power, the predicted energy consumption corresponding to the initial control instruction is determined.

[0010] Further, the application further proposes that, the state information includes the initial pose, the initial motion information and the unmanned aerial vehicle information, based on the state information and the predicted state, the multiple candidate state transformation modes of the unmanned aerial vehicle transforming to the predicted state are determined, including: based on the initial pose and the initial motion information, the initial state of the unmanned aerial vehicle is determined; based on the unmanned aerial vehicle information, the initial state and the state transformation planning of the predicted state, the multiple candidate state transformation modes are obtained, the unmanned aerial vehicle information includes the power component information and the weight of the unmanned aerial vehicle, and the multiple candidate state transformation modes are the state transformation modes that can be realized by the unmanned aerial vehicle.

[0011] Further, the application further proposes that, based on the state information, the environment information and the multiple candidate state transformation modes, the energy consumption corresponding to each candidate state transformation mode is determined, including: based on the state information and the multiple candidate state transformation modes, the initial energy consumption corresponding to each candidate state transformation mode is determined; based on the environment information and the multiple candidate state transformation modes, the environment energy correction coefficient corresponding to each candidate state transformation mode is determined; the initial energy consumption corresponding to each candidate state transformation mode is multiplied by the corresponding environment energy correction coefficient to obtain the energy consumption corresponding to each candidate state transformation mode.

[0012] Further, the application also proposes that, based on the state information and the plurality of candidate state transformation modes, the initial energy consumption corresponding to each candidate state transformation mode is determined, including: based on the initial pose, the initial motion information and the plurality of candidate state transformation modes, the pose change set and the motion information change set corresponding to each candidate state transformation mode are determined, the pose change set includes a plurality of poses when the corresponding candidate state transformation mode is used for state transformation, and the motion information change set includes a plurality of motion information when the corresponding candidate state transformation mode is used for state transformation; based on the pose change set and the motion information change set corresponding to each candidate state transformation mode, the initial energy consumption corresponding to each candidate state transformation mode is determined; the environmental information includes the weather type and the wind speed, and based on the environmental information and the plurality of candidate state transformation modes, the environmental energy correction coefficient corresponding to each candidate state transformation mode is determined, including: based on the weather type, the wind speed and the plurality of candidate state transformation modes, the influence degree of the environment on each candidate state transformation mode is determined; the influence degree of the environment on each candidate state transformation mode is linearly transformed to obtain the environmental energy correction coefficient.

[0013] Further, the application also proposes that, based on the instruction intention and the task information, the target control instruction of the unmanned aerial vehicle is determined from the plurality of candidate control instructions, including: based on the instruction intention and the task information, the instruction screening description information of the unmanned aerial vehicle is determined, the instruction screening description information is used to indicate the instruction screening manner, and the instruction intention is obtained through the user terminal; based on the instruction screening description information, the target control instruction of the unmanned aerial vehicle is determined from the plurality of candidate control instructions.

[0014] Further, the application also proposes that, based on the instruction intention and the task information, the instruction screening description information of the unmanned aerial vehicle is determined, including: the instruction intention and the task information are encoded respectively to obtain the instruction intention feature of the instruction intention and the task information feature of the task information; the instruction intention feature and the task information feature are fused to obtain the description information generation feature; the description information generation feature is decoded to obtain the instruction screening description information; based on the instruction screening description information, the target control instruction of the unmanned aerial vehicle is determined from the plurality of candidate control instructions, including: the instruction screening manner corresponding to the instruction screening description information is determined; the plurality of candidate control instructions are screened by using the instruction screening manner to obtain the target control instruction.

[0015] In one aspect, a multi-modal data fusion-based unmanned aerial vehicle endurance intelligent optimization system is provided, and the system includes: The acquisition module is configured to, in the case that the UAV is automatically performing a target task in a cruising optimization mode, acquire, in response to an initial control instruction of the UAV from a user terminal, an instruction intention of the initial control instruction, state information of the UAV, environment information of an environment in which the UAV is located, and task information of the target task, the initial control instruction being used to control the UAV to change a state, the user terminal being communicatively connected to the control terminal, and the control terminal being communicatively connected to the UAV. The first instruction determination module is configured to determine a plurality of candidate control instructions based on the initial control instruction, the state information, and the environment information, the state of the UAV after executing the candidate control instruction and the initial control instruction being the same, and the energy consumed by the UAV for executing the candidate control instruction being less than that for executing the initial control instruction. The second instruction determination module is configured to determine a target control instruction of the UAV from the plurality of candidate control instructions based on the instruction intention and the task information. The control module is configured to control the UAV to execute the target control instruction.

[0016] In an aspect, a computer device is provided, which includes one or more processors and one or more memories, and at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the method for intelligent optimization of cruising of a UAV based on multi-modal data fusion.

[0017] In an aspect, a computer readable storage medium is provided, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the method for intelligent optimization of cruising of a UAV based on multi-modal data fusion.

[0018] In an aspect, a computer program product or a computer program is provided, which includes program code stored in a computer readable storage medium, and a processor of a computer device reads the program code from the computer readable storage medium, and the processor executes the program code to enable the computer device to perform the method for intelligent optimization of cruising of a UAV based on multi-modal data fusion. As can be seen from the above, the method and system for intelligent optimization of cruising of a UAV based on multi-modal data fusion provided by the present application generate low-energy candidate instructions by fusing multi-modal data and intelligently select the optimal solution, effectively reduce energy consumption under the premise of maintaining the flight state, and have the technical effects of improving energy utilization efficiency and prolonging the effective cruising time of the UAV. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a schematic diagram of an implementation environment of a multi-modal data fusion-based unmanned aerial vehicle endurance intelligent optimization method provided by an embodiment of the present application; Figure 2 is a flowchart of a multi-modal data fusion-based unmanned aerial vehicle endurance intelligent optimization method provided by an embodiment of the present application; Figure 3 is a flowchart of generating a plurality of candidate control instructions provided by an embodiment of the present application; Figure 4 is a flowchart of determining a target control instruction provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of a multi-modal data fusion-based unmanned aerial vehicle endurance intelligent optimization system provided by an embodiment of the present application; Figure 6 is a structural schematic diagram of a control terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0022] In the present application, the terms "first", "second", etc. are used to distinguish the same or similar items with basically the same function, and it should be understood that there is no logical or time sequence relationship between "first", "second", "nth", and the number and execution order are not limited.

[0023] Unmanned aerial vehicle: full name "unmanned aerial vehicle" or "unmanned aerial vehicle", English usually called Unmanned Aerial Vehicle or Drone. Core feature: it is a kind of aircraft that does not need human pilots to operate directly on the machine. Control mode: usually controlled by ground operators through remote controller, or through on-board computer system according to preset program or artificial intelligence for autonomous flight.

[0024] Endurance optimization: refers to the time or distance that the unmanned aerial vehicle can work continuously after one energy supply. "Optimization" refers to the process of making a certain performance index reach a better state by improving design, technology or strategy.

[0025] Artificial Intelligence (AI) is the use of digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain better results.

[0026] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It is a specialized field that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to enabling computers to have intelligence. It is applied in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0027] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0028] In the related art, the optimization of the endurance of the unmanned aerial vehicle mainly depends on hardware improvement and single-dimensional control strategy. At the hardware level, the endurance time is improved by increasing the battery capacity or reducing the weight of the machine body, but it faces the problems of high cost and physical limitations. At the software control level, preset flight path planning or adjustment of flight parameters based on single sensor data is often used, which cannot comprehensively consider the coordination relationship between flight state, environmental variables and task target. After the user sends a control instruction, the unmanned aerial vehicle can only mechanically execute the preset action, and even if there is an equivalent low-energy path, the system lacks the ability to dynamically generate alternative instructions, resulting in suboptimal energy utilization.

[0029] To solve the above problems, the inventors found that the traditional control method has the defects of missing multi-modal data fusion and rigid instruction generation mechanism. First of all, it is noticed that there may be multiple paths with different energy consumptions for the unmanned aerial vehicle to perform the same state transformation, but the existing system does not establish a state equivalence analysis model. Secondly, it is found that the user's instruction intention and task target are not effectively included in the control decision-making process, resulting in deviation of the optimization direction from the actual demand. Further thinking about how to build a dynamic candidate instruction generation mechanism to predict the energy consumption difference by fusing environmental data and machine state. Finally, the technical route of generating equivalent low-energy instructions through multi-modal data collaborative analysis, combined with intention recognition and task demand to select the optimal instruction is formed.

[0030] Figure 1 is an implementation environment schematic diagram of a UAV endurance intelligent optimization method based on multi-modal data fusion provided by an embodiment of the present application, referring to Figure 1 The implementation environment can include a UAV 110, a control terminal 120, and a user terminal 130.

[0031] The control terminal 120 does not exist on the UAV 110, and the control terminal 120 is relatively independent of the UAV 110. The control terminal 120 and the UAV 110 are in communication connection, and the control terminal 120 can interact with the UAV 110. The user terminal 130 and the control terminal 120 are in communication connection, and the user terminal 130 can interact with the control terminal 120. In the embodiment of the present application, when the user uses the user terminal 130 to send a control instruction to the UAV 110, the control terminal 120 needs to be forwarded. In the case that the UAV is in the endurance optimization mode, the control terminal 120 will use the technical solution provided by the embodiment of the present application to process the control instruction sent by the user terminal 130. In the case that the UAV is not in the endurance optimization mode, the control terminal 120 directly forwards the control instruction sent by the user terminal 130 to the UAV 110.

[0032] In order to solve the above problems, referring to Figure 2 Taking the control terminal as an example of the execution subject, the present application proposes the following technical solution.

[0033] 201. In the case that the UAV is in the endurance optimization mode and automatically executes a target task, in response to an initial control instruction of the user terminal to the UAV, the instruction intention of the initial control instruction, the state information of the UAV, the environment information of the environment where the UAV is located, and the task information of the target task are obtained. The initial control instruction is used to control the UAV to change the state, the user terminal and the control terminal are in communication connection, and the control terminal and the UAV are in communication connection; 202. Based on the initial control instruction, the state information, and the environment information, a plurality of candidate control instructions are determined, the state of the UAV after executing the candidate control instruction and the initial control instruction is the same, and the energy consumed by the UAV executing the candidate control instruction is less than the initial control instruction; 203. Based on the instruction intention and the task information, a target control instruction of the UAV is determined from the plurality of candidate control instructions; 204. The UAV is controlled to execute the target control instruction.

[0034] The candidate control instruction refers to an alternative instruction set that can achieve the same state but with lower energy consumption by analyzing the predicted state corresponding to the initial instruction and deducing it in reverse. Specifically, a state equivalence verification algorithm can be used to achieve this, and the final state consistency of different control sequences can be verified by establishing a UAV dynamics model. The instruction intent refers to the operation purpose implied by the user when sending the control instruction. Natural language processing techniques can be used to analyze the user's input voice or text instructions and extract key operation target elements. The state information includes the real-time pose, motion parameters, and hardware state of the UAV. Three-dimensional space coordinates, velocity vectors, and power component working states can be obtained by using inertial navigation systems, GPS modules, and sensor fusion algorithms. The environmental information includes meteorological conditions and geographical features. Wind speed data can be collected using onboard meteorological sensors, and terrain elevation information can be obtained using digital elevation models. The task information refers to the target constraint conditions of the current operation, and the waypoint coordinates, operation area boundaries, and priority parameters output by the task planning system can be used.

[0035] Specifically, when the user sends a control instruction through the terminal, the control terminal synchronously collects the current flight attitude of the UAV, the power system load, and the surrounding wind speed data. The target state corresponding to the instruction is deduced through the inverse dynamics model, and the theoretical energy consumption value is calculated. Then, the environment database and the body performance model are called to generate multiple control sequences that can achieve the same target state, such as adjusting the flight altitude to reduce the motor load using the wind environment, or reducing the energy peak consumption by optimizing the acceleration curve. Each candidate instruction needs to be verified for state equivalence to ensure that the UAV reaches the predetermined position and attitude after execution. Finally, the operation target in the user's original instruction (such as quickly reaching the specified area) and the current task requirement (such as prioritizing shooting stability) are input into the decision model to filter out the lowest energy consumption instruction that meets both the operation intent and the task constraints.

[0036] Compared with related technologies, the existing scheme can only passively execute user instructions and cannot actively find equivalent low-energy consumption paths. The present scheme constructs a multi-modal data fusion framework, quantifies the impact of environmental airflow on the power system as an energy consumption correction coefficient, and predicts the energy consumption difference of different control sequences based on real-time battery state. At the same time, the instruction intent analysis module is introduced to avoid deviating from the user's operation target due to the pursuit of low energy consumption alone. Traditional methods can only maintain the original instruction when encountering adverse wind conditions, resulting in a sharp increase in energy consumption. However, the present scheme can automatically generate alternative instructions to adjust the flight altitude or path, and utilize airflow characteristics to reduce energy consumption.

[0037] By the technical solution, the dynamic optimization of the UAV control instruction is realized, the energy consumption is effectively reduced under the premise of ensuring the task execution effect. The equivalent low-energy consumption instruction is generated through multi-source data fusion analysis, and the problem of low energy utilization rate of the traditional method is solved. In combination with the intention recognition and the filtering mechanism of the task constraint, it is ensured that the optimized instruction meets the user operation expectation and meets the work requirements, and the control deviation caused by simply pursuing energy saving is avoided.

[0038] The application further provides the following technical solutions, please refer to Figure 3 Taking the control terminal as an example, the method comprises the following steps.

[0039] 301. Determine the predicted state of the UAV and the predicted energy consumption corresponding to the initial control instruction based on the initial control instruction, the state information and the environment information; 302. Determine a plurality of candidate state transformation modes in which the UAV transforms to the predicted state based on the state information and the predicted state; 303. Determine the energy consumption corresponding to each candidate state transformation mode based on the state information, the environment information and the plurality of candidate state transformation modes; 304. Determine a plurality of reference state transformation modes from the plurality of candidate state transformation modes based on the energy consumption corresponding to each candidate state transformation mode, the plurality of reference state transformation modes being the candidate state transformation modes with the energy consumption less than the predicted energy consumption; 305. Generate a plurality of candidate control instructions based on the plurality of reference state transformation modes.

[0040] The predicted state refers to the flight state that the UAV may reach after executing the initial control instruction, which can be realized by adopting the initial pose, initial motion information and initial control instruction to perform dynamic model calculation, and is used to simulate the state change after the instruction execution. The predicted energy consumption refers to the energy estimation required for executing the initial control instruction, which can be realized by adopting the power parameters of the dynamic components and the battery state information to perform dynamic calculation, and is used to establish the energy consumption benchmark. The candidate state transformation mode refers to different flight paths or action combinations that can reach the same final state as the initial control instruction, which can be realized by generating a plurality of feasible paths through a path planning algorithm, and is used to provide the generation basis of the alternative instruction. The reference state transformation mode refers to the candidate state transformation mode with lower energy consumption than the initial control instruction, which can be realized by comparing the energy consumption of the candidate mode with the predicted energy consumption for screening, and is used to screen the more energy-saving instruction scheme.

[0041] Specifically, when the UAV receives an initial control instruction, first, according to the current pose, speed and other state parameters and the content of the control instruction, the target state after execution is predicted through a kinematics model, and the estimated energy consumption of the instruction is calculated in combination with the power system efficiency and battery output characteristics. Subsequently, a plurality of different flight paths capable of reaching the same target state are generated through a path planning algorithm, for example, by using different combinations of flight altitudes, speed curves or attitude adjustment sequences. For each candidate path, the actual energy consumption is calculated in combination with the current wind speed, air density and other environmental parameters, for example, the power correction of the wind resistance impact of the flight path against the wind. Through energy consumption comparison, all candidate paths with lower energy consumption than the initial instruction are selected, which are converted into corresponding control instruction sequences to form a plurality of candidate control instructions.

[0042] Compared with related technologies, the existing scheme only executes a single control instruction sent by the user and does not analyze equivalent low-energy consumption paths, for example, the conventional system directly executes the flight altitude adjustment instruction specified by the user without considering the energy consumption difference of different climbing angles in the crosswind environment. The present scheme dynamically plans a plurality of equivalent paths and quantifies the energy consumption difference by fusing environmental data and body state data, thereby automatically generating an optimized instruction set, solving the energy consumption optimization blind area problem caused by the lack of multi-modal data fusion in related technologies.

[0043] Through the above technical scheme, the present application can generate a plurality of equivalent control instructions with lower energy consumption while ensuring the completion of the user instruction target. For example, in a headwind environment, a low-altitude flight path is selected to replace the high-altitude path specified by the user, and the ground effect is used to reduce the wind resistance impact, thereby reducing the energy consumption of the power system. The present scheme effectively solves the energy waste problem caused by the single path of the conventional control instruction execution mechanism, and improves the endurance of the UAV in complex environments.

[0044] The present application further proposes a UAV endurance intelligent optimization method based on multi-modal data fusion, comprising the following steps: determining the predicted state of the UAV based on the initial pose, initial motion information and initial control instruction, the initial motion information including the initial speed, initial motion direction and initial acceleration; determining the predicted energy consumption corresponding to the initial control instruction based on the initial control instruction and UAV information.

[0045] The initial pose refers to the position and attitude parameters of the unmanned aerial vehicle in a three-dimensional space, and can be specifically obtained by a global positioning system combined with an inertial measurement unit positioning method, and is used to establish a current space reference of the unmanned aerial vehicle. The initial motion information refers to the motion state parameters of the unmanned aerial vehicle, and can be specifically collected by a speed sensor, an accelerometer and a gyroscope, and is used to describe the dynamic behavior characteristics of the unmanned aerial vehicle. The unmanned aerial vehicle information refers to parameters related to the hardware configuration of the unmanned aerial vehicle, and can be specifically read from a flight control system storage module, and includes power component specifications, battery performance indicators and whole machine weight data, and is used to calculate physical constraint conditions of energy consumption.

[0046] Specifically, after the user sends an initial control instruction to the unmanned aerial vehicle, the system first analyzes the action demand corresponding to the instruction. Based on the current spatial position, flight speed, motion direction and acceleration data of the unmanned aerial vehicle, and in combination with the adjustment amplitude of the control instruction, the predicted pose and motion parameters after executing the instruction are deduced through a kinematics model. At the same time, an energy consumption calculation model is established according to the power system specifications of the unmanned aerial vehicle, the current working state of the battery and the weight parameters of the machine body. The model accurately calculates the energy consumption value required for executing the control instruction by analyzing the matching relationship between the motor power demand and the battery output characteristics, and provides a quantitative basis for subsequent optimization instruction screening.

[0047] Compared with related technologies, the traditional scheme usually directly executes actions according to control instructions without fully considering the dynamic influence of the current motion state of the unmanned aerial vehicle on energy consumption. The energy estimation method in related technologies is mostly based on theoretical nominal parameters without dynamic correction in combination with the actual working state of the battery. The present scheme fuses real-time motion parameters and hardware state data, so that the prediction model can reflect the energy consumption characteristics under real flight conditions, and the accuracy of energy consumption prediction is improved.

[0048] Through the above technical scheme, the present application effectively solves the problem of insufficient accuracy of traditional control instruction energy consumption prediction, can accurately evaluate the energy cost of different control strategies, and provides a reliable basis for generating equivalent low-energy consumption alternative instructions. By dynamically integrating motion state and hardware parameters, the energy consumption estimation deviation caused by ignoring real-time dynamic factors is avoided, and the optimized control instruction not only meets the state transformation demand but also realizes energy consumption minimization.

[0049] The present application further proposes that the predicted pose and the predicted motion information of the unmanned aerial vehicle are determined based on the initial pose, the initial motion information and the initial control instruction, and the predicted state is determined based on the predicted pose and the predicted motion information. The unmanned aerial vehicle information includes power component information, battery state information and unmanned aerial vehicle weight, the state transformation demand power of the unmanned aerial vehicle is determined based on the initial control instruction, the power component information and the unmanned aerial vehicle weight, and the predicted energy consumption corresponding to the initial control instruction is determined based on the state transformation demand power and the battery state information.

[0050] wherein the initial pose refers to the three-dimensional spatial position and attitude angle of the UAV when receiving the initial control instruction, which can be realized by combining GPS coordinates and gyroscope data, and is used to describe the spatial state of the UAV. The initial motion information refers to the velocity, motion direction and acceleration of the UAV, which can be realized by real-time data collected by an inertial measurement unit, and is used to reflect the dynamic characteristics of the UAV. The predicted pose refers to the three-dimensional spatial position and attitude angle that the UAV is expected to reach after executing the initial control instruction, which can be calculated and generated by a kinematics model combined with control instruction parameters, and is used to predict the state change result. The predicted motion information refers to the expected velocity, motion direction and acceleration of the UAV after executing the initial control instruction, which can be generated by dynamics simulation or historical data fitting, and is used to evaluate the dynamic impact of the control instruction. The power component information refers to the performance parameters of the power system of the UAV, such as motor efficiency and propeller thrust coefficient, which can be obtained by device specification parameters or measured data, and is used to calculate power demand. The battery state information includes remaining capacity, battery temperature and internal resistance, which can be monitored in real time by a battery management system, and is used to evaluate the energy supply capacity. The weight of the UAV refers to the total mass including the load, which can be obtained by a weighing sensor or a preset parameter, and is used to calculate power demand. The state transformation demand power refers to the total power required to implement the control instruction, which can be calculated by kinematics equations combined with the weight of the UAV and the efficiency of the power component, and is used to quantify the energy consumption benchmark.

[0051] Specifically, in determining the predicted state, first, according to the initial pose, the initial motion information and the initial control instruction, the predicted pose and the predicted motion information are derived by a kinematics model. For example, if the initial control instruction is to climb a height, based on the current height, vertical velocity and acceleration, combined with the target height and acceleration parameters of the climbing instruction, the vertical coordinate change quantity in the predicted pose and the velocity change curve in the predicted motion information are calculated. In calculating the predicted energy consumption, the pose transformation mode and the motion transformation mode are analyzed according to the initial control instruction, such as horizontal displacement or rotation angle, combined with the weight of the UAV to calculate the basic power to overcome gravity or inertia, and then the basic power is corrected according to the motor efficiency parameters in the power component information to obtain the state transformation demand power. Subsequently, based on the remaining capacity, temperature and internal resistance parameters in the battery state information, the discharge efficiency of the battery under the current working condition is evaluated, and finally the predicted energy consumption corresponding to the initial control instruction is calculated.

[0052] Compared with the related art, the existing scheme usually only estimates energy consumption according to the remaining power or simple flight distance, without considering the influence of power component efficiency difference or battery state dynamic change on energy consumption. For example, the traditional method may only calculate energy consumption based on motor rated power and instruction execution time, without considering the change of internal resistance caused by battery temperature rise. The present scheme modifies the basic power by introducing power component information, and combines real-time battery state parameters, so that the energy consumption prediction is closer to the actual working condition.

[0053] Through the above technical scheme, the present application can more accurately predict the energy consumption corresponding to different control instructions, thereby providing a reliable basis for subsequent screening of low-energy consumption candidate instructions. For example, under the same flight target, the system can identify the control instruction with lower energy consumption due to higher power component efficiency or better battery discharge efficiency, avoid energy waste caused by prediction deviation, and improve the endurance of the unmanned aerial vehicle.

[0054] The present application further proposes to determine the pose transformation mode and the motion transformation mode indicated by the initial control instruction, determine the state transformation basic power of the unmanned aerial vehicle based on the pose transformation mode, the motion transformation mode and the weight of the unmanned aerial vehicle, modify the state transformation basic power based on the power component information to obtain the state transformation demand power; the battery state information includes the remaining power, the battery temperature and the battery internal resistance, the current battery state is determined based on the remaining power, the battery temperature and the battery internal resistance, and the predicted energy consumption corresponding to the initial control instruction is determined based on the current battery state and the state transformation demand power.

[0055] The pose transformation mode refers to the position and attitude adjustment path of the unmanned aerial vehicle in three-dimensional space, which can be realized by using waypoint sequence or Euler angle change amount, and is used to describe the pose migration process of the unmanned aerial vehicle from the initial state to the target state. The power component information includes motor efficiency curve, propeller thrust coefficient and transmission loss parameter, which can be obtained through experiment calibration or dynamic monitoring, and is used to modify the deviation between theoretical calculation power and actual consumption. The battery internal resistance refers to the impedance characteristic of the internal conductive material of the battery, which can be measured by direct current internal resistance test method or alternating current impedance spectrum method, and the instantaneous output capacity of the battery can be evaluated in combination with the battery temperature and the remaining power.

[0056] Specifically, when the initial control instruction is triggered, the pose adjustment parameters and motion mode change requirements contained therein are first analyzed, and the theoretical power required to maintain the state transformation is calculated in combination with the self-weight of the unmanned aerial vehicle. Subsequently, the theoretical power is dynamically compensated according to the actual performance parameters of the power system, for example, the power margin is increased when the motor efficiency is lower than the rated value. In the battery state evaluation process, the remaining power is used to judge the upper limit of energy reserve, the battery temperature affects the change of internal resistance, and the internal resistance value is directly related to the energy transmission loss, and the three together build a battery instantaneous discharge model, and then the energy consumption required by the instruction is calculated in combination with the modified demand power.

[0057] Compared with the related art, the conventional scheme usually only estimates the endurance time according to the remaining battery power, without considering the influence of power assembly efficiency fluctuation and battery dynamic characteristics on energy consumption. For example, the actual consumption may be higher than the theoretical value due to the increased battery internal resistance in a low-temperature environment, and the conventional method cannot capture such deviation. The present scheme realizes fine modeling of energy consumption by introducing a power assembly correction coefficient and multi-dimensional battery state parameters.

[0058] Through the above technical scheme, the present application can more accurately predict the energy consumption difference corresponding to different control instructions, providing a reliable basis for subsequent screening of low-energy consumption equivalent instructions. Especially in complex environments or battery performance degradation scenarios, it can effectively avoid the misjudgment of endurance time caused by power estimation deviation, and improve the energy utilization efficiency of the unmanned aerial vehicle.

[0059] The present application further proposes to determine the initial state of the unmanned aerial vehicle based on the initial pose and initial motion information, and to plan state transformation based on the unmanned aerial vehicle information, the initial state and the predicted state, to obtain a plurality of candidate state transformation modes, wherein the unmanned aerial vehicle information includes power assembly information and unmanned aerial vehicle weight, and the candidate state transformation mode is a state transformation mode that can be realized by the unmanned aerial vehicle.

[0060] The initial pose refers to the position and attitude parameters of the unmanned aerial vehicle in three-dimensional space, which can be realized by fusion calculation of GPS coordinates, gyroscope data and accelerometer data, and is used to represent the spatial state reference of the unmanned aerial vehicle. The initial motion information refers to the speed, direction and acceleration parameters of the unmanned aerial vehicle, which can be realized by real-time data collected by an inertial measurement unit, and is used to describe the motion state baseline of the unmanned aerial vehicle. The state transformation planning refers to the process of generating a feasible path according to the hardware characteristics of the unmanned aerial vehicle and the target state, which can be realized by a trajectory optimization algorithm based on a dynamics model, and the candidate scheme that meets the performance of the power assembly and the weight limit is selected through constraint conditions. The power assembly information refers to the motor power, propeller specification and propulsion efficiency parameters, which can be realized by calling a device parameter library, and is used to evaluate the energy demand threshold of different state transformation modes. The weight of the unmanned aerial vehicle refers to the mass data of the whole machine including the load, which can be realized by real-time monitoring of an on-board weight sensor, and is used to calculate the basic power required for state transformation.

[0061] Specifically, after obtaining the initial pose and initial motion information, an initial state model is constructed through coordinate transformation and kinematic equations. Based on the difference between the model and the target predicted state, a variety of state transformation paths are generated using a constrained optimization algorithm. During the planning process, the power component information is used to set the upper limit of the motor thrust, and the weight of the unmanned aerial vehicle is used to calculate the inertia compensation requirement, to ensure that the generated candidate mode meets the actual hardware capability. For example, when height lifting needs to be achieved, the planning algorithm will generate multiple schemes including vertical climbing, inclined climbing and segmented climbing, each of which meets the maximum output limit of the power system. The generation process of the candidate mode set is completed through multi-objective optimization, which prioritizes paths with lower energy consumption while ensuring that the target state is reached.

[0062] Compared with related technologies, the traditional scheme only generates control instructions according to preset flight paths or single sensor data, without considering the impact of unmanned aerial vehicle power system performance and weight parameters on energy consumption. For example, in the same flight task, related technologies may directly use the shortest path instructions, ignoring the problem of motor overload caused by large-angle turning in strong wind environments. The present scheme introduces power component information and weight parameters as planning constraints to ensure that the generated candidate modes are within the range of hardware that can be tolerated, avoiding the failure of state transformation due to insufficient power, while effectively reducing the risk of abnormal energy consumption.

[0063] Through the above technical solutions, the present application can generate a set of feasible state transformation modes that meet the physical characteristics of the unmanned aerial vehicle, providing effective candidates for subsequent low-energy consumption instruction selection. By taking the performance of the power system and the weight parameter as core constraint conditions, the feasibility of the candidate modes in actual execution is ensured, avoiding the invalidation of control instructions due to hardware limitations, while providing a reliable path selection basis for energy consumption optimization.

[0064] The present application further proposes determining the initial energy consumption corresponding to each candidate state transformation mode based on state information and a plurality of candidate state transformation modes; determining an environmental energy correction coefficient corresponding to each candidate state transformation mode based on environmental information and a plurality of candidate state transformation modes; and multiplying the initial energy consumption corresponding to each candidate state transformation mode and the corresponding environmental energy correction coefficient to obtain the energy consumption corresponding to each candidate state transformation mode.

[0065] The initial energy consumption refers to the theoretical energy consumption required for the unmanned aerial vehicle to perform a specific state transformation, which can be calculated through dynamic trajectory calculation of the pose change set and the motion information change set. The pose change set includes all spatial coordinate points in the candidate state transformation process, and the motion information change set includes sequences of motion parameters such as velocity and acceleration. This feature is used to establish a quantitative relationship between state transformation and basic energy consumption.

[0066] The environmental energy correction coefficient refers to the influence weight of the external environment on energy consumption, and can be calculated by a model of the effect of weather type and wind speed on flight resistance, for example, the correction coefficient can be set to a value greater than 1 under the condition of headwind. This feature is used to quantify the superimposed effect of environmental factors on actual energy consumption.

[0067] Specifically, in calculating the energy consumption of the candidate state transformation mode, first, the pose change set and the motion information change set are generated according to the initial pose and motion information of the unmanned aerial vehicle, and the instantaneous power at each time is calculated through a dynamics model and is integrated to obtain the initial energy consumption. Subsequently, an energy correction model is established according to real-time environmental data, for example, in heavy rain weather, the correction coefficient can be dynamically adjusted based on the impact force of raindrops on the rotor. Finally, the initial energy consumption is multiplied by the correction coefficient to obtain the actual energy consumption prediction value after the comprehensive environmental interference.

[0068] Compared with related technologies, the traditional method only estimates the energy consumption according to the battery output power, without distinguishing the differences in state transformation trajectories and environmental interference factors. For example, the existing flight path planning system only calculates the theoretical power consumption of straight flight, without considering the actual rotor power fluctuation caused by crosswind. The present scheme realizes accurate energy consumption prediction under the coupling action of multiple factors by separating the basic energy consumption and the environmental correction term.

[0069] Through the above technical solutions, the present application effectively solves the problem of missing environmental interference factors in the energy consumption evaluation of the unmanned aerial vehicle. For example, in a strong crosswind environment, the traditional method may underestimate the actual energy consumption, resulting in a calculation error of the endurance, while the present scheme can accurately identify the high energy consumption characteristics of the headwind path by introducing the environmental correction coefficient, thereby avoiding the selection of unreasonable control instructions. This technical means makes the energy consumption evaluation of the candidate instructions more close to the real flight conditions, and provides reliable data support for subsequent screening of low energy consumption instructions.

[0070] The present application further proposes determining, based on the initial pose, the initial motion information, and the plurality of candidate state transformation modes, a pose change set and a motion information change set corresponding to each candidate state transformation mode, the pose change set including a plurality of poses when the corresponding candidate state transformation mode is used for state transformation, and the motion information change set including a plurality of motion information when the corresponding candidate state transformation mode is used for state transformation; determining, based on the pose change set and the motion information change set corresponding to each candidate state transformation mode, an initial energy consumption corresponding to each candidate state transformation mode; determining, based on the weather type, the wind speed, and the plurality of candidate state transformation modes, an influence degree of the environment on each candidate state transformation mode; and performing linear transformation on the influence degree of the environment on each candidate state transformation mode to obtain an environmental energy correction coefficient.

[0071] The pose change set refers to the spatial position and attitude sequence experienced by the unmanned aerial vehicle during the candidate state transformation process, which can be generated by combining real-time attitude data collected by an inertial measurement unit with a position trajectory prediction algorithm, and is used to quantify the spatial displacement characteristics under different transformation modes. The motion information change set refers to the time sequence changes of dynamic parameters such as velocity and acceleration during the candidate state transformation process, which can be obtained by simulating and calculating the candidate path through a kinematics model, and is used to reflect the demand differences of different transformation modes on the power system. The weather type and wind speed refer to environmental parameters obtained through a meteorological sensor or an external data interface, such as real-time data collected by a barometer and an anemometer, which are used to evaluate the influence of air flow on flight resistance. The environmental energy correction coefficient refers to a quantitative adjustment value of the influence of environmental factors on the energy consumption of the candidate transformation mode, which can be calculated by a regression model based on historical flight data, and is used to correct the deviation between theoretical energy consumption and actual energy consumption.

[0072] Specifically, when determining the initial energy consumption of the candidate state transformation mode, the displacement in the pose change set and the acceleration data in the motion information change set are decomposed, and the theoretical kinetic energy change is calculated by combining the mass parameters of the unmanned aerial vehicle, and then the theoretical power consumption of the power system is derived. For example, for a candidate transformation mode containing climbing action, the height increment in the pose change set and the vertical acceleration data in the motion information change set are input into the power calculation model, and the basic energy consumption is estimated by combining the motor efficiency curve. When calculating the environmental energy correction coefficient, according to the angle between the current wind speed and the flight direction of the candidate transformation mode, and combining the air density parameters corresponding to the weather type, a resistance influence factor matrix is established. For example, when the candidate transformation mode contains an upwind flight path, by the product relationship of the resistance factor and the flight duration, a linear transformation coefficient is generated to dynamically adjust the basic energy consumption.

[0073] Compared with related technologies, the traditional method usually estimates energy consumption only according to the remaining power or the simple path length, without considering the coupling effect of flight attitude dynamic change and environmental resistance. For example, the existing flight path planning system still calculates energy consumption under windless conditions in an upwind environment, resulting in actual power consumption exceeding expectations during actual flight. However, the present scheme can accurately quantify the load demand of different transformation modes on the power system through joint analysis of the pose change set and the motion information change set, and dynamically compensate the additional power consumption caused by air flow disturbance by introducing the environmental energy correction coefficient, thereby improving the accuracy of energy consumption prediction.

[0074] By the technical solution, the application can effectively solve the energy consumption evaluation deviation problem caused by ignoring the correlation between the environmental variables and the flight state in the traditional unmanned aerial vehicle control system. By establishing the pose and motion information change set of the candidate transformation mode, and combining the environmental parameter dynamic correction energy consumption calculation model, the energy consumption prediction result is more in line with the actual flight conditions, and reliable data foundation is provided for subsequent screening of low energy consumption control instructions. For example, when encountering side wind interference, the system can automatically identify the additional resistance caused by the heading adjustment, and generate a corrected energy consumption evaluation value, so as to preferentially select the transformation mode affected less by the air flow.

[0075] The application further proposes the following technical solution, see Figure 4 Taking the control terminal as an execution subject, the method includes the following contents.

[0076] 401. Based on the instruction intention and the task information, determine the instruction screening description information of the unmanned aerial vehicle, the instruction screening description information is used to indicate the instruction screening mode, the instruction intention is obtained through the user terminal, and the instruction intention is obtained through the user terminal. 402. Based on the instruction screening description information, determine the target control instruction of the unmanned aerial vehicle from the plurality of candidate control instructions.

[0077] The instruction intention refers to the flight state or task target expected to be achieved by the control instruction sent by the user terminal, and can be specifically realized by performing semantic analysis on the voice or text input by the user through natural language processing technology, and is used to clearly indicate the user's control demand for the unmanned aerial vehicle. The task information refers to the target task related parameters of the unmanned aerial vehicle currently executed, for example, waypoint coordinates, task priority, time limit, which can be extracted by a task planning module to structure data, and is used to constrain the direction of instruction screening. The instruction screening description information refers to the screening condition set generated by fusing the instruction intention and the task information, and can specifically generate multi-dimensional decision parameters by using feature vector splicing or attention mechanism, and is used to dynamically adjust the evaluation weight of the candidate instruction. The encoding refers to converting the instruction intention and the task information into numerical features that can be processed by a machine, for example, using a word embedding model to vectorize the text intention, or extracting the spatiotemporal features of the task parameters through a convolutional neural network. The decoding refers to converting the fused features into specific screening logic, for example, mapping into decision tree rules or probability distribution through a fully connected layer, and is used to quantify the matching degree of the candidate instruction. The instruction screening mode refers to the evaluation strategy defined according to the screening description information, for example, a multi-objective optimization algorithm based on energy consumption priority, path smoothness or risk coefficient, and is used to select the optimal solution from the candidate instruction.

[0078] Specifically, when the user sends a control instruction through the terminal, the system first performs semantic analysis on the instruction intention, such as identifying the speed requirement in "quickly reach the target point", and simultaneously obtaining the obstacle distribution of the current route or the remaining power limit from the task information. Subsequently, both are converted into high-dimensional feature vectors through the encoding layer, and a comprehensive screening description information is generated by using the feature fusion module, such as fusing the speed requirement and the power limit into "select the fastest path within the remaining power limit". The decoding layer converts the fused features into specific screening rules, such as setting the energy consumption weight to 0.7 and the path length to 0.3 of the evaluation function. Finally, the system sorts the candidate instructions according to the function, and selects the instruction with the highest comprehensive score as the target control instruction.

[0079] In some specific embodiments, the encoding process can extract the time sequence correlation features in the instruction intention by using a bidirectional long short-term memory network, and the task information is modeled by using a graph neural network to model the spatial topological relationship. In the fusion stage, the contribution of the two types of features is dynamically adjusted by using a cross-attention mechanism, such as increasing the weight of the time dimension in an emergency task. In the decoding process, a reinforcement learning model can be used to dynamically update the screening rules according to historical execution data, such as automatically reducing the priority of the height change instruction in a strong wind environment.

[0080] Compared with related technologies, the existing scheme usually only selects a control instruction according to a single dimension (such as the remaining power), and cannot coordinate the user intention and the task constraint. For example, when the traditional system receives the "accelerate flight" instruction, it may directly increase the motor power, but ignores the headwind resistance or the remaining flight range in the task information, resulting in actual energy consumption exceeding the expectation. However, the present scheme can combine the acceleration requirement in the user intention and the wind speed data in the task environment by fusing encoding and dynamic decoding, to generate a screening rule that meets the intention and conforms to the energy consumption limit, such as selecting the instruction to climb the height to take advantage of the airflow to fly against the wind.

[0081] Through the above technical solutions, the present application solves the problem of single dimension of instruction screening and inability to balance user demand and task constraints in related technologies. By feature-level fusion of instruction intention and task information, the screening process can take into account both the control target and the environmental constraints, such as meeting the user-specified delivery time in a logistics distribution task, and selecting a path that bypasses a low-wind-speed area according to the real-time power. This improves the adaptation accuracy of the control instruction and avoids energy waste or task interruption caused by instruction conflicts.

[0082] The application further proposes a method for determining a UAV instruction screening description information based on instruction intention and task information, including encoding the instruction intention and the task information to obtain instruction intention features and task information features, fusing the two to generate description information generation features and decoding to obtain instruction screening description information; and determining a target control instruction from candidate control instructions based on a screening method corresponding to the description information.

[0083] Among them, the instruction intention features refer to converting the semantic information of the user instruction into a structured vector representation through natural language processing technology, which can be implemented by using a Transformer-based encoder model, for example, using a BERT model to perform context encoding on the instruction text. The task information features refer to converting the task parameters and constraints into numerical features, which can be implemented by using an embedding layer to vectorize discrete parameters such as task type, priority, and time limit. The description information generation features refer to a multi-dimensional feature space that fuses the relevance of intention and task, which can be implemented by using an attention mechanism to dynamically adjust the weight ratio of the two types of features. The instruction screening method refers to a strategy for selecting instructions according to feature matching degree, which can be implemented by using cosine similarity to calculate the matching degree of the candidate instructions and the description information, and selecting the instruction with the highest similarity as the target control instruction.

[0084] Specifically, after the UAV control terminal receives the user instruction, the instruction text is first input into the pre-trained language model for encoding to extract a high-dimensional feature vector containing operation intentions such as flight direction and speed adjustment. At the same time, the parameters such as waypoint coordinates and shooting duration in the task information are converted into equal-dimensional numerical vectors through an embedding layer. The two types of features interact through an attention fusion module to generate joint features that reflect the degree of association between intention and task. The joint features are decoded through an LSTM network to form description information containing screening conditions, such as "preferentially selecting a side wind avoidance path" or "limiting the change amplitude of the pitch angle". Finally, the optimal control instruction that meets both the user's intention and the task's demand is selected by calculating the matching degree of the candidate instructions and the description information.

[0085] Compared with related technologies, the traditional method only screens instructions through keyword matching or fixed rules, and cannot effectively fuse dynamically changing user intentions and complex task parameters. For example, the existing system can only adjust the flight height according to the preset rules when encountering sudden side winds, but cannot select a flight path that takes into account stability and efficiency in combination with the time sensitivity of the current shooting task. The present scheme realizes multi-modal feature fusion of intention and task through a deep learning model, so that the instruction screening process can dynamically adapt to the optimization needs in different scenarios.

[0086] By the technical solution, the intelligent screening of the control instruction is realized, and the instruction selection deviation problem caused by the single data dimension in the traditional method is solved. By accurately analyzing the association between the user intention and the task constraint, the optimal instruction that can reduce the energy consumption and meet the operation purpose is effectively screened out, and the endurance optimization effect of the unmanned aerial vehicle in the multi-task scene in the complex environment is significantly improved.

[0087] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described one by one here.

[0088] Figure 5 is a structural schematic diagram of an unmanned aerial vehicle endurance intelligent optimization system based on multi-modal data fusion provided by an embodiment of the present application, referring to Figure 5 The system comprises an acquisition module 501, a first instruction determination module 502, a second instruction determination module 503, and a control module 504.

[0089] The acquisition module 501 is configured to, in the case that the unmanned aerial vehicle is in an endurance optimization mode to automatically execute a target task, acquire, in response to an initial control instruction of the unmanned aerial vehicle by a user terminal, an instruction intention of the initial control instruction, state information of the unmanned aerial vehicle, environment information of an environment where the unmanned aerial vehicle is located, and task information of the target task, the initial control instruction being used to control the unmanned aerial vehicle to change state, the user terminal being in communication connection with the control terminal, and the control terminal being in communication connection with the unmanned aerial vehicle.

[0090] The first instruction determination module 502 is configured to determine a plurality of candidate control instructions based on the initial control instruction, the state information, and the environment information, the state of the unmanned aerial vehicle after executing the candidate control instruction and the initial control instruction being the same, and the energy consumed by the unmanned aerial vehicle executing the candidate control instruction being less than the initial control instruction.

[0091] The second instruction determination module 503 is configured to determine a target control instruction of the unmanned aerial vehicle from the plurality of candidate control instructions based on the instruction intention and the task information.

[0092] The control module 504 is configured to control the unmanned aerial vehicle to execute the target control instruction.

[0093] It should be noted that the above embodiment provides a multi-modal data fusion based unmanned aerial vehicle endurance intelligent optimization system. When the model is warned, only the above-mentioned division of each functional module is used as an example for illustration. In actual application, the above-mentioned functions can be distributed to be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the multi-modal data fusion based unmanned aerial vehicle endurance intelligent optimization system and the multi-modal data fusion based unmanned aerial vehicle endurance intelligent optimization method embodiment provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0094] Figure 6 A control terminal 120 provided by an embodiment of the present application is shown in the structural diagram. The control terminal 120 can have a large difference due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 601 and one or more memories 602. The one or more memories 602 store at least one computer program, which is loaded and executed by the one or more processors 601 to implement the method provided by each method embodiment described above. Of course, the control terminal 120 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for realizing the functions of the device, and will not be described here.

[0095] In an exemplary embodiment, a computer readable storage medium, such as a memory including a computer program, is also provided. The computer program can be executed by a processor to complete the multi-modal data fusion based unmanned aerial vehicle endurance intelligent optimization method in the above embodiment. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0096] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes program code stored in a computer readable storage medium. The processor of the computer device reads the program code from the computer readable storage medium. The processor executes the program code, so that the computer device executes the above multi-modal data fusion based unmanned aerial vehicle endurance intelligent optimization method.

[0097] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0098] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for intelligent optimization of UAV endurance based on multimodal data fusion, characterized in that, The method, executed by the control terminal of the unmanned aerial vehicle, includes: When the drone is in the endurance optimization mode and automatically executing the target task, in response to the user terminal's initial control command to the drone, the command intent of the initial control command, the state information of the drone, the environmental information of the environment in which the drone is located, and the task information of the target task are obtained. The initial control command is used to control the drone to change its state. The user terminal is in communication connection with the control terminal, and the control terminal is in communication connection with the drone. Based on the initial control command, the state information, and the environmental information, multiple candidate control commands are determined. The state of the UAV after executing the candidate control command is the same as that after executing the initial control command, and the energy consumed by the UAV in executing the candidate control command is less than that of the initial control command. Based on the instruction intent and the task information, the target control instruction for the UAV is determined from the plurality of candidate control instructions; Control the drone to execute the target control commands.

2. The method according to claim 1, characterized in that, The determination of multiple candidate control commands based on the initial control command, the status information, and the environmental information includes: Based on the initial control command, the status information, and the environmental information, the predicted state of the UAV and the predicted energy consumption corresponding to the initial control command are determined. Based on the state information and the predicted state, multiple candidate state transformation methods for the UAV to transform into the predicted state are determined; Based on the state information, the environmental information, and the multiple candidate state transition methods, determine the energy consumption corresponding to each of the candidate state transition methods; Based on the energy consumption corresponding to each of the candidate state transformation modes, a plurality of reference state transformation modes are determined from the plurality of candidate state transformation modes. The plurality of reference state transformation modes are candidate state transformation modes whose corresponding energy consumption is less than the predicted energy consumption. Based on the multiple reference state transformation methods, the multiple candidate control commands are generated.

3. The method according to claim 2, characterized in that, The state information includes initial pose, initial motion information, and UAV information. The step of determining the predicted state of the UAV and the predicted energy consumption corresponding to the initial control command based on the initial control command, the state information, and the environmental information includes: Based on the initial pose, the initial motion information, and the initial control command, the predicted state of the UAV is determined. The initial motion information includes initial velocity, initial motion direction, and initial acceleration. Based on the initial control command and the UAV information, the predicted energy consumption corresponding to the initial control command is determined.

4. The method according to claim 3, characterized in that, Determining the predicted state of the UAV based on the initial pose, the initial motion information, and the initial control command includes: Based on the initial pose, the initial motion information, and the initial control command, the predicted pose and predicted motion information of the UAV are determined; based on the predicted pose and the predicted motion information, the predicted state is determined. The drone information includes power component information, battery status information, and drone weight. The step of determining the predicted energy consumption corresponding to the initial control command based on the initial control command and the drone information includes: Based on the initial control command, the power component information, and the weight of the UAV, the power requirement for the UAV's state transition is determined; based on the power requirement for the state transition and the battery status information, the predicted energy consumption corresponding to the initial control command is determined.

5. The method according to claim 4, characterized in that, The process of determining the power requirement for state transitions of the UAV based on the initial control command, the power component information, and the UAV weight includes: The pose transformation mode and motion transformation mode indicated by the initial control command are determined; based on the pose transformation mode, the motion transformation mode, and the weight of the UAV, the basic power for state transformation of the UAV is determined; the basic power for state transformation is corrected based on the power component information to obtain the power required for state transformation of the UAV. The battery status information includes remaining charge, battery temperature, and battery internal resistance. Determining the predicted energy consumption corresponding to the initial control command based on the power demand for the state transition and the battery status information includes: Based on the remaining battery power, the battery temperature, and the battery internal resistance, the current battery state is determined; based on the current battery state and the power required for the state transition, the predicted energy consumption corresponding to the initial control command is determined.

6. The method according to claim 2, characterized in that, The state information includes initial pose, initial motion information, and UAV information. The step of determining multiple candidate state transition methods for the UAV to transform into the predicted state based on the state information and the predicted state includes: Based on the initial pose and the initial motion information, the initial state of the UAV is determined; Based on the UAV information, the initial state, and the predicted state, a state transformation plan is performed to obtain the multiple candidate state transformation methods. The UAV information includes power component information and UAV weight. The multiple candidate state transformation methods are the state transformation methods that the UAV can achieve.

7. The method according to claim 6, characterized in that, The step of determining the energy consumption corresponding to each of the candidate state transition modes based on the state information, the environmental information, and the multiple candidate state transition modes includes: Based on the state information and the multiple candidate state transformation methods, determine the initial energy consumption corresponding to each candidate state transformation method; Based on the environmental information and the multiple candidate state transformation methods, determine the environmental energy correction coefficient corresponding to each of the candidate state transformation methods; The energy consumption corresponding to each of the candidate state transformation methods is obtained by multiplying the initial energy consumption corresponding to the corresponding environmental energy correction coefficient.

8. The method according to claim 7, characterized in that, The step of determining the initial energy consumption corresponding to each of the candidate state transition modes based on the state information and the multiple candidate state transition modes includes: Based on the initial pose, the initial motion information, and the multiple candidate state transformation modes, a pose change set and a motion information change set corresponding to each candidate state transformation mode are determined. The pose change set includes multiple poses when the state is transformed using the corresponding candidate state transformation mode, and the motion information change set includes multiple motion information when the state is transformed using the corresponding candidate state transformation mode. Based on the pose change set and the motion information change set corresponding to each candidate state transformation mode, the initial energy consumption corresponding to each candidate state transformation mode is determined. The environmental information includes weather type and wind speed. Based on the environmental information and the multiple candidate state transition methods, the determination of the environmental energy correction coefficient corresponding to each candidate state transition method includes: Based on the weather type, the wind speed, and the multiple candidate state transformation modes, the degree of influence of the environment on each of the candidate state transformation modes is determined; a linear transformation is performed on the degree of influence of the environment on each of the candidate state transformation modes to obtain the environmental energy correction coefficient.

9. The method according to claim 1, characterized in that, The step of determining the target control command for the UAV from the plurality of candidate control commands based on the command intent and the mission information includes: Based on the instruction intent and the task information, instruction filtering description information of the UAV is determined. The instruction filtering description information is used to indicate the method of instruction filtering. The instruction intent is obtained through the user terminal. Based on the instruction filtering description information, the target control instruction of the UAV is determined from the plurality of candidate control instructions.

10. A UAV endurance intelligent optimization system based on multimodal data fusion, characterized in that, The system includes: The acquisition module is used to acquire the instruction intent of the initial control command, the state information of the drone, the environmental information of the environment in which the drone is located, and the task information of the target task in response to the initial control command of the user terminal to the drone when the drone is in the endurance optimization mode and automatically executing the target task. The initial control command is used to control the drone to change its state. The user terminal is communicatively connected to the control terminal, and the control terminal is communicatively connected to the drone. The first instruction determination module is used to determine multiple candidate control instructions based on the initial control instruction, the state information, and the environmental information. The state of the UAV after executing the candidate control instructions is the same as that after executing the initial control instruction, and the energy consumed by the UAV in executing the candidate control instructions is less than that of the initial control instruction. The second instruction determination module is used to determine the target control instruction of the UAV from the plurality of candidate control instructions based on the instruction intent and the task information; The control module is used to control the UAV to execute the target control commands.

Images