Solar-powered unmanned aerial vehicle flight control method and device, electronic equipment and storage medium

Abstract

The application provides a solar unmanned aerial vehicle flight control method, comprising the following steps: establishing a multi-level agent architecture based on the information transmission relationship among the whole machine system, subsystem and component of the solar unmanned aerial vehicle; training the multi-level agent architecture to obtain a distributed neural network control structure and parameters thereof; making decisions based on the agent information through the neural network, and outputting control instructions to the whole machine system, subsystem or component of the solar unmanned aerial vehicle which has a physical mapping relationship with the agent corresponding to the neural network, so as to control the whole machine system, subsystem or component of the solar unmanned aerial vehicle to perform actions. The application constructs a distributed neural network control structure through physical mapping, generates control instructions for specific rudder surfaces and motors, effectively solves the problem that the number of intermediate layers in the reinforcement learning of the solar unmanned aerial vehicle flight control system is large and it is difficult to study the characteristics of the weights in different layers, and improves the decision-making ability and endurance of the solar unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to the field of solar-powered unmanned aerial vehicle (UAV) flight control technology, specifically to a solar-powered UAV flight control method, device, electronic equipment, and storage medium. Background Technology

[0002] Near-space solar-powered drones (UAVs) possess advantages such as long endurance, high flight altitude, and flexible deployment, making them a key area of ​​research and development for low-speed near-space aircraft both domestically and internationally. However, given the current development level of solar cells and energy storage batteries both domestically and internationally, achieving extended endurance of several weeks or even months for solar-powered UAVs remains challenging. Achieving this goal relies on the concerted efforts of all aircraft subsystems. Among these, optimizing flight paths and attitudes to maximize peak solar energy tracking while maintaining low flight energy consumption, thereby maximizing the UAV's net energy gain, plays a crucial role in enhancing its long-endurance capability.

[0003] Current research on UAV flight control systems using machine learning commonly employs deep fully connected networks, simultaneously inputting state parameters and obtaining the final control command through fitting a large number of network parameters. This approach treats the solar-powered UAV as a whole, ignoring the inherent physical connections within the aircraft. This leads to irrelevant input information interfering with the output command, such as GPS coordinates being used as parameters for throttle position. The increased number of control surfaces and motors means an increase in the output parameters of the neural network, causing reinforcement learning methods to fall into the "curse of dimensionality," resulting in slow algorithm convergence, decreased controller performance, and difficulty in meeting the requirements for fine-grained joint control of control surfaces and motors. Furthermore, it struggles to guide the design of distributed solar-powered UAV flight control systems. The deep neural networks commonly used in reinforcement learning significantly increase the number of intermediate layers, making it increasingly difficult to study the characteristics of weights within different layers. The diversity of input / output variables and decision-making behaviors of solar-powered UAVs also complicates interpretation. In addition, current research on artificial intelligence flight controllers is mostly focused on numerical simulations, with relatively little experimental research. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for controlling the flight of a solar-powered unmanned aerial vehicle (UAV), which at least partially solves one of the aforementioned technical problems.

[0005] The first aspect of this invention provides a flight control method for a solar-powered unmanned aerial vehicle (UAV), comprising: establishing a multi-level intelligent agent architecture based on the information transmission relationship between the entire system, subsystems, and actuators of the solar-powered UAV; wherein the multi-level intelligent agent architecture includes three intelligent agent levels, each intelligent agent level containing multiple intelligent agents, and the intelligent agent level having a one-to-one physical mapping relationship with the entire system, all subsystems, and all actuators of the solar-powered UAV; training the multi-level intelligent agent architecture to obtain a distributed neural network control structure and its parameters, wherein the distributed neural network control structure includes three neural network levels, each neural network level including multiple neural networks established on an intelligent agent basis; making decisions based on intelligent agent information through the neural network, and outputting control commands to the entire system, subsystems, or actuators of the solar-powered UAV that have a physical mapping relationship with the intelligent agents corresponding to the neural networks, so as to control the entire system, subsystems, or actuators of the solar-powered UAV to perform actions.

[0006] Optionally, the three agent levels are a top-level agent, a secondary agent, and a bottom-level agent. The top-level agent is the physical mapping of the entire system, responsible for filtering and organizing global information and transmitting the organized information to the corresponding secondary agents. The secondary agents are the physical mappings of subsystems, responsible for further filtering and organizing the information transmitted by the top-level agent and transmitting the organized information to the corresponding bottom-level agents. The bottom-level agents are the physical mappings of components, responsible for receiving information transmitted by the secondary agents and using this information to make control commands for specific control surfaces and actuators such as motors.

[0007] Optionally, the three neural network layers are a top-level neural network layer, a secondary neural network layer, and a bottom-level neural network layer; the decision-making based on agent information through the neural network includes: the bottom-level neural network fuses the output results of the secondary agent and the information of multiple agents at the current level, and makes a decision based on the information fusion result, outputting control commands to the solar-powered drone component that has a physical mapping relationship with the bottom-level agent; wherein, the agent information in the bottom-level neural network is the information of the actuating component corresponding to the bottom-level neural network.

[0008] Optionally, making decisions based on agent information through a neural network further includes: a secondary neural network fusing the output of the top-level agent and the information of multiple agents at the current level, and making decisions based on the information fusion results, outputting control commands to a solar-powered UAV subsystem that has a physical mapping relationship with the secondary agent; wherein, the agent information in the secondary neural network is the information of the subsystem corresponding to the secondary neural network.

[0009] Optionally, making decisions based on agent information through a neural network further includes: a top-level neural network fusing information from multiple agents at all levels and making decisions based on the global information fusion results, and outputting control commands to the entire solar-powered UAV system that has a physical mapping relationship with the top-level agent; wherein, the agent information in the top-level neural network is the information of the entire system corresponding to the top-level neural network.

[0010] Optionally, training the multi-level agent architecture to obtain a distributed neural network control structure also includes: training the agent through reinforcement learning based on the distributed neural network control structure to maximize decision-making efficiency.

[0011] Optionally, the reinforcement learning training process includes: modeling and training the control system between agents of different levels through Markov game process, so that the agents can make action decisions that maximize their own benefits while ensuring the overall benefit of the multi-level agent architecture is maximized; and conducting repeated game training between agents of the same level to avoid malicious competition among multiple agents in the process of maximizing their own benefits, so that agents can effectively ensure the overall benefit of the same level agent architecture while maximizing their own benefits.

[0012] A second aspect of the present invention provides a flight control device for a solar-powered unmanned aerial vehicle (UAV), comprising: a construction module for establishing a multi-level intelligent agent architecture based on the information transmission relationships between the entire system, subsystems, and actuators of the solar-powered UAV; wherein the multi-level intelligent agent architecture includes three intelligent agent levels, each intelligent agent level containing multiple intelligent agents, and the intelligent agent level having a one-to-one physical mapping relationship with the entire system, all subsystems, and all actuators of the solar-powered UAV; a training module for training the multi-level intelligent agent architecture to obtain a distributed neural network control structure, wherein the distributed neural network control structure includes three neural network levels, each neural network level including multiple neural networks established on an intelligent agent basis; and a decision module for making decisions based on intelligent agent information through the neural network, and outputting control commands to the entire system, subsystems, or actuators of the solar-powered UAV that have a physical mapping relationship with the intelligent agents corresponding to the neural networks, so as to control the entire system, subsystems, or actuators of the solar-powered UAV to perform actions.

[0013] A third aspect of the present invention provides an electronic device, characterized in that it comprises: one or more processors of the same or different kinds; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described in any of the preceding claims.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium, characterized in that it stores executable instructions thereon, which, when executed by a processor, cause the processor to perform the method described in any of the preceding claims.

[0015] In this embodiment of the invention, starting from the information transmission relationship within the solar-powered UAV itself, a neural network control structure is established through physical mapping relationships. An optimization method based on game theory is introduced to generate control commands for specific components, forming a joint control strategy for distributed components. This solves the problem in existing technologies where the reinforcement learning in solar-powered UAV flight control systems has a large number of intermediate layers, making it difficult to study the characteristics of weights within different layers. This improves the decision-making ability and endurance of the solar-powered UAV.

[0016] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0018] Figure 1 A flowchart illustrating the flight control method for a solar-powered unmanned aerial vehicle (UAV) in an embodiment of the present invention is shown schematically.

[0019] Figure 2A The diagram schematically illustrates the structure of a physical model of a solar-powered unmanned aerial vehicle (UAV) in an embodiment of the present invention.

[0020] Figure 2B This illustration schematically shows a structure diagram of a distributed neural network control structure based on a multi-level intelligent agent architecture in an embodiment of the present invention.

[0021] Figure 3 This illustration shows a schematic diagram of the refined joint control technology route of solar-powered UAV control surfaces and motors based on multi-agent cooperative game theory in an embodiment of the present invention.

[0022] Figure 4 A block diagram of a solar-powered drone flight control device in an embodiment of the present invention is shown schematically.

[0023] Figure 5 A block diagram of an electronic device according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0027] In the description of this invention, it should be understood that the terms "longitudinal", "length", "circumferential", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the subsystem or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0028] Throughout the accompanying drawings, identical elements are represented by the same or similar reference numerals. Conventional structures or configurations may be omitted where they might cause confusion in understanding the invention. Furthermore, the shapes, sizes, and positional relationships of the components in the drawings do not reflect actual size, scale, or actual positional relationships. Additionally, any reference numerals placed between parentheses in the claims should not be construed as limiting the claims.

[0029] Similarly, to simplify the invention and aid in understanding one or more of the various disclosed aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. The use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0031] The purpose of this invention is to provide a flight control method for solar-powered unmanned aerial vehicles (UAVs), which effectively solves the problem of the large number of intermediate layers in reinforcement learning in current solar-powered UAV flight control systems, thereby improving the decision-making ability and endurance of solar-powered UAVs.

[0032] The present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0033] Figure 1 The flowchart illustrating the flight control method for a solar-powered unmanned aerial vehicle (UAV) in an embodiment of the present invention is shown schematically.

[0034] like Figure 1 As shown, the flight control method for this solar-powered drone may include operations S110 to S130, for example.

[0035] The S110 is operated to establish a multi-level intelligent agent architecture based on the information transmission relationship between the whole system, subsystems and actuators in the solar-powered UAV.

[0036] The multi-level intelligent agent architecture includes three intelligent agent levels, each containing multiple intelligent agents. The intelligent agent level has a one-to-one physical mapping relationship with the entire solar-powered drone system, all subsystems, and all actuating components.

[0037] Operate S120 to train the multi-level intelligent agent architecture and obtain the distributed neural network control structure and its parameters.

[0038] The distributed neural network control structure includes three neural network layers, and each neural network layer includes multiple neural networks established by agents.

[0039] Operation S130 makes decisions based on agent information through a neural network and outputs control commands to the solar-powered drone system, subsystem, or component that has a physical mapping relationship with the agent corresponding to the neural network, so as to control the solar-powered drone system, subsystem, or component to perform actions.

[0040] Currently, solar-powered drones generally possess distributed control and propulsion characteristics, enabling precise joint control of each control surface and motor to fully utilize their capabilities. This not only enhances solar energy acquisition through rational attitude control but also reduces unnecessary maneuvers and lowers flight energy consumption. Addressing the coupling issues of aerodynamics, control, and energy, this application's technical solution introduces a multi-level, multi-agent cooperative game approach. By constructing a distributed neural network control structure based on physical mapping relationships through the information transmission relationships between the entire solar-powered drone system, subsystems, and components, and incorporating random and repeated game processes, a precise joint control strategy for the solar-powered drone's control surfaces and motors is established with the goal of maximizing net energy gain.

[0041] In this embodiment, the three agent levels of the multi-level agent architecture are a top-level agent, a secondary agent, and a bottom-level agent. The top-level agent is the physical mapping of the entire system, responsible for filtering and organizing global information and transmitting the organized information to the corresponding secondary agents. The secondary agents are the physical mappings of subsystems, responsible for further filtering and organizing the information transmitted by the top-level agent and transmitting the organized information to the corresponding bottom-level agents. The bottom-level agents are the physical mappings of the components, responsible for receiving the information transmitted by the secondary agents and using this information to issue control commands for specific control surfaces and actuators such as motors.

[0042] Figure 2A The diagram illustrates the physical model structure of the UAV in an embodiment of the present invention, as follows: Figure 2A As shown, the physical model of this UAV includes a three-level structure: the overall system, subsystems, and components. Subsystems include a servo system, a propulsion system, and an energy system. The servo system consists of multiple control surfaces, the propulsion system consists of multiple motors, and the energy system parameters include battery capacity and system efficiency. Ellipses in the figure indicate the omission of similar components from the aircraft. This embodiment elucidates the information transmission relationships between the overall system, subsystems, and components of the solar-powered UAV, and constructs a corresponding multi-level intelligent agent architecture through physical mapping relationships.

[0043] The flight control system, as the central control unit of the entire aircraft, has a physical mapping relationship with the top-level intelligent agent. The top-level agent is responsible for filtering and organizing global information and transmitting the organized information to the corresponding secondary intelligent agents. The propulsion system, servo system, and energy system, as subsystems, each have a one-to-one physical mapping relationship with a secondary intelligent agent. The secondary intelligent agents are responsible for further filtering and organizing the information transmitted by the top-level agent and transmitting the organized information to the corresponding lower-level intelligent agents. The components of the aircraft subsystems each have a one-to-one physical mapping relationship with a lower-level intelligent agent. The lower-level intelligent agents are responsible for receiving information transmitted by the secondary intelligent agents and using this information to output control commands for specific control surfaces and motors, etc. The components of the aircraft subsystems include control surfaces and motors with distributed functions.

[0044] In this embodiment, the distributed neural network control structure consists of three neural network layers: a top-level neural network layer, a secondary neural network layer, and a bottom-level neural network layer. Each neural network layer includes multiple neural networks established on an agent-by-agent basis. Specifically, the top-level neural network layer is composed of a top-level neural network, the secondary neural network layer is composed of multiple secondary neural networks corresponding to different subsystems, and the bottom-level neural network layer is composed of multiple bottom-level neural networks corresponding to different components.

[0045] In this embodiment, multiple neural networks in the distributed neural network control structure have information transmission relationships, such as information transmission between different neural networks within the same neural network layer and information transmission between different neural network layers. Specifically, this includes:

[0046] The lower-level neural network fuses the output of the secondary agent with information from multiple agents at the current level, makes decisions based on the fusion results, and outputs control commands to the solar-powered drone components that have a physical mapping relationship with the lower-level agent, thereby achieving precise control of the solar-powered drone components. The agent information in the lower-level neural network refers to the information of the actuators corresponding to that lower-level neural network.

[0047] The secondary neural network fuses the output of the top-level agent with information from multiple agents at the current level, makes decisions based on the information fusion results, and outputs control commands to the solar-powered UAV subsystem that has a physical mapping relationship with the secondary agent; wherein, the agent information in the secondary neural network is the information of the subsystem corresponding to that secondary neural network.

[0048] The top-level neural network fuses information from multiple agents at all levels and makes decisions based on the global information fusion results, outputting control commands to the entire solar-powered UAV system that has a physical mapping relationship with the top-level agent; where the agent information in the top-level neural network is the information of the entire system corresponding to the top-level neural network.

[0049] Figure 2B The diagram illustrates the structure of a distributed neural network control structure based on a multi-level agent architecture in an embodiment of the present invention. This distributed neural network control structure includes a top-level neural network layer, secondary neural network layers, and a bottom-level neural network layer. In this invention, the neural network control structure is based on a multi-level agent architecture, constructing different neural network layers for different agent levels. Each neural network layer contains multiple neural networks, and information transmission relationships exist between these multiple neural networks, such as information transmission between different neural networks within the same neural network level and information transmission between different neural network levels. Figure 2B As can be seen, embodiments of the present invention can feed back the information fusion results of the lower-layer neural network to the upper-layer neural network, and further process these information fusion results to achieve information transmission between different neural network layers. The distributed neural network control structure provided by embodiments of the present invention establishes corresponding neural network layers according to the agent layer, and constructs multiple neural networks for multiple agents at the same agent layer. The structure is clear, which facilitates weight calculation, and establishes information transmission relationships between each neural network, realizing information exchange between agents in different neural networks. While maintaining a clear structure, it effectively ensures the comprehensiveness of information, making the generated flight control commands more in line with actual needs.

[0050] The significance of this key technology lies in eliminating interference from invalid information to actuators such as control surfaces and motors from the design stage, improving training speed and fault tolerance, and providing network structure input for the next step of training the flight controller. Through game theory among multiple agents, a low-level strategy is generated and combined with long-term planning to achieve flight strategy generation over long flight time spans.

[0051] In this embodiment, the neural network makes decisions based on agent information, including: training the agent through reinforcement learning based on a distributed neural network control structure to maximize decision-making efficiency and effectively improve the endurance of the solar-powered drone.

[0052] Multi-agent systems can learn action strategies and achieve superior performance through reinforcement learning based on rules and environmental feedback. However, many applications of artificial intelligence involve large-scale environments and complex learning tasks. These applications often place higher demands on agents, requiring not only individual agents to make wise actions but also multiple agents to collaboratively make informed decisions through limited communication. Therefore, effective reinforcement learning strategies need to be developed for each agent in multi-agent scenarios. Multi-agent reinforcement learning typically involves three basic elements: agent, policy, and utility function. Agents are a group of individuals with autonomous decision-making awareness, each capable of independently interacting with the environment. To maximize their gains and minimize their losses, each agent develops its own action strategy based on observation and learning of the actions of other agents. A policy, in multi-agent reinforcement learning, refers to the strategy each agent develops to maximize its gains and minimize its losses. The formulation of an agent's policy is based on its perception of the environment, and the agent is influenced by the policies of other agents during the policy-making process. Considering the individual needs of each agent and its dependencies on the environment and other agents, each agent has its own utility function. Typically, the utility function is the difference between the total benefit gained and the total cost incurred by an agent in achieving various goals. In multi-agent scenarios, during the learning process from the surrounding environment and other agents, each agent aims to maximize its own utility function as its ultimate goal.

[0053] In this embodiment, the process of training multiple agents through reinforcement learning includes training the agents using game theory. Specifically, this involves conducting Markov game training among agents at different levels using utility functions until a Stackelberg equilibrium is reached, at which point the game is stopped. Repeated game training is then conducted among agents at the same level until a Nash equilibrium is reached, at which point the game is stopped.

[0054] In multi-agent reinforcement learning, each agent has its own utility function and aims to maximize its own utility value by autonomously learning and formulating strategies based on observation and interaction with the environment. Since each agent does not consider the impact of its strategy on the utility functions of other agents during autonomous learning, competition or cooperation can occur due to the interaction between multiple agents. Considering the various complexities of agent interactions, game theory is commonly used to analyze agent decisions. Different game theory frameworks can be used to simulate interaction scenarios for different multi-agent reinforcement learning scenarios, broadly categorized into three types: static games, repeated games, and Markov games. Static games are the most basic form of simulating agent interaction. In static games, all agents make decisions simultaneously, and each agent makes only one decision action. Repeated games involve multiple agents making the same decision action multiple times in the same state; therefore, each agent's total utility function is the sum of the utility value derived from each decision action. Compared to static games, repeated games significantly avoid vicious competition among multiple agents, thereby increasing the overall sum of the total utility value of all agents. A Markov game can be viewed as a Markov process in which multiple agents make action decisions multiple times under various states. Markov games simulate the general situation of multiple agents making multiple decisions; each agent, based on its own state, observes the environment and predicts the actions of other agents to make the optimal action decision that improves its own utility function. This invention conducts Markov game training among agents at different levels and repeated game training among agents at the same level. By combining Markov games and repeated games, it effectively avoids malicious competition among multiple agents in the process of maximizing their own utility, ensuring that while maximizing their own utility, agents also maximize the overall utility of the multi-level agent architecture. This invention embeds the concept of multi-agent game cooperation into an artificial intelligence flight controller, allowing each system and component to have complete control capabilities while simultaneously cooperating under overall constraints.

[0055] By establishing a distributed neural network control structure for reinforcement learning training of multiple agents, each agent generates its own action strategy to optimize its own energy consumption or help improve the overall solar energy acquisition of the aircraft. By constructing a reasonable utility function, a Markov game process is conducted among agents at different levels, aiming to reach a Stackelberg equilibrium. Repeated game processes are conducted among agents at the same level, aiming to reach a Nash equilibrium depending on whether these agents are related. Based on the collection of actions of lower-level agents by higher-level agents, a layer-by-layer information fusion method is studied to generate final control commands in the flight control system. This achieves precise "end-to-end" generation from environment and self-state to control commands, forming a multi-agent cooperative game-based flight control command generation technology to generate underlying strategies and realize refined joint control of the solar-powered UAV's control surfaces and motors.

[0056] Figure 3 This illustration shows a schematic diagram of the refined joint control technology route for the control surfaces and motors of a solar-powered UAV based on multi-agent cooperative game theory in an embodiment of the present invention. Figure 3 As shown in the figure, the top-level strategy includes a hierarchical reinforcement learning principle based on the option framework, where "option" refers to state classification. A training environment is constructed, including annual solar radiation energy, the UAV's remaining energy, and six-degree-of-freedom flight dynamics equations, i.e., training environment 1, training environment 2... training environment n in the figure. Reinforcement learning techniques are used to train the UAV to autonomously identify "options," decomposing the large-scale spatiotemporal flight of the solar-powered UAV into a combination of several "options" in time sequence, and then using the sub-policies corresponding to each "option" to guide flight. Combining the cluster centers and sum-variance (SSE) of each "option," uncertainty is injected into the training environment through domain randomization to ensure the solar-powered UAV's ability to cope with certain environmental shifts. At the interface between two "options," a method is studied to construct a transition function using error coefficients to ensure a smooth transition between different policies. Through flexible classification of the solar-powered UAV's flight parameters and environmental parameters, together with reinforcement learning based on the option framework principle, a mechanism for generating adaptive flight strategies for solar-powered UAVs over large spatiotemporal spans is formed. High-level long-term planning is achieved through hierarchical reinforcement learning techniques. At a high level, the drone is trained to autonomously identify "options," while at a low level, it is trained to form sub-strategy clusters. The spatiotemporal span is decomposed into temporal combinations of "options," and the overall flight strategy is composed of sub-strategies corresponding to the "options" (extracted from the sub-strategy clusters). Error coefficients are used to ensure a smooth transition between different strategies, forming a top-level strategy selection technology for autonomous flight of solar-powered drones based on the option framework principle.

[0057] This invention establishes a multi-level intelligent agent architecture based on the information transmission relationships between the overall system, subsystems, and components of a solar-powered unmanned aerial vehicle (UAV). Based on this architecture, a distributed neural network control structure is established to generate flight control commands, thereby achieving flight control of the UAV. Starting with the information transmission relationships among the various subsystems of the solar-powered UAV, this invention establishes a neural network control structure through physical mapping relationships. Under energy constraints, it introduces an optimization method based on game theory to generate control commands for specific control surfaces and motors, forming a joint control strategy for distributed control surfaces and motors. This solves the problem in existing technologies where the reinforcement learning in solar-powered UAV flight control systems has a large number of intermediate layers, making it difficult to study the characteristics of weights within different layers. This effectively improves the decision-making ability and endurance of solar-powered UAVs.

[0058] This invention also provides a solar-powered unmanned aerial vehicle (UAV) flight control device, as described below. Figure 4 The solar-powered drone flight control device according to an embodiment of the present invention will be described.

[0059] Figure 4 A block diagram of a solar-powered drone flight control device is shown schematically in an embodiment of the present invention.

[0060] like Figure 4 As shown, the solar-powered unmanned aerial vehicle (UAV) flight control device 400 includes a construction module 410, a training module 420, and a decision module 430.

[0061] Module 410 is constructed, for example, by executing the above reference. Figure 1 The described operation S110 is used to establish a multi-level intelligent agent architecture based on the information transmission relationship between the whole system, subsystems and actuators of the solar-powered UAV; wherein, the multi-level intelligent agent architecture includes three intelligent agent levels, each containing multiple intelligent agents, and the intelligent agent level has a one-to-one physical mapping relationship with the whole system, all subsystems and all actuators of the solar-powered UAV.

[0062] Training module 420, for example, executing the above reference. Figure 1 The described operation S120 is used to train a multi-level agent architecture to obtain a distributed neural network control structure and its parameters. The distributed neural network control structure includes three neural network layers, and each layer includes multiple neural networks established on an agent-by-agent basis.

[0063] Decision module 430, for example, executing the above reference Figure 1The described operation S130 is used to make decisions based on agent information through a neural network, and output control commands to the solar-powered drone system, subsystem, or component that has a physical mapping relationship with the agent corresponding to the neural network, so as to control the solar-powered drone system, subsystem, or component to perform actions.

[0064] Any one or more of the modules or units according to the embodiments of this disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules or units according to the embodiments of this disclosure can be implemented by splitting into multiple modules. Any one or more of the modules or units according to the embodiments of this disclosure can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-chips, substrate systems, packaging systems, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner of integrating or packaging circuits, or implemented by any one of software, hardware, and firmware, or by a suitable combination of any of these. Alternatively, one or more of the modules or units according to the embodiments of this disclosure can be at least partially implemented as computer program modules, which can perform corresponding functions when the computer program module is run.

[0065] For example, any plurality of the construction module 410, training module 420, and decision module 430 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the construction module 410, training module 420, and decision module 430 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a substrate system, a packaged system, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the construction module 410, training module 420, and decision module 430 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0066] This invention also provides an electronic device, which is described below with reference to... Figure 5 An electronic device according to an embodiment of the present invention will be described.

[0067] Figure 5 A block diagram of an electronic device according to an embodiment of the present invention is shown schematically. For example... Figure 5 As shown, the electronic device 500 includes a processor 510 and a computer-readable storage medium 520. The electronic device 500 can perform the functions described above. Figure 1 The method described is for flight control of solar-powered drones.

[0068] Specifically, processor 510 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 510 may also include onboard memory for caching purposes. Processor 510 may be used for executing reference... Figure 1 The method flow described according to embodiments of this disclosure refers to a single processing unit or multiple processing units performing different actions.

[0069] Computer-readable storage medium 520 can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, readable storage media can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. Specific examples of readable storage media include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); memories such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0070] Computer-readable storage medium 520 may include computer program 521, which may include code / computer-executable instructions that, when executed by processor 510, cause processor 510 to perform, for example, the above-described combination. Figure 1 The described method and any variations thereof.

[0071] Computer program 521 can be configured to have computer program code, for example, including computer program modules. For example, in an exemplary embodiment, the code in computer program 521 may include one or more program modules, such as module 521A, module 521B, ... It should be noted that the division and number of modules are not fixed. Those skilled in the art can use appropriate program modules or combinations of program modules according to the actual situation. When these combinations of program modules are executed by processor 510, the processor 510 can perform, for example, the above-described combinations... Figure 1 The described method and any variations thereof.

[0072] According to embodiments of this disclosure, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0073] The specific embodiments described above provide a more detailed explanation of the technical solution of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A solar-powered unmanned aerial vehicle flight control method, comprising: establishing a multi-level agent architecture based on information transmission relationships between a full-machine system, subsystems, and actuating components in a solar-powered unmanned aerial vehicle; wherein the multi-level agent architecture comprises three agent levels, each agent level comprising a plurality of agents, and the agent levels and the full-machine system, all subsystems, and all actuating components in the solar-powered unmanned aerial vehicle have a one-to-one physical mapping relationship; training the multi-level agent architecture to obtain a distributed neural network control structure and parameters thereof, wherein the distributed neural network control structure comprises three neural network levels, and each neural network level comprises a plurality of neural networks established in units of agents; making decisions based on agent information through the neural networks and outputting control instructions to the full-machine system, subsystems, or actuating components of the solar-powered unmanned aerial vehicle corresponding to the neural networks in a physical mapping relationship to control the full-machine system, subsystems, or actuating components of the solar-powered unmanned aerial vehicle to perform actions; wherein the plurality of neural networks in the distributed neural network control structure have information transmission relationships, and the training of the multi-level agent architecture to obtain a distributed neural network control structure further comprises: based on the distributed neural network control structure, performing reinforcement learning training on the agents to maximize the decision benefit; the reinforcement learning training process comprises: modeling and performing control system training between agents at different levels through a Markov game process to make action decisions that maximize their own benefits while ensuring that the total benefit of the multi-level agent architecture is maximized; developing repeated game training between agents at the same level to avoid malicious competition among multi-agents in the process of maximizing their own benefits, so that the agents maximize their own benefits while effectively ensuring that the total benefit of the multi-level agent architecture is maximized; based on the collection of actions of lower-level agents by upper-level agents, researching agent layer-by-layer information fusion methods to generate final control instructions in the flight control system.

2. The method of claim 1, wherein, The three agent levels are top-level agents, secondary-level agents, and bottom-level agents; wherein the top-level agents are physical mappings of the full-machine system, responsible for filtering and arranging global information, and transmitting the arranged information to corresponding secondary-level agents; the secondary-level agents are physical mappings of the subsystems, responsible for further filtering and arranging the information transmitted by the top-level agents, and transmitting the arranged information to corresponding bottom-level agents; the bottom-level agents are physical mappings of the actuating components, responsible for receiving information transmitted by the secondary-level agents and making control instructions for specific rudder surfaces and motor actuating components using the information.

3. The method of claim 2, wherein, The three neural network levels are top-level neural network levels, secondary-level neural network levels, and bottom-level neural network levels; and the making decisions based on agent information through the neural networks comprises: The bottom-level neural network fuses the secondary agent output result and the plurality of agent information of the current level, and makes a decision according to the information fusion result, and outputs a control instruction to the actuating component in a physical mapping relationship with the bottom-level agent; wherein the agent information in the bottom-level neural network is the information of the actuating component corresponding to the bottom-level neural network.

4. The method of claim 3, wherein, The decision making based on the agent information by the neural network further includes: The secondary neural network fuses the top-level agent output result and the plurality of agent information of the current level, and makes a decision according to the information fusion result, and outputs a control instruction to the solar unmanned aerial vehicle subsystem in a physical mapping relationship with the secondary agent; wherein the agent information in the secondary neural network is the information of the subsystem corresponding to the secondary neural network.

5. The method of claim 4, wherein, The decision making based on the agent information by the neural network further includes: The top-level neural network fuses the plurality of agent information of all levels, and makes a decision according to the global information fusion result, and outputs a control instruction to the solar unmanned aerial vehicle whole machine system in a physical mapping relationship with the top-level agent; wherein the agent information in the top-level neural network is the information of the whole machine system corresponding to the top-level neural network.

6. A solar unmanned aerial vehicle flight control device, comprising: A construction module for establishing a multi-level agent architecture based on the information transmission relationship between the whole machine system, the subsystem and the actuating component in the solar unmanned aerial vehicle; wherein the multi-level agent architecture includes three agent levels, each agent level includes a plurality of agents, and the agent level and the whole machine system, all subsystems and all actuating components in the solar unmanned aerial vehicle have a one-to-one physical mapping relationship; A training module for training the multi-level agent architecture to obtain a distributed neural network control structure and its parameters, wherein the distributed neural network control structure includes three neural network levels, and each neural network level includes a plurality of neural networks established in units of agents; A decision module for making a decision based on agent information by the neural network, and outputting a control instruction to the whole machine system or subsystem or actuating component of the solar unmanned aerial vehicle corresponding to the agent in a physical mapping relationship with the neural network, so as to control the whole machine system or subsystem or actuating component of the solar unmanned aerial vehicle to perform an action; The information transmission relationship exists between the plurality of neural networks in the distributed neural network control structure, the multi-level intelligent agent architecture is trained to obtain the distributed neural network control structure, and the training of the multi-level intelligent agent architecture based on the distributed neural network control structure includes: reinforcement learning training of the intelligent agent based on the distributed neural network control structure, so as to maximize the decision benefit; the reinforcement learning training process includes: modeling and controlling the system through a Markov game process between intelligent agents at different levels, so that the intelligent agents make action decisions to maximize their own benefits while ensuring that the total benefit of the multi-level intelligent agent architecture is maximized; repeated game training is carried out between intelligent agents at the same level to avoid malicious competition of multi-intelligent agents in the process of maximizing their own benefits, so that the intelligent agents maximize their own benefits while effectively ensuring that the total benefit of the multi-level intelligent agent architecture is maximized; based on the collection of the actions of the intelligent agent at the next level, the intelligent agent layer-by-layer information fusion method is researched to generate a final control instruction in the flight control system.

7. An electronic device, comprising: The method comprises: one or more processors of the same or different types; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer program product has executable instructions stored thereon, and the executable instructions are executed by the processor to implement the method of any one of claims 1 to 5.

Images