Unmanned aerial vehicle cluster digital twinning system and method based on biological crowd learning

Abstract

The invention discloses an unmanned aerial vehicle cluster digital twinning system based on biological crowd learning and a method thereof, the system architecture comprises a resource layer, a simulation layer, a communication layer and an application layer, the resource layer is responsible for managing software platform resources, the simulation layer is a platform core and supports a simulation process of a virtual model, and the communication layer is responsible for communicating with the application layer. The communication layer establishes communication links between various entities and models connected with the platform software, the application layer manages application services based on digital twinning and provides a user interface, and the method comprises the steps of building an unmanned aerial vehicle twinning model, installing a communication interface, updating model parameters by using Bayesian reasoning and an optimization algorithm, and establishing an unmanned aerial vehicle twinning model. And supporting a training task of the unmanned aerial vehicle cluster based on crowd-sourcing learning and deep reinforcement learning by using a twin model and simulation. According to the method, the function requirements of unmanned aerial vehicle cluster monitoring, prediction, updating and learning training can be comprehensively met, the model parameter updating efficiency and accuracy are improved, and the intelligent level and operation efficiency of unmanned aerial vehicle cluster task execution are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a drone cluster digital twin system and method based on biological swarm intelligence learning, belonging to the field of drone digital twins. Background Art

[0002] As a cutting-edge integrated technology, the core of digital twin technology is to create a virtual mapping of physical entities and achieve seamless connection between the physical world and the digital world. This technology simulates, analyzes, predicts and optimizes the state of physical entities through real-time feedback of data, and thus plays an important role in many fields. Digital twin technology can be widely used in aerospace, industrial manufacturing, medical services, transportation and other fields. In these fields, digital twins not only improve product quality and production efficiency, but also ensure safety and save resources. As an important technical support for Industry 4.0, digital twin technology provides a powerful tool for analysis and decision-making of complex systems through its high-precision simulation and prediction capabilities.

[0003] The characteristics of digital twins can be summarized from five dimensions: model, data, connection, service / function, and physics. The model dimension requires digital twin models to have high fidelity, high reliability, and high precision. The data dimension emphasizes the comprehensiveness of data and the ability to update in real time and dynamically. The connection dimension highlights the seamless connection between the physical world and the virtual world. The service / function dimension covers a series of application services such as product design, operation monitoring, and energy consumption optimization. The physical dimension reflects the inseparable relationship between digital twins and physical entities. Based on these characteristics, a recognized digital twin system model architecture is the digital twin five-dimensional model architecture, which provides a comprehensive framework for digital twin technology. This architecture emphasizes the importance of physical entities, virtual models, data connections, service functions, and information-physical interactions. As the basis of digital twins, physical entities collect data through sensors and measurement technologies to provide original information for virtual models. Virtual models use advanced simulation technology to create accurate digital representations of physical entities. Data, as a link between the physical world and the virtual world, ensures that virtual models can reflect the status and changes of physical entities in real time.

[0004] At present, research on digital twin technology for drone swarms, especially the local decision-making of individual behavior and centralized evaluation of group behavior in multi-agent scenarios, and how to improve the intelligence level and operational efficiency of drone swarm mission execution, need to be urgently addressed. Summary of the invention

[0005] 1. Purpose of the invention:

[0006] This paper provides a UAV cluster digital twin system and method based on biological swarm intelligence learning. The platform aims to create a digital twin software platform for unmanned clusters. Through precise digital twin technology, it can realize comprehensive monitoring, intelligent prediction, real-time update and efficient learning and training of unmanned single equipment and clusters when performing typical tasks. The present invention provides a comprehensive solution to enhance the intelligence level of unmanned clusters, optimize task execution strategies, and improve overall operating efficiency and reliability.

[0007] 2. Technical solution:

[0008] Aiming at the typical mission execution scenarios of unmanned swarms, this paper takes unmanned single equipment and swarms as the research objects and establishes a UAV swarm digital twin system based on biological swarm intelligence learning. The system architecture is as follows Figure 1 As shown:

[0009] A drone swarm digital twin system based on biological swarm intelligence learning is divided into four levels, namely resource layer, simulation layer, communication layer and application layer.

[0010] (1) The resource layer is the foundation of the unmanned cluster digital twin platform and is mainly responsible for managing static software platform resources. It includes modules such as virtual model library, twin database, application algorithm library, and communication protocol library.

[0011] Virtual model library: The virtual model library includes digital twin models corresponding to physical entities. Virtual models can include physical appearance, dynamics model, task logic model, simulation interface model, etc. These models are usually stored in a modular form, including configurable model parameters, and each virtual model has a unique physical entity as a mapping.

[0012] Twin database: The twin database is responsible for storing various necessary data generated during the operation of the platform, including various data generated in the digital twin platform, such as direct simulation data of physical entities and virtual models, virtual-real communication data between modules, log data, etc.

[0013] Application algorithm library: The application algorithm library includes packaged specific algorithm models, such as optimization algorithms, machine learning algorithms, etc. Some applications in the digital twin platform software involve the call of different algorithms according to different scenarios. For example, when using the twin environment for deep reinforcement learning training, the algorithm with encapsulated input and output interfaces can be called as needed to achieve more efficient learning for different scenarios and different objects.

[0014] Communication protocol library: The communication protocol is responsible for storing various communication protocol formats. Different communication protocols and interface definitions are required between physical entities, virtual entities, and virtual and real individuals in the digital twin platform, such as wireless data transmission networking communication based on serial port protocol, or data exchange between virtual models based on TCP / UDP.

[0015] The virtual model is used by users to perform basic operations on the virtual model, including:

[0016] Model import: The model import function can import platform virtual models in a specific format. These models are composed of multiple modular component models (such as controllers, motors or batteries, etc.) and then used as simulation instances in a twin world.

[0017] Platform model creation: Platform model creation is used to organize modular component models into a platform instance model. In this link, the key parameters in the module will be determined, and the platform model will become a mapping of a specific physical object.

[0018] Model definition: Digital twin models can include models from multiple fields, such as mechanical models, electrical models, thermal models, etc., and can also include multi-dimensional models, such as physical models, three-dimensional models, behavioral models, etc. [2].

[0019] Model storage: The same type of physical entity components corresponds to a fixed virtual model template, and different individuals modify key parameters based on this template. Therefore, the storage of the model can be simplified to the storage of types and key parameters. In the digital twin platform software, the model storage stores the address and key parameter list of the model template based on the XML format.

[0020] The last part of the resource layer involves the processing of all key data involved in the operation of the digital twin platform software. It includes the following modules:

[0021] Historical data import: The data import module enables the digital twin platform software to import various types of data generated by the user before, such as simulation data, actual installation data, log data, etc., and is compatible with common mainstream data formats such as csv, xml or txt.

[0022] Data filtering: Data filtering allows users to search and view data based on its fields, and repackage and save the data of interest to facilitate subsequent processing and support top-level complex applications such as feature extraction or machine learning.

[0023] Data processing: Data processing mainly pre-processes specific types of data such as sensor data, including outlier processing, noise smoothing, and duplicate data removal.

[0024] Data Reinjection: Supports output data generated from simulation models and physical implementations to be used as input for further simulations, allowing top-level applications to reanalyze past simulation routines.

[0025] (2) The simulation layer is the core of the digital twin platform, focusing on supporting the simulation process of the virtual model. It consists of two parts: simulation configuration management and simulation engine. Simulation configuration management covers simulation instance configuration, algorithm parameter configuration, scenario configuration, and communication configuration, allowing users to set the simulation environment and parameters according to specific needs.

[0026] Simulation instance configuration: Simulation instance configuration is used to clarify the initialization requirements for the platform model instance initialization involved in the simulation scenario of the twin world according to prior requirements, including setting its initial position, load configuration, performance envelope, etc.

[0027] Algorithm parameter configuration: Algorithm parameter configuration is used to configure the parameters of the algorithm executed in the top-level application according to user expectations and application scenarios, such as the search weight parameters and boundary values ​​of the optimization algorithm, the learning rate, discount factor, exploration rate of the reinforcement learning algorithm, etc.

[0028] Scenario configuration: Scenario configuration involves the configuration of environmental factors of the simulation execution scenario, such as the number and size of obstacles, the configuration of virtual targets, and the materials and general models of the rendering design. It is configured according to user needs and randomly deployed within a certain range to support interaction in learning.

[0029] Communication configuration: The digital twin platform involves multiple connections. The data content of the communication between the parties is different. It is necessary to configure the field composition between the two communicating parties. Usually it includes a fixed packet header and packet body to distinguish the type and specific content of different data.

[0030] The simulation engine is built on the Isaac SIM software of the Nvidia Omniverse platform, which is an advanced robot simulation and development platform designed to provide researchers and developers with a powerful tool to design, test and optimize the behavior and performance of robots in complex environments. The platform uses Nvidia's powerful Graphical Processing Unit (GPU) and deep learning technology to provide a highly realistic virtual environment that supports users to perform large-scale parallel simulations with unprecedented speed and efficiency. The advantages of Isaac SIM include its ease of use, flexibility and scalability. It supports a variety of Robot Operating Systems (ROS) and common robot hardware interfaces, allowing existing ROS users to seamlessly migrate to the platform. At the same time, its modular design allows users to customize the simulation environment and experiments according to their needs. In addition, Isaac SIM also provides a rich API and plug-in system to facilitate the integration of third-party tools and extended functions, further enhancing its application potential in the field of robot simulation. Therefore, the ROS-based control program carried by the task machine in the ROS-based real-installation platform can be easily migrated to the Isaac SIM environment, and it provides great convenience for the real-time parallel simulation of the digital twin model.

[0031] (3) The communication layer plays a vital role in the digital twin technology framework. It establishes the communication links between various entities and models connected to the digital twin platform software. The communication layer not only includes direct serial interface communication with physical devices, but also covers wireless communication hardware such as XBEE radio modules or radio frequency data transmission, as well as network communication based on wired LAN. In addition, communication supports the transmission and storage of configuration files and log files. In order to handle the data interaction between the remote server and the on-site edge computing resources, the communication layer adopts intranet penetration technology to build a communication bridge based on the public Internet, which enhances the applicability and flexibility of the platform. In order to support isomorphism with the actual model, the twin model supports the same MAVLink protocol as the actual flight control, and communicates with the ROS control program of the mission aircraft through the interface.

[0032] (4) The application layer is the top layer in the digital twin platform architecture, dedicated to managing digital twin-based applications and services and creating a user-friendly operation interface. Relying on the support of the resource layer, simulation layer, and communication layer, the application layer builds a comprehensive access point for application services.

[0033] White-box monitoring allows users to indirectly monitor the physical drone by monitoring the virtual model, saving communication resources and achieving effective monitoring;

[0034] Model update: Correct and update model parameters through data records during mission execution to maintain consistency between the digital twin model and the physical drone;

[0035] Task prediction: using simulation environment to create a highly similar virtual environment, quickly predicting the potential results of tasks and providing decision-making references for users;

[0036] Training and learning, assisting machine learning algorithms in training, reducing conversion errors, and improving algorithm performance and efficiency. The application layer enables users to easily interact with the digital twin platform software to achieve comprehensive management and optimization of unmanned cluster tasks.

[0037] Based on a drone swarm digital twin system based on biological swarm intelligence learning, a drone swarm digital twin method of the present invention is as follows:

[0038] Step 1: Establish a digital twin model based on the physical structure of the drone. The virtual drone model can be divided into several sub-modules according to the autopilot, sensor, power, and communication categories. In each module, the model is established according to the mechanism of the actual physical entity.

[0039] First, the autopilot model is established according to the flight control law of the physical UAV object. The autopilot is an important component of the virtual UAV model and the core component that determines the flight state of the UAV. It is responsible for converting the desired control input into the control signal of the motor.

[0040] Then, the power system model is established based on the physical drone object. The power system is responsible for converting the control signal of the controller into the force that changes the motion state of the drone. At the same time, the status of each component in the system is also an important indicator of concern to users, such as the actual speed of the motor, the current of the electric control, and the voltage and remaining power of the battery. At the end of the controller modeling in the previous part, the desired speed control output signal of the motor can be obtained. In the power system modeling part, the desired speed output will be used to calculate the actual motor speed of the drone, and virtual models will be established for the brushless motor, electronic speed regulator and battery in turn.

[0041] Finally, sensor models are built based on the payload of the physical drone object. They collect information about the surrounding environment in the simulation system, enabling the drone to perceive and adapt to complex flight conditions. For physical drones, sensor modeling is a necessary component. In order to simulate the control input conditions of the drone more realistically, it is necessary to design simulation sensors according to the configuration of the physical drone and provide input for some key communication messages. In order to be close to the actual situation, necessary noise and errors should be added during modeling.

[0042] Step 2: Build a practical communication interface. In the communication logic of the practical drone, only the mission machine uses the ROS system for internal control, and the communication and control between the flight control and the mission machine use the MAVLink communication protocol, so it is necessary to increase the support for the initialization and processing of MAVLink protocol messages in the drone model. Therefore, a communication interface model of the drone model is built based on Pymavlink, so that the drone can process and send MAVLink messages, and then support external programs or software and hardware to control the virtual drone model through the MAVLink protocol. At the same time, considering that MAVROS has been widely used in ROS, in order to improve the efficiency of controlling drones through the MAVLink protocol, the communication interface model will also consider the use requirements of MAVROS's common topics and services to support the corresponding MAVLink protocol.

[0043] Step 3: Update the parameters of the digital twin model. Using Bayesian reasoning and crowd-intelligence-inspired optimization algorithms, the model data parameters are updated using data samples collected during the operation of the real drone. A digital twin is a virtual mirror of a physical entity that can provide an important reference for the behavior and state of the drone during its life cycle. This requires continuous updating of model parameters to adapt to changes caused by wear, environmental conditions, and other operational factors.

[0044] When updating parameters, multiple combinations may produce similar model outputs, and their performance parameters generally do not change suddenly. Therefore, in order to prevent large jumps in results, Bayesian inference methods are used for regular parameter updates. The maximum a posteriori estimate of the parameter θ The calculation of is as follows:

[0045]

[0046] Where L(D|θ) is the maximum likelihood obtained using a dataset D of length k under given drone model parameters θ. P(θ) represents the prior probability of θ, which is assumed to obey a Gaussian distribution. In addition, logarithmic operations are used to simplify calculations. Using the model output x| θ , and the actual state of the next state in the dataset D The negative sign is added because we want the error to be as small as possible, while the likelihood is expected to be as large as possible.

[0047] Furthermore, an optimization algorithm inspired by swarm intelligence learning is introduced to solve the Bayesian reasoning problem. First, the population position is initialized using chaotic mapping. Logistic mapping is a common method used in chaotic mapping to simulate the growth of biological populations. The initial population of the algorithm is randomly initialized in the range of [0,1], then undergoes several chaotic mappings, and finally converts to the search space range. The mathematical expression of Logistic mapping is as follows:

[0048] x n+1 =ax n (1-x n ) (3)

[0049] Among them, x n It represents the position vector of the individual in the range [0,1] before the nth chaotic mapping. In practice, in order to improve efficiency, the fitness of the original randomly initialized population is compared with the fitness of the initialized population processed by the chaotic mapping, and finally the better half is retained for algorithm iteration.

[0050] Furthermore, the Marine Predators Algorithms (MPA) algorithm is used as the basis for optimization iteration. The algorithm iteration is divided into three stages. The first stage uses the following formula:

[0051]

[0052] Among them, Prey and Elite represent the prey and predator matrices, and the predator in each round is transformed from the prey with the highest fitness. B is a normally distributed random vector representing Brownian motion, P and R are constants, usually taking values ​​of (0,1).

[0053] The second stage formula is as follows:

[0054]

[0055] Where R L is a Levy random vector, is an adaptive parameter that controls compensation for prey motion.

[0056] The third stage formula is as follows:

[0057]

[0058] Furthermore, the Trap-Avoidance Operator (TAO) is introduced to improve the effectiveness of the algorithm in dealing with real-world problems, enabling individuals to achieve significant improvements in quality. Specifically, at the end of each iteration, if the random number is less than a threshold, the TAO operation is performed on the individual. The mathematical expression of TAO update is as follows:

[0059]

[0060] Among them, λ1 and λ2 are random uniform numbers in the range of (-1,1) and (-0.5,0.5) respectively. μ1 and μ2 are values ​​of random number Δ in the range of 0 to 1. TAO uses these random conditions, taking the optimal individual and the average position as reference, to make the population more diverse, thereby escaping from the local optimal solution.

[0061] Step 4: Conduct deep reinforcement learning training based on swarm intelligence learning. Introduce swarm intelligence learning and carry out training tasks in the form of distributed control and centralized learning in multi-agent scenarios. By using the established digital model and high-precision simulation, as well as the computing resources of the remote server, this system can better support the training tasks of drone clusters based on deep reinforcement learning.

[0062] Similar to the self-organizing behavior in group intelligence learning, in a multi-agent system, intelligence is distributed among the agents, and each agent only has local information and computing power. At the same time, individuals can usually only perceive their local environment and partial states of other units in the group, and they affect global behavior through local interactions. Therefore, local observation information is established according to the mission scenario requirements and the observation capabilities of the drone.

[0063] In a group, the decision-making process of each agent is usually based on simple rules or strategies, which are continuously learned and optimized through interaction with the environment. In deep reinforcement learning tasks, the behavior of individual drones is relatively simple, such as speed and position control.

[0064] One of the goals of group intelligence learning is to generate complex collective behaviors through collaboration or competition among individuals. In deep reinforcement learning, actions are generated through a distributed policy network, while the evaluation network receives information from all agents and updates parameters. This reflects the characteristics of local decision-making for individual behavior and centralized evaluation of group behavior.

[0065] To achieve this training goal, Multi-agent proximal policy optimization (MAPPO) is used as the training algorithm for deep reinforcement learning. For drone swarms, each drone is regarded as an agent in MAPPO, and the drone swarm formation process is regarded as a Markov decision process, which can be defined as Where S, A, P and R are state space, action space, state transition probability and reward function respectively. As a distributed agent, the drone generates an action based on local observations and a shared strategy, interacts with the environment and generates rewards. Its policy network only uses local state information, while the evaluation network uses global state information. Specifically, each agent receives a local observation and outputs an action probability, and all agents use a policy network. The evaluation network of each agent receives the observations of all agents and outputs a value, which is used to update the policy network.

[0066] 3. Advantages and effects:

[0067] The present invention proposes a UAV swarm digital twin system and method based on biological swarm intelligence learning: 1. The present invention provides an innovative UAV swarm digital twin system framework, which integrates the resource layer, simulation layer, communication layer and application layer to form an efficient and reasonable workflow, which can fully meet the functional requirements of UAV swarm monitoring, prediction, updating and learning and training; 2. The present invention adopts an optimization algorithm based on swarm intelligence inspiration to update model parameters in real time, takes Bayesian reasoning as the basic framework, and solves the Bayesian reasoning problem by using chaotic mapping to initialize population positions and trap avoidance operations to improve the efficiency and accuracy of model parameter updating; 3. The present invention constructs a UAV deep reinforcement learning method that combines swarm intelligence learning and digital twin technology, and realizes distributed control and centralized learning through the multi-agent proximal policy optimization (MAPPO) algorithm, solves the problems of local decision-making of individual behavior and centralized evaluation of group behavior in multi-agent scenarios, and significantly improves the intelligence level and operation efficiency of UAV swarm task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a framework diagram of the drone swarm digital twin system and its method.

[0069] Figure 2 It is the trajectory diagram of speed control mode.

[0070] Figure 3 It is the motor simulation curve.

[0071] Figure 4 It is the ESC simulation curve.

[0072] Figure 5 It is a battery simulation curve chart.

[0073] Figure 6 It is a battery life prediction chart.

[0074] Figure 7 It is the comparison curve between the GPS sensor output position and the true value.

[0075] Figure 8 It is the comparison curve between the angular velocity output by the IMU sensor and the true value.

[0076] Fig. 9 It is a comparison chart of the fitness curve of model parameter update.

[0077] Fig.10 It is the comparison between the simulation trajectory and the real trajectory before and after the model parameters are updated.

[0078] Fig.11 This is a comparison chart of the reward curve during the policy network training process.

[0079] Fig.12 This is a comparison chart of the average reward curve per round for a single drone. DETAILED DESCRIPTION

[0080] The effectiveness of the system and method proposed in the present invention is verified through a specific drone cluster digital twin example. The specific steps of the system and method are as follows:

[0081] Step 1: Establish a digital twin model based on the physical structure of the drone.

[0082] (1) Autopilot

[0083] Taking the actual flight control system equipped with Ardupilot firmware as an example, the design of the autopilot follows its actual logical architecture.

[0084] In terms of horizontal control, the control law of the UAV virtual controller model can be described by the following formula:

[0085]

[0086] in, is the desired horizontal speed control quantity, which is firstly controlled by the position P controller using the position Pos of the drone on the horizontal plane. x,y With expected position Then, the speed PID controller uses the horizontal speed deviation to convert it into the desired roll angle φ and pitch angle θ. Then, in the attitude control loop, the PD controller is used to calculate the speed change component required to bring the aircraft to the desired attitude angle. It should be noted that the expected yaw angle The calculation of the expected roll angle and the expected pitch angle is different. The latter is obtained according to formula (12), while the former is divided into two cases: when the control mode includes yaw angle control, the expected yaw angle is directly specified by the control command of the upper task machine, otherwise

[0087]

[0088] For vertical control, the control law of the UAV virtual controller model can be described by the following formula:

[0089]

[0090]

[0091] As shown in equations (15) and (16), the position P controller is first used to set the position Pos on the Z axis to z and expectations The deviation is converted into the expected vertical speed That is, the climb rate, which is then differentiated from the level control, and the speed PID controller converts the speed deviation into the required desired longitudinal acceleration Furthermore, considering the motor speed ω in the hovering state h satisfy

[0092] 4k F ω h 2 =mg (17)

[0093] Where K F is the thrust coefficient of the motor propeller, m is the mass of the drone, and g is the acceleration due to gravity.

[0094] Combining the above horizontal and vertical control laws, considering the X-frame configuration of the drone, the speed variable causing the vertical speed variable and the speed variable causing the attitude change and the speed of the four motors satisfy the following relationship:

[0095]

[0096] Through this hierarchical control strategy, users can input the desired control amount from different positions of the control loop according to the scene in which the drone is located, thereby achieving accurate and stable control in various flight modes, further providing more flexible options for the execution of top-level tasks and improving flight efficiency and safety. Taking speed control as an example, the simulation curve is as follows Figure 2 shown.

[0097] (2) Power system

[0098] The final output of the motor modeling part is the average input voltage Um And the working current I m The expression is:

[0099]

[0100]

[0101] Among them, and R m are the working current and internal resistance of the motor respectively. The no-load speed of the motor is proportional to the no-load input voltage U m0 , this ratio is called the rated no-load motor constant K V0 , M is the torque generated by a single propeller.

[0102] For the ESC, considering the internal resistance of the battery, the input voltage U e and current I e It can be expressed as:

[0103] Ue=Ub-IbRb (21)

[0104] I e =σI m ,I e ≤I eMax (twenty two)

[0105] Among them I eMax is the maximum operating current of the ESC. Its specific value is generally provided by the ESC product manufacturer. b and R b Represent the current and internal resistance of the battery respectively. r For a multi-rotor drone with 2 motors, each motor is equipped with an ESC, and the power supply is used to power all the ESCs and onboard devices. The battery current can be expressed as:

[0106] Ib=nrIe+IFC+Icom (23)

[0107] Among them, I FC and I com The working current of the flight control device and the mission machine on the drone is respectively. For the quadcopter drone hardware platform involved in this article, the power supply specifications of the flight control device and the Raspberry Pi mission machine are both 5V1A. Therefore, I FC and I com Usually 1A is acceptable.

[0108] For the battery, users are most concerned about the current remaining battery power and the estimated battery life. The remaining battery power is generally expressed by the battery state of charge (SoC). Through experiments, for a lithium polymer (Li-Po) battery composed of 4 cells, the relationship between its open circuit voltage and SoC can be expressed by the following polynomial:

[0109]

[0110] Among them, the open circuit voltage V oc is the sum of the battery's operating voltage and the voltage drop across the battery's internal resistance, so V oc ≈U b The Soc of a battery or battery pack at a given time is the ratio of the available power at that moment to the total power available when fully charged. Expressed as a percentage, 100% when fully charged and 0% when empty. The most commonly used method of calculating Soc is the ampere-hour method:

[0111]

[0112] Among them, t0 is the initial time, C T Indicates the total battery capacity in milliampere hours (mAh). Average battery discharge current and the current battery capacity to estimate the battery life T b .

[0113]

[0114] The simulations were conducted using three configurations of the powertrain, such as Figures 3 to 6 shown.

[0115] (3) Sensor system

[0116] For the research of physical drones, the most basic and critical sensors include the Global Positioning System (GPS) and the Inertial Measurement Unit (IMU).

[0117] Taking into account the measurement error, the following relationship is satisfied between the GPS measurement value and the real value of the simulation environment:

[0118]

[0119] Among them, p is the real position in the local ENU coordinate system obtained directly from the simulation system. is the measurement value given by the GPS simulation module, and Δt represents the simulation step size. η prepresents Gaussian white noise with a mean of zero, simulated using a normally distributed random variable. p It represents a slowly changing random walk diffusion process, which is a noise process with "memory", that is, the current noise value will be affected by the previous noise value. It is worth noting that for a continuous-time random process, the variance grows linearly with time. Therefore, when simulating a continuous-time random process in a discrete-time system, it is necessary to ensure that the variance grows consistently, that is, the standard deviation is equal to the square root of the step size. Directly proportional.

[0120] IMU consists of a gyroscope and an accelerometer, which are used to measure the angular velocity and acceleration in the drone's body coordinate system. Similar to GPS, the angular velocity measurement output by the IMU simulation module and acceleration measurements With Gaussian white noise and random walk process, the expression is as follows:

[0121]

[0122] Among them, ω and a are the real angular velocity and acceleration in the body coordinate system directly obtained from the simulation system. η g and η a The subtable indicates that the mean is zero and the variance is σ ng and σ na Gaussian white noise. g and b a A random walk diffusion process representing slowly changing angular velocity and acceleration measurements.

[0123] The sensor was simulated and verified, and the output curves and true value comparison curves of GPS and IMU were obtained as follows: Figures 7 and 8 shown.

[0124] Step 2: Build the actual communication interface

[0125] In the communication logic of the actual drone, only the mission machine uses the ROS system for control, and the communication and control between the flight control and the mission machine use the MAVLink communication protocol, so it is necessary to increase the support for the initialization and processing of MAVLink protocol messages in the drone model. Therefore, the communication interface model of the drone model is built based on Pymavlink, so that the drone can process and send MAVLink messages, and then support external programs or software and hardware to control the virtual drone model through the MAVLink protocol. At the same time, considering that MAVROS has been widely used in ROS to improve the efficiency of controlling drones through the MAVLink protocol, the communication interface model will also consider the use requirements of MAVROS's common topics and services to support the corresponding MAVLink protocol.

[0126] Step 3: Update digital twin model parameters

[0127] Before updating the model, it is necessary to collect flight data of the physical drone. The data samples include the position and velocity in the North East Earth (NED) coordinate system, the attitude angle and its angular rate expressed in Euler angles, and the desired speed calculated by the mission computer as control input, which are collected at a frequency of 10Hz. This paper selects a drone that meets the above configuration to perform basic parameter identification. Assume that the baseline values ​​of the drone identification parameters are, I xx =0.0153, I yy =0.0153, I zz =0.0257, k f =1.23×10 -5 , k M =1.68×10 -7 .

[0128] A variety of classic and latest optimization algorithms are used for comparison, including: the original marine predator algorithm (MPA), pigeon optimization algorithm (PIO), Harris Hawk optimization algorithm (HHO), gray wolf optimization algorithm (GWO), golden sine algorithm (GSA), rime optimization algorithm (ROA), and the fitness curve changes are as follows: Fig. 9 The comparison of the trajectory diagram of the simulation model and the real data in the same task before and after the parameter update is shown in Fig.10 shown.

[0129] Step 4: Conduct deep reinforcement learning training based on crowd intelligence learning

[0130] Consider a drone that can obtain the state of surrounding units through sensors. Given its target position, the state vector s of the drone is defined as:

[0131] s=[g,n,o] (29)

[0132] Among them, g represents the target state information of the UAV in the surrounding direction, g i (i=0,1,…,m) represents the vector formed by the inverse of the target distance in m directions. If no target is detected in this direction, the corresponding g i = 0. Similarly, n and o represent the status information of neighboring drones and obstacles, satisfying

[0133]

[0134] Among them, d g Indicates the distance from the current position of the drone to the target position, d n and d o They represent the distances to neighboring drones and the physical boundaries of obstacles, respectively.

[0135] Considering the actual control law of UAVs, the action space of any individual UAV is defined as follows:

[0136]

[0137] Among them, h and v are the horizontal direction and speed of the drone, respectively. represents the action strategy set of UAV i, the value range of h is m directions evenly divided on the horizontal plane, and v is a discrete speed value.

[0138] For the reward function, define the formation reward function r f for

[0139]

[0140] tr[·] represents the trace of the matrix, and represent the normalized Laplace matrix of the current cluster configuration and the normalized Laplace matrix of the ideal configuration respectively.

[0141] Scale reward function r s , defined as follows:

[0142]

[0143] Among them, Φ is the neighboring drone of drone i, D des is the expected distance between UAVs.

[0144] Target reward function r a , defined as follows:

[0145] r a =D t-1 -D t (34)

[0146] Among them, D t It indicates the distance between the drone and the target at that moment, including the ideal distance and noise. t-1 Indicates the distance between the drone and the target at the last moment.

[0147] Consider three drones flying in a triangular formation with an expected interval of 5m. They take off from a certain area and fly over the target in formation. The normalized Laplace matrix of the expected formation is:

[0148]

[0149] Fig.11The weighted total reward curve for each round of a single drone is shown. The policy network is also deployed on a real drone for flight testing. Since the control law logic is taken into account during training, the policy network trained in the twin environment can be easily transferred to the real object. Fig.12 It can be seen that the test flight trajectory is basically consistent with the twin decision trajectory. It can be seen that the deep reinforcement learning decision model trained through the twin environment is also reliable in the real world.

Claims

1. A drone swarm digital twin method based on biological swarm intelligence learning, characterized by: The method includes: Step 1: Modeling based on the physical structure of the drone The virtual drone model is divided into several sub-class modules according to the autopilot class, sensor class, power class, and communication class. Models are established in each module according to the mechanism of actual physical entities. Specifically, the autopilot model is first established according to the flight control law of the physical drone object; then the power system model is established according to the physical drone object; finally, the sensor model is established according to the load of the physical drone object; Step 2: Build the actual communication interface In the communication logic of the actual drone, the mission aircraft uses the ROS system for internal control, and the communication and control between the flight control and the mission aircraft uses the MAVLink communication protocol; The communication interface model of the drone model is built based on Pymavlink, so that the drone can process and send MAVLink messages, and then support external programs or software and hardware to control the virtual drone model through the MAVLink protocol; at the same time, the communication interface model also supports the corresponding MAVLink protocol by using the common topics and service requirements of MAVROS; Step 3: Update digital twin model parameters Using Bayesian reasoning and crowd-intelligence-based optimization algorithms, we use data samples collected during the operation of real drones to update model data parameters; Step 4: Conduct deep reinforcement learning training based on crowd intelligence learning Introducing crowd intelligence learning, in multi-agent scenarios, training tasks are carried out in the form of distributed control and centralized learning; multi-agent proximal strategy optimization is used as the training algorithm for deep reinforcement learning; For UAV swarms, each UAV is regarded as an intelligent agent in MAPPO, and the UAV swarm formation process is regarded as a Markov decision process, which is defined as Among them, S, A, P and R are state space, action space, state transition probability and reward function respectively; the drone, as a distributed agent, generates an action based on local observations and a shared strategy, interacts with the environment and generates rewards; its policy network only uses local state information, while the evaluation network uses global state information; specifically, each agent receives a local observation and outputs an action probability, and all agents use a policy network; the evaluation network of each agent receives the observations of all agents and outputs a value, which is used to update the policy network.

2. The method according to claim 1, characterized in that: In the step three, the Bayesian inference method is used to perform regular parameter updates; the model output x| θ , and the actual state of the next state in the dataset D The likelihood is calculated by taking the mean squared error between ; logarithmic operations are used to simplify the calculation, and the negative sign is added because we want the error to be as small as possible, while the likelihood is expected to be as large as possible.

3. The method according to claim 2, characterized in that: An optimization algorithm inspired by swarm intelligence learning is further introduced to solve the Bayesian reasoning problem. First, the population position is initialized using chaotic mapping. The initial population of the algorithm is randomly initialized in the range of [0,1], and then undergoes several chaotic mappings before finally converting to the search space range.

4. The method according to claim 3, characterized in that: The ocean predator algorithm is further used as the basis for optimization iteration.

5. The method according to claim 4, characterized in that: A trap avoidance operation is further introduced: at the end of each round of iteration, if the random number is less than the threshold DF, a TAO operation is performed on the individual.

6. A drone swarm digital twin system based on biological swarm intelligence learning, used to implement the method described in any one of claims 1 to 5, characterized in that: The system includes resource layer, simulation layer, communication layer and application layer; The resource layer is responsible for managing static software system resources, which includes virtual model library, twin database, application algorithm library and communication protocol library modules; The simulation layer is the core of the digital twin platform and supports the simulation process of the virtual model. It consists of two parts: simulation configuration management and simulation engine. Simulation configuration management covers simulation instance configuration, algorithm parameter configuration, scenario configuration and communication configuration, allowing users to set simulation environment and parameters according to specific needs. The communication layer establishes the communication links between various entities and models connected to the digital twin system software; the communication layer not only includes direct serial interface communication with physical devices, but also covers wireless communication hardware and network communication based on wired LAN; in addition, the communication layer integrates FTP protocol to support the transmission and storage of configuration files and log files; the communication layer adopts intranet penetration technology to build a communication bridge based on the public Internet; The application layer is the top layer in the digital twin platform architecture, which manages digital twin-based applications and services and creates a user-friendly operation interface.

7. The system according to claim 6, characterized in that: The virtual model library includes digital twin models corresponding to physical entities, and the virtual models include physical appearance, dynamics model, task logic model, and simulation interface model; these models are stored in a modular form, including configurable model parameters, and each virtual model has a unique physical entity as a mapping; The twin database includes various data generated in the digital twin platform, such as direct simulation data of physical entities and virtual models, virtual-real communication data between modules, and log data; The application algorithm library: the application algorithm library includes packaged specific algorithm models; The communication protocol library: the communication protocol is responsible for storing various communication protocol formats; Different communication protocols and interface definitions are used between physical entities, between virtual entities, and between virtual and real individuals in the digital twin platform; The virtual model is used by the user to perform basic operations on the virtual model, including: Model import: used to import a platform virtual model in a specific format. The model is composed of multiple modular component models and then used as a simulation instance in a twin world. Platform model creation: used to build modular component models into a platform instance model. In this link, the key parameters of the module will be determined, and the platform model will become a mapping of a specific physical object; Model definition; Model storage: The storage of models is simplified to the storage of types and key parameters. In the digital twin platform software, model storage stores the address and key parameter list of the model template in XML format. The last part of the resource layer involves the processing of all key data involved in the operation of the digital twin platform software, which includes the following modules: Historical data import: The data import module enables the digital twin platform software to import various data generated by the user before; Data filtering: Data filtering supports users to search and view data based on data fields, and repackage and save the data of interest; Data processing: Data processing preprocesses specific types of data, including outlier processing, noise smoothing, and duplicate data removal; Data Reinjection: Supports output data generated from simulation models and physical implementations to be used as input for further simulations, allowing top-level applications to reanalyze past simulation routines.

8. The system according to claim 6, characterized in that: The simulation configuration specifically includes: Simulation instance configuration: The simulation instance configuration is used to specify the initialization requirements of the platform model instance initialization involved in the simulation scenario of the twin world according to the prior requirements; Algorithm parameter configuration: Algorithm parameter configuration is used to configure the parameters of the algorithm executed in the top-level application according to user expectations and application scenarios; Scenario configuration: Scenario configuration involves the configuration of environmental factors of the simulation execution scenario, which is configured and randomized according to user needs to support interaction in learning; Communication configuration: The digital twin platform involves multiple connections. The data content of the communications between the parties is different. It is necessary to clarify the field composition between the two communicating parties through configuration; including fixed packet headers and packet bodies to distinguish the types and specific contents of different data.

9. The system according to claim 6, characterized in that: The application layer manages digital twin-based applications and services and creates a user-friendly operation interface, including: White-box monitoring allows users to indirectly monitor the physical drone by monitoring the virtual model, saving communication resources and achieving effective monitoring; Model update: Correct and update model parameters through data records during mission execution to maintain consistency between the digital twin model and the physical drone; Task prediction: using simulation environment to create a highly similar virtual environment, quickly predicting the potential results of tasks and providing decision-making references for users; Training learning, assists machine learning algorithms in training and reduces conversion errors.

Images