Unmanned aerial vehicle and unmanned surface vehicle formation control method and device based on unreal engine

Abstract

The application discloses a method and device for controlling multiple unmanned aerial vehicles and unmanned ships based on an unreal engine, and the method comprises the following steps: constructing a kinematic model of a multi-agent formation; constructing a first experimental simulation environment; constructing a double-loop alternating unmanned ship control model; migrating a deep reinforcement learning model trained in a particle model environment to the first experimental simulation environment; fine-tuning the trained deep reinforcement learning model in combination with the kinematic model and the double-loop alternating unmanned ship control model to obtain a multi-agent formation control model; deploying the multi-agent formation control model in a second experimental simulation environment for verification; and using the multi-agent formation control model to control the multi-agent formation. The application can maintain stable tracking with low error under continuous multi-trajectory point tracking, improve the training efficiency of the model in the simulation environment, retain high performance and precision of the model, improve the decision effect, and can be widely applied to the technical field of multi-agent control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent control, and in particular to a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method and device based on an Unreal Engine. BACKGROUND

[0002] At present, deep reinforcement learning is widely used in multi-agent control and decision-making scenarios. In the problem of multi-agent cooperative formation control, the core control objectives include the gathering of unmanned aerial vehicles, the formation of a formation, and the stable maintenance of the formation structure during movement. The multi-agent formation control method based on deep reinforcement learning (DRL) not only learns dynamic behavior in a complex environment through data-driven methods, but also exhibits stronger adaptability and robustness when faced with high-dimensional state spaces and diversified task requirements. These characteristics make the formation control model based on deep reinforcement learning have certain advantages in handling collaborative tasks of large-scale multi-agent systems. However, the multi-agent formation control model based on deep reinforcement learning has the problems of poor training fidelity under the traditional particle model and insufficient real-time decision-making response performance of the trained model in dynamic environments.

[0003] In summary, the technical problems in the related art need to be improved. SUMMARY

[0004] Embodiments of the present application aim to at least partially solve one of the technical problems in the related art. To this end, the main purpose of the embodiments of the present application is to propose a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method and device based on an Unreal Engine, which can maintain stable tracking with low error under continuous multi-trajectory point trajectory tracking, and at the same time, improve the model training efficiency in the simulation environment, retain high performance and accuracy of the model, and improve the decision-making effect of the model in the agent formation scenario.

[0005] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on an Unreal Engine, which comprises the following steps:

[0006] constructing a kinematic model of a multi-agent formation; wherein the multi-agent formation includes an unmanned aerial vehicle formation and an unmanned surface vehicle formation;

[0007] constructing a first experimental simulation environment of the multi-agent formation based on an Unreal Engine;

[0008] constructing a double-loop alternating unmanned surface vehicle control model based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned surface vehicle control model includes a heading controller and a distance controller;

[0009] construct a particle model environment, and pre-train a deep reinforcement learning model in the particle model environment;

[0010] migrate the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tune the trained deep reinforcement learning model in combination with the kinematic model and the double-loop alternating unmanned ship control model, and obtain a multi-agent formation control model;

[0011] deploy the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation.

[0012] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a multi-unmanned aerial vehicle and unmanned ship formation control device based on Unreal Engine, which comprises the following modules:

[0013] a kinematic model construction module for constructing a kinematic model of a multi-agent formation; wherein the multi-agent formation includes an unmanned aerial vehicle formation and an unmanned ship formation;

[0014] an experimental simulation environment construction module for constructing a first experimental simulation environment of the multi-agent formation based on Unreal Engine;

[0015] an unmanned ship control model construction module for constructing a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned ship control model includes a heading controller and a distance controller;

[0016] a deep reinforcement learning model pre-training module for constructing a particle model environment and pre-training a deep reinforcement learning model in the particle model environment;

[0017] a model migration and fine-tuning training module for migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tuning the trained deep reinforcement learning model in combination with the kinematic model and the double-loop alternating unmanned ship control model, and obtaining a multi-agent formation control model;

[0018] a multi-agent formation control model acquisition module for deploying the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation.

[0019] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0020] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0021] To achieve the above object, another aspect of the embodiments of the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the above method.

[0022] The embodiments of the present application at least have the following beneficial effects: the present application provides a multi-unmanned aerial vehicle and unmanned ship formation control method and device based on Unreal Engine, which constructs a kinematic model of multi-agent formation; wherein the multi-agent formation includes an unmanned aerial vehicle formation and an unmanned ship formation; a first experimental simulation environment of the multi-agent formation is constructed based on Unreal Engine; a double-loop alternating unmanned ship control model is constructed based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned ship control model includes a heading controller and a distance controller; a particle model environment is constructed, and a deep reinforcement learning model is pre-trained in the particle model environment; the deep reinforcement learning model trained in the particle model environment is migrated to the first experimental simulation environment, and the trained deep reinforcement learning model is fine-tuned in combination with the kinematic model and the double-loop alternating unmanned ship control model to obtain a multi-agent formation control model; the multi-agent formation control model is deployed in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation. The embodiments of the present application utilize the high simulation and multi-platform collaborative test environment provided by the Unreal Engine platform itself, combine the kinematic model, and design the alternating control of the heading controller and the distance controller according to the characteristics of the Unreal Engine platform, which can still maintain stable tracking with low error under continuous multi-trajectory point trajectory tracking, and builds a necessary and solid bridge for the particle model of the unmanned ship to the Unreal Engine simulation platform to realize more realistic control effect. Moreover, through the migration process from the particle model to the Unreal Engine simulation environment, the migration of the control strategy is realized, the model training efficiency of the deep reinforcement learning model in the simulation environment is improved, the high performance and precision of the deep reinforcement learning model are reserved, and the good decision-making effect of the deep reinforcement learning model in the agent formation scene is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1is a step flowchart of a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine provided by an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a coordinate system of an unmanned aerial vehicle provided by an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a control speed design of an unmanned aerial vehicle provided by an embodiment of the present application;

[0027] Figure 5 is a planar schematic diagram of a coordinate system of an unmanned surface vehicle provided by an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a control speed design of an unmanned surface vehicle provided by an embodiment of the present application;

[0029] Figure 7 is a right rear view of a grid body model of an unmanned aerial vehicle provided by an embodiment of the present application;

[0030] Figure 8 is a right rear view of a grid body model of an unmanned surface vehicle provided by an embodiment of the present application;

[0031] Figure 9 is a top view of a grid body model of an unmanned surface vehicle provided by an embodiment of the present application;

[0032] Figure 10 is a control flowchart of an unmanned surface vehicle provided by an embodiment of the present application;

[0033] Figure 11 is an internal flowchart of a PID controller of an unmanned surface vehicle provided by an embodiment of the present application;

[0034] Figure 12 is a distance deviation curve diagram in Experiment One provided by an embodiment of the present application;

[0035] Figure 13 is a heading angle deviation curve diagram in Experiment One provided by an embodiment of the present application;

[0036] Figure 14 is a comparison diagram of an expected trajectory and an actual trajectory in Experiment One provided by an embodiment of the present application;

[0037] Figure 15 is an expected trajectory diagram in Experiment Two provided by an embodiment of the present application;

[0038] Figure 16 is an actual trajectory diagram in Experiment Two provided by an embodiment of the present application;

[0039] Figure 17 is a comparison diagram of expected trajectory and actual trajectory in the second experiment provided by the embodiment of the present application;

[0040] Figure 18 is a schematic diagram of obstacle detection of the unmanned ship provided by the embodiment of the present application;

[0041] Figure 19 is a schematic diagram of each formation and environment layout in the Unreal Engine provided by the embodiment of the present application;

[0042] Figure 20 is a decision trajectory diagram obtained after fine-tuning training in the Unreal Engine provided by the embodiment of the present application;

[0043] Figure 21 is a structural schematic diagram of the multi-unmanned aerial vehicle and unmanned ship formation control device based on the Unreal Engine provided by the embodiment of the present application;

[0044] Figure 22 is a hardware structure schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present application clearer, the following will be further described in detail in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. When the following description relates to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0046] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0047] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0049] In recent years, deep reinforcement learning has been widely applied to multi-agent control and decision-making scenarios. In the problem of multi-agent cooperative formation control, the core control objectives include the gathering of unmanned aerial vehicles, the formation of a formation, and the stable maintenance of the formation structure during movement. Currently, the most widely used methods mainly include the leader-follower method, the artificial potential field method, and the kinematics-based control method, etc.

[0050] The leader-follower method is a classical and widely used strategy in multi-agent cooperative formation control. The core idea is to guide other followers to complete the formation task through a small number of leaders (usually agents with global target information). In this method, the leader is responsible for path planning, target point assignment, and global formation structure maintenance, while the follower adjusts its position and motion state to maintain the relative relationship with the leader through observation and perception of the leader. Usually, this method ensures that the formation structure always meets a specific geometric shape during movement by pre-setting fixed relative distance or angle constraints. The leader-follower method has the advantages of simple implementation and high computational efficiency, especially in scenarios with a small number of agents. However, it has a high dependence on leaders, and once the leaders fail or make errors in complex environments, the entire formation may not function properly. In addition, as the number of agents increases, the information exchange and collaboration needs between followers significantly increase, limiting the applicability of the method in large-scale systems.

[0051] Artificial potential field method is a multi-agent cooperative control method based on virtual force field, widely used in formation control, obstacle avoidance and path planning tasks. The core idea of this method is to construct a virtual potential field for the agent, in which the target point, obstacles and other agents are modeled as attractive and repulsive sources to guide the motion behavior of the agent. Specifically, the target point exerts an attractive force on the agent, causing it to move towards the target direction, while obstacles or other agents exert a repulsive force to avoid collisions or maintain proper formation structure. The resultant force of the agent is obtained by vector superposition of attractive and repulsive forces, thereby determining its next motion direction and speed. Artificial potential field method is simple to implement and computationally efficient, especially in local environment modeling and single-agent obstacle avoidance; however, the parameters of the potential field function need to be carefully designed and debugged to adapt to different tasks or environments; in addition, the traditional potential field method is prone to local optimal problem, especially in complex obstacle environment, the agent may be trapped in a certain position and unable to move forward. In addition, when the number of agents is large, the complexity of the repulsive force interaction increases significantly, which may lead to unstable formation structure or decreased control performance.

[0052] Kinematics-based control method is a multi-agent cooperative control method that takes the kinematic model of the agent as the core, by directly describing the kinematic variables such as position, velocity and acceleration of the agent to achieve formation formation, maintenance and trajectory tracking tasks. This method usually takes the pre-set motion trajectory or formation constraint as the target, decouples the position relationship and velocity coupling relationship between agents, and designs the control law to ensure that each agent moves along the expected path and maintains the geometric shape of the formation. The form of the kinematic model is usually a differential equation, which describes the change of the motion state of the agent with time, and can accurately reflect the motion characteristics of the agent. Kinematics-based control method has the characteristics of high control accuracy and rigorous theoretical basis, especially stable in static and simple dynamic environment; however, its high dependence on the physical model of the agent limits its adaptability to complex environments, when the number of agents increases or the complexity of the environment increases, the system needs higher computing resources to update the kinematic constraints in real time, which affects the control efficiency. In addition, this method usually assumes that the agent can accurately execute the control instructions, while in actual scenarios, the dynamics limitations of the agent and environmental uncertainties may cause deviations, thereby affecting the stability of the formation.

[0053] The leader-follower method, artificial potential field method, and kinematics-based control method of the above examples are traditional formation control techniques that have provided a solid theoretical foundation for subsequent research due to their excellent performance in precise formation control tasks. However, these classic methods are significantly effective in scenarios with a small number of agents, but their adaptability to large-scale formations is clearly limited. When the number of agents increases, the complexity of the system's dynamic modeling increases rapidly, and the computational load grows exponentially, leading to a significant decline in the feasibility and efficiency of real-time control. To address these issues, multi-agent formation control methods based on deep reinforcement learning (DRL) have gradually become a research hotspot in recent years. Compared to traditional methods, deep reinforcement learning not only learns dynamic behavior in complex environments through data-driven methods, but also exhibits greater adaptability and robustness when faced with high-dimensional state spaces and diverse task requirements. These characteristics make deep reinforcement learning-based formation control models advantageous in handling large-scale multi-agent system coordination tasks. Additionally, many deep reinforcement learning-based models are not completely independent of traditional methods, often incorporating classic control techniques such as the leader-follower model or artificial potential field method to improve control effectiveness. In terms of the environment for deploying agents, control of agents is primarily based on the point mass model, which assumes that after making a decision to act, agents move at a constant speed in a straight line, i.e., their position at the next decision execution time only depends on the execution time and speed. This assumption simplifies the environment, but it also ignores the effects of external forces, making it difficult to effectively model real-world situations. Therefore, in addition to the initial point mass model, algorithms need to complete multi-agent tasks in environments with higher realism to achieve better practical results. In terms of control, deep control models, i.e., deep learning control models, are a type of model widely used in the control system field in recent years. These models use deep neural networks to model the dynamic behavior of complex systems and use these models for system control. The core advantage of deep learning control is its powerful non-linear modeling capability, which can automatically learn system features from a large amount of data, providing solutions for complex systems where traditional control methods are ineffective. UE4 (Unreal Engine 4), which was originally used for game development, has integrated various environment models and dynamic models in recent years and is gradually being used for physical simulation and visualization. The high-fidelity environment built in UE4 can perform controller performance verification well and provide accurate kinematics for multi-agent cooperative control.However, the multi-agent formation control model based on deep reinforcement learning has the problems of poor training fidelity under the traditional particle model and insufficient real-time decision response performance of the trained model in a dynamic environment.

[0054] Therefore, in the embodiments of the present application, a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method and device based on Unreal Engine are provided. The scheme uses the high simulation and multi-platform cooperative test environment provided by the Unreal Engine platform itself, and designs an alternating control of the heading controller and the distance controller in combination with the kinematic model for the characteristics of the Unreal Engine platform. The scheme can maintain stable tracking with low error under continuous multi-trajectory point trajectory tracking, and builds a necessary and solid bridge for transferring the particle model of the unmanned surface vehicle to the Unreal Engine simulation platform to achieve more realistic control effect. Furthermore, the scheme realizes the migration of the control strategy by constructing the migration process of the particle model to the Unreal Engine simulation environment, improves the model training efficiency of the deep reinforcement learning model in the simulation environment, retains the high performance and precision of the deep reinforcement learning model, and improves the good decision effect of the deep reinforcement learning model in the agent formation scenario.

[0055] The multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine provided in the embodiments of the present application relates to the technical field of multi-agent control. The multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto. The server end can be configured as a standalone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network. The software can be an application that implements the multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine, and the like, but is not limited to the above forms.

[0056] The application is operable with numerous general purpose or special purpose computer system environments or configurations. Examples include: personal computers, server computers, hand held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0057] Referring to Figure 1 , Figure 1 is an optional step flowchart of a method for multi-UAV and unmanned ship formation control based on Unreal Engine provided by an embodiment of the application, Figure 1 The method in the embodiment can include, but is not limited to, steps S101 to S106.

[0058] Step S101: constructing a kinematic model of the multi-agent formation; wherein the multi-agent formation includes a UAV formation and an unmanned ship formation;

[0059] In some embodiments, step S101 can include: determining a UAV coordinate system and a UAV movement mode of the UAV formation; determining an unmanned ship coordinate system and an unmanned ship movement mode of the unmanned ship formation; constructing a kinematic model corresponding to the UAV formation according to the UAV coordinate system and the UAV movement mode; and constructing a kinematic model corresponding to the unmanned ship formation according to the unmanned ship coordinate system and the unmanned ship movement mode.

[0060] In the embodiment of the application, the kinematic model is the basis for multi-agent formation control, which accurately describes the movement law of the UAV and the unmanned ship in space, including the relationship between the position, velocity, acceleration and other parameters and time. By constructing the kinematic model of the multi-agent formation, the movement characteristics of each agent can be clearly understood, providing a theoretical basis for the subsequent design of effective control strategies, ensuring that the formation moves according to the expected trajectory and formation during the movement process, and ensuring the smooth execution of the formation task.

[0061] Step S102: constructing a first experimental simulation environment of the multi-agent formation based on Unreal Engine;

[0062] In some embodiments, step S102 can include: creating a new level in the Unreal Engine and importing a water body marine scene in the level; creating a drone blueprint class in the Unreal Engine and importing a drone mesh model in the drone blueprint class to obtain a drone blueprint; creating an unmanned ship blueprint class in the Unreal Engine and importing an unmanned ship mesh model in the unmanned ship blueprint class to obtain an unmanned ship blueprint; importing the drone blueprint and the unmanned ship blueprint into the water body marine scene to complete the construction of the first experimental simulation environment.

[0063] In the embodiments of the present application, by constructing the first experimental simulation environment of the multi-agent formation, a near-real experimental condition can be provided for the multi-agent formation, so that researchers can fully test and verify the control algorithm and strategy of the formation without the need to actually build an expensive and complex physical experimental platform, greatly reducing the experimental cost, improving the research and development efficiency, and at the same time, problems that can be encountered in the actual environment can be found and solved in advance.

[0064] Step S103, constructing a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned ship control model includes a heading controller and a distance controller.

[0065] In some embodiments, the step of executing the control of the double-loop alternating unmanned ship control model can include: during the process of the unmanned ship performing a task, calculating an expected position of the unmanned ship according to a decision control instruction for the unmanned ship at the current time generated by the multi-agent formation control model; wherein the decision control instruction includes a decision speed and a decision execution time; calculating an expected heading angle between the expected position and a current position of the unmanned ship; correcting the expected heading angle and calculating a target heading angle deviation between the corrected expected heading angle and an actual heading angle of the unmanned ship at the current time; when an absolute value of the target heading angle deviation is greater than a preset heading angle deviation threshold, activating the heading controller to control the yaw angle of the unmanned ship; and when the absolute value of the target heading angle deviation is less than the preset heading angle deviation threshold, activating the distance controller to control the distance of the unmanned ship.

[0066] In the embodiments of the present application, in the control of the unmanned aerial vehicle, because the kinetic model is simplified, the speed given by the decision network (also known as a decision maker, i.e., a multi-agent formation control model) is directly used to control the unmanned aerial vehicle, without the need for additional design of a control model. In the control of the unmanned surface vehicle, considering that the decision space is five discrete control speeds, but the value and effect of the decision are mainly based on the position of the unmanned surface vehicle, only controlling the speed of the unmanned surface vehicle will introduce an error that is difficult to estimate in the decision evaluation part; therefore, the present application chooses to directly control the position of the unmanned surface vehicle using a controller, and assumes that the unmanned surface vehicle moves at a constant speed in a straight line according to the decision speed within the decision execution time. Specifically, the present application uses a double-loop alternating control of a heading controller and a distance controller to enable the unmanned surface vehicle to complete the task of reaching the expected position.

[0067] In the embodiments of the present application, the double-loop alternating unmanned surface vehicle control model decomposes the control of the unmanned surface vehicle into two relatively independent and mutually cooperative subsystems of heading control and distance control. Through the alternating control of the heading controller and the distance controller, stable tracking with low error can be maintained under continuous multi-trajectory point trajectory tracking. The heading controller is responsible for ensuring that the unmanned surface vehicle travels according to the predetermined heading, ensuring the consistency of the direction of travel of the formation as a whole; the distance controller precisely controls the distance between the unmanned surface vehicle and other members in the formation or target points, maintaining the stability of the formation shape. This double-loop alternating control method can more flexibly and accurately cope with complex and variable marine environments, improve the control accuracy and robustness of the unmanned surface vehicle formation, and enable the formation to better adapt to various dynamic changes when performing tasks.

[0068] Step S104, constructing a particle model environment, and pre-training a deep reinforcement learning model in the particle model environment;

[0069] In the embodiments of the present application, the particle model environment simplifies the physical properties of the multi-agent, treats the agent as a particle, and removes the influence of complex shapes, sizes, and other factors on training, thereby greatly reducing the computational complexity and accelerating the training speed of the deep reinforcement learning model. Pre-training in the particle model environment enables the model to quickly learn basic control strategies and behavior patterns, lays a good foundation for subsequent fine-tuning training in a more complex simulation environment, improves the overall training efficiency, and reduces training time and resource consumption.

[0070] Step S105, migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tuning the trained deep reinforcement learning model in combination with the kinematic model and the double-loop alternating unmanned surface vehicle control model, and obtaining a multi-agent formation control model;

[0071] In some embodiments, step S105 can include: migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, obtaining the control sequence generated by the double-ring alternating unmanned ship control model, and obtaining the motion state of the multi-agent formation in the kinematic model; inputting the control sequence and the motion state into the trained deep reinforcement learning model for fine-tuning training to obtain the multi-agent formation control model.

[0072] In the embodiments of the present application, by migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment closer to the real environment for fine-tuning training, the general knowledge learned in the pre-training stage can be fully utilized, and at the same time, the detailed physical information and complex scenes in the first experimental simulation environment are combined to make the model further learn the control strategy adapted to the actual environment. Combined with the kinematic model and the double-ring alternating unmanned ship control model, the model can ensure accurate control of the formation heading and distance on the basis of following the motion law of the agent, and finally the obtained multi-agent formation control model has stronger adaptability and control precision, and can better cope with various challenges in actual tasks.

[0073] Step S106, deploying the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation.

[0074] In some embodiments, step S106 can include: constructing a second experimental simulation environment different from the first experimental simulation environment based on the Unreal Engine; deploying the multi-agent formation control model in the second experimental simulation environment, verifying the multi-agent formation control model, and obtaining the average total number of collisions and the average reward convergence value in the verification process; determining whether the multi-agent formation control model passes the model verification according to the average total number of collisions and the average reward convergence value, and if the multi-agent formation control model passes the model verification, dynamically controlling the multi-agent formation performing the task with the verified multi-agent formation control model.

[0075] The main difference between the first experimental simulation environment for training and the second experimental simulation environment for verification includes but is not limited to the number and distribution of obstacles in the experimental simulation environment. It should be noted that the setting of the experimental simulation environment can be set according to the actual application, and the embodiments of the present application do not limit this.

[0076] In the embodiment of the present application, the generalization ability and robustness of the multi-agent formation control model can be comprehensively tested by verifying in a second experimental simulation environment different from the first experimental simulation environment. Through testing under different scenarios and conditions, it can be ensured that the model can stably and reliably operate under various unknown environments and effectively control the multi-agent formation to complete the predetermined task. The model verification process is conducive to discovering potential problems that may exist in the model in actual application, providing a direction for further optimization and improvement of the model, and ultimately improving the success rate and reliability of the entire multi-agent formation system in actual tasks.

[0077] The steps S101 to S106 shown in the embodiment of the present application use the high simulation and multi-platform cooperative test environment provided by the Unreal Engine platform itself, combine the kinematic model, and design the alternate control of the heading controller and the distance controller according to the characteristics of the Unreal Engine platform, which can maintain stable tracking with low error under continuous multi-trajectory point tracking, and build a necessary and solid bridge for the point model of the unmanned ship to transfer to the Unreal Engine simulation platform to achieve more realistic control effect. Moreover, by constructing the migration process of the point model to the Unreal Engine simulation environment, the migration of the control strategy is realized, the model training efficiency of the deep reinforcement learning model in the simulation environment is improved, the high performance and precision of the deep reinforcement learning model are retained, and the good decision-making effect of the deep reinforcement learning model in the agent formation scenario is improved.

[0078] To explain the principle of the technical scheme of the present application, the overall process of the present application will be described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present application and cannot be regarded as a limitation of the present application.

[0079] Please refer to Figure 2 For example Figure 2As shown, the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application mainly includes four steps, which are: (1) design the unmanned aerial vehicle / ship kinematic model: establish the coordinate system of the unmanned aerial vehicle / ship, simplify the freedom of unmanned aerial vehicle / ship movement according to the actual situation, and design a reasonable control speed for the unmanned aerial vehicle / ship; (2) build a multi-unmanned aerial vehicle and unmanned ship formation environment: the environment is built in the Unreal Engine, and the wave model of the environment needs to be designed, as well as the use of three-dimensional modeling of the unmanned aerial vehicle / ship and the design of the mechanical model; (3) design the unmanned aerial vehicle / ship control model: design the double-loop alternating control of the heading controller and the distance controller of the unmanned ship, build the data interaction between the Unreal Engine and the controller, and ensure that the unmanned ship completes the task of reaching the expected position; (4) migrate the particle model to the simulation environment: design a simulation environment similar to the particle model environment in the Unreal Engine, migrate the deep reinforcement learning model trained in the particle model to the simulation environment for fine-tuning training, and finally verify the model in the simulation environment different from the fine-tuning training based on the Unreal Engine. In the embodiments of the present application, the Unreal Engine used is UE4, and hereinafter the Unreal Engine can be described as UE4.

[0080] As shown in Figure 2 , the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application builds a high simulation environment based on the Unreal Engine, designs the dynamics model of the unmanned aerial vehicle / ship and the corresponding controller, so as to realize the migration of the deep reinforcement learning algorithm from the particle model to the simulation environment, and the specific implementation process is as follows:

[0081] Step (1), design the kinematic model of the unmanned aerial vehicle / ship:

[0082] (1-1) The embodiments of the present application select a quadrotor unmanned aerial vehicle and a double-paddle unmanned ship as the dynamics model of the unmanned aerial vehicle and the unmanned ship, respectively.

[0083] (1-2) The unmanned aerial vehicle coordinate system is selected, which is divided into the earth coordinate system E and the body coordinate system B.

[0084] Among them, the earth coordinate system E, also known as the inertial coordinate system, takes a fixed point on the ground as the origin , takes the positive east direction as the positive direction of the axis, takes the positive north direction as the positive direction of the axis, and according to the right-hand coordinate system principle, the vertical upward direction is the positive direction of the axis.

[0085] The body coordinate system B is fixed on the unmanned aerial vehicle, as shown in Figure 3 , the coordinate moves and rotates with the position and attitude of the unmanned aerial vehicle, and the origin is generally taken as the center of gravity of the unmanned aerial vehicle, and the head direction is taken as the The positive direction of the axis, the port side direction as The positive direction of the axis, the vertical upward direction according to the principle of the right-hand coordinate system The positive direction of the axis; wherein, , , And The four rotors of the unmanned aerial vehicle generate upward thrusts in the same direction as the body coordinate system axis.

[0086] (1-3) Selection of unmanned aerial vehicle movement mode. The quad-rotor unmanned aerial vehicle adjusts the speed of the rotors by controlling the speed of the four motors (the motor direction of the quad-rotor unmanned aerial vehicle is unchanged during operation), realizes the change of the four lifting forces and the rotating torques, and controls the basic movement of the six degrees of freedom of the quad-rotor unmanned aerial vehicle.

[0087] As shown in Figure 4 , the embodiments of the present application mainly consider the two degrees of freedom movement (forward and backward movement, lateral movement) of the unmanned aerial vehicle in the x-y plane, and design five control speeds (v , , , and ) for the unmanned aerial vehicle, , , , and represent five decision actions of going forward, going left, going right, going backward and staying in place, so as to facilitate the unmanned aerial vehicle to complete the corresponding task. Since the dynamic model of the unmanned aerial vehicle has more input control variables (the speed of the four rotors), the size and inertia are small, and the movement mode is very flexible and variable in actual application, the controller of the unmanned aerial vehicle is simplified, the control process of the unmanned aerial vehicle from the original state to the expected speed is ignored, and the control of the unmanned aerial vehicle is regarded as the case that the speed of the unmanned aerial vehicle at the next moment can be directly controlled according to the decision action. It should be noted that the control mode of the unmanned aerial vehicle is the same in the simulation environment as in the particle model environment.

[0088] (1-4) Selection of unmanned boat coordinate system. The unmanned boat movement can also be regarded as six degrees of freedom rigid body movement, which is usually described by two right-hand Cartesian coordinate systems, similar to the unmanned aerial vehicle, one is the earth-fixed coordinate system E , , , ); the other is the body-fixed coordinate system B , , , ), also known as the body coordinate system, the x-y plane thereof is as shown in Figure 5As shown. The definition of the geodetic fixed coordinate system is similar to that of the Earth coordinate system in a UAV, the difference being... The axis points due west instead of due east. According to the principle of the right-hand coordinate system... The z-axis points vertically downwards because surface targets generally don't leap out of the water; therefore, taking downwards as the z-axis, the y-axis must also be changed accordingly. For example... Figure 5 As shown, the body coordinate system is fixed on the unmanned surface vessel (USV) and moves with it, with the origin at... Located at the center of gravity of the unmanned surface vessel, the x-axis points to the bow, the y-axis points to the starboard side, and the z-axis points downwards. Figure 5 In This refers to the heading angle, and v refers to the actual speed of the unmanned surface vessel. Figure 5 The diagram illustrates the body coordinate system of the unmanned surface vessel (USV) and its motion in the x and y planes. The translational velocity and motion in the x and y planes are achieved by combining yaw motion. Since the USV only has two thrusters, it cannot achieve lateral motion in the y direction. Therefore, displacement in the y direction needs to be achieved by changing the heading angle.

[0089] (1-5) Selection of the unmanned surface vessel's movement mode. For example... Figure 6 As shown, this application embodiment considers the controller distribution and inertia of the unmanned surface vessel (USV). In this application embodiment, the motion of the USV is simplified to a two-degree-of-freedom motion in the xy plane (forward and backward motion, yaw motion), and five control velocities are designed for the USV. , , , and ), , , , as well as These represent five decision actions: forward, left-forward, right-forward, backward, and stationary, to facilitate the unmanned surface vessel (USV) in completing its tasks. Pitch, heave, and roll motions in the xz and yz planes of the USV are ignored here. Furthermore, since the USV uses a twin-propeller hull model, it cannot be directly controlled to perform lateral movement; therefore, lateral movement is also not considered in this application. It should be noted that in this embodiment, the five control speeds of the USV (…) , , , and (Use uppercase) This indicates that the unmanned surface vessel has five control speeds ( , , , and (Use lowercase v to represent it.)

[0090] Step (2), constructing multi-UAV and USV formation environment:

[0091] (2-1) New level in UE4 platform, import water body marine scene, water body marine component wave attenuation water depth D is set to 20.48 m, the maximum wave height offset is set to 0, and the wave type is selected as Gerstner wave, which is a way of simulating water surface wave form and belongs to a kind of linear wave superposition method. For Gerstner wave, the wave number , the minimum wavelength , the maximum wavelength , and the wavelength drop . The rest is set according to the default setting, so as to complete the relatively stable sea level environment layout.

[0092] (2-2) Create UAV blueprint class, import UAV grid model. The UAV grid model is shown in Figure 7 . For the UAV, the application embodiment selects a quadrotor UAV model, but because the kinetic model is simplified, the UAV is controlled directly according to the speed given by the decision network, and the mass of the simulation physical part is not set.

[0093] (2-3) Create USV blueprint class, import USV grid model, the structure of the USV grid model is shown in Figure 8 and Figure 9 , two propellers with thrust direction along the x-axis direction of the hull are set at the position of the propeller at the tail of the hull, and six symmetrically distributed buoyancy components are set on both sides of the hull, and the radius of each buoyancy component is set to 1 m.

[0094] Among them, the angle resistance coefficient of the buoyancy component is changed to 1.5, and the rest is kept as the default setting, and the specific parameter setting is shown in Table 1:

[0095] Table 1: Buoyancy parameter setting table

[0096]

[0097] The distribution positions of the buoyancy component and the propeller component are shown in Table 2:

[0098] Table 2: Buoyancy component and propeller component distribution position setting table

[0099]

[0100] The rest of the simulation physical quantity settings are as follows: mass 200 kg, linear damping 0.001 N·s / m, and angular damping coefficient is set to 0 N·m·s / rad.

[0101] (2-4) Repeat steps (2-2) and (2-3) until the number of formation requirements are met, and drag the set unmanned aerial vehicles and unmanned boats to the water body marine scene. The embodiment of the application sets a three-formation scene of three leader unmanned aerial vehicles (m=3) and five follower unmanned boats (n=5), that is, the total number of agents in the Block scene is set to 18; secondly, the yaw angle (yaw angle) of each follower unmanned boat is set to 0, the Z-axis height is fixed to 50, and the X-axis and Y-axis coordinate values are determined according to the number and total number of the follower unmanned boats.

[0102] Step (3), design the unmanned aerial vehicle / boat control model:

[0103] (3-1) Control model selection. In the control of unmanned aerial vehicles, because the dynamic model is simplified, the speed given by the decision network (also known as a decision maker, that is, the multi-agent formation control model described above) is directly used to control the unmanned aerial vehicle, and no additional control model needs to be designed. In the control of unmanned boats, considering that the decision space is five discrete control speeds, but the value and effect of the decision are mainly based on the position of the unmanned boat, and only controlling the speed of the unmanned boat will introduce an error that is difficult to estimate in the decision evaluation part; therefore, the application chooses to use a controller to directly control the position of the unmanned boat, and assumes that the unmanned boat moves at a constant speed in a straight line according to the decision speed within the decision execution time. The unmanned boat control flow diagram and the internal process diagram of the unmanned boat PID controller are shown in Figure 10 and Figure 11 , wherein the calculation formula of the expected position of the unmanned boat is as follows:

[0104] ;

[0105] , wherein Figure 10 represents the decision displacement, and the calculation formula of the decision displacement is as follows:

[0106] .

[0107] (3-2) Controller design. As shown in Figure 10 , a classic PID controller is used to reduce the deviation between the expected position and the current position , so that the unmanned boat gradually approaches and finally stays at the expected position . Specifically, the application uses a double-loop alternating control of a heading controller and a distance controller to enable the unmanned boat to complete the task of reaching the expected position, and the two controllers are manually tuned. The specific implementation process of the double-loop alternating control of the heading controller and the distance controller is as follows: ​

[0108] Assume that at the unmanned surface vehicle gets control commands (decision speed and ) according to the decision network. The expected position can be calculated by the formula of expected position . Specifically, first calculate the expected heading angle between the expected position and the current position , the expected heading angle is calculated as follows:

[0109] ;

[0110] wherein and are the deviations between the expected position and the current position in the axes of the geodetic coordinate system and . In addition, in order to distinguish between forward and backward movements, and to prevent the unmanned surface vehicle from turning the entire bow to move backward when moving backward, take the positive and negative signs of as markers for forward and backward movements respectively , the expression of the marker is as follows:

[0111] ;

[0112] wherein is a sign function that takes the positive or negative sign of the value.

[0113] Subsequently, the expected heading angle is corrected according to the marker to obtain the corrected expected heading angle . The calculation formula for correcting the expected heading angle is as follows:

[0114] ;

[0115] Next, calculate the heading angle deviation between the corrected expected heading angle and the actual heading angle of the unmanned surface vehicle at the current time . The calculation formula for calculating the heading angle deviation between the corrected expected heading angle and the actual heading angle of the unmanned surface vehicle at the current time is as follows:

[0116] ;

[0117] As shown in FIG. 3, when the absolute value of the heading angle deviation between the modified expected heading angle Figure 11 and the actual heading angle of the USV at the current time is greater than the preset heading angle deviation threshold (set as 0.1 rad), the yaw angle controller (i.e. the heading controller) is activated for yaw angle control. After manual tuning, the PID parameters of the heading controller can be determined as , , i.e. the heading controller is simplified as a proportional controller. The torque control quantity is obtained through the proportional link, and the calculation formula of the torque control quantity is as follows:

[0118] ;

[0119] Then, the thrusts on the two propellers of the USV (F1, F2) are obtained according to the torque control quantity , and at this time, the calculation formula of the thrusts on the two propellers of the USV (F1, F2) is as follows:

[0120] ;

[0121] wherein, d is the distribution interval of the two propellers, which can be obtained from Table 2 , and here, d is taken as an approximate value for convenience of calculation.

[0122] When the absolute value of the heading angle deviation between the modified expected heading angle and the actual heading angle of the USV at the current time is less than the preset heading angle deviation threshold , the distance controller is activated for distance control, and at this time, the distance deviation between the expected position and the current position needs to be calculated, and the calculation formula of the distance deviation is as follows:

[0123] ;

[0124] wherein, the mark ​​​​​​​​​The introduction of the purpose is to give the distance deviation direction, avoid the distance deviation is always positive when the overshoot to bring the wrong feedback direction, make the control system divergent. After manual setting, the PID parameters of the distance controller can be determined as , , , that is, the distance controller is simplified to a proportional derivative controller. The thrust control quantity is obtained through the proportional and derivative links The calculation formula of thrust control quantity

[0125] ;

[0126] Wherein, is the distance deviation at the last control time, the initial value is set to 0; is the control time, the initial value is set to 0.01s, which is adjusted in real time according to the transmission delay in the subsequent control process.

[0127] Then, according to the thrust control quantity , the thrust is distributed, since the two propellers are symmetrically distributed about the x axis, so the symmetric distribution can be directly obtained. At this time, the calculation formula of the thrust on the two propellers on the unmanned ship is as follows:

[0128] ;

[0129] After the above control process, the thrust calculated by the controller is split into thrust and , then the thrust and are corrected to obtain and , finally the corrected thrust and are transmitted to the simulation platform, and the next control time of the unmanned ship position and heading angle fed back by the simulation platform are received, and the current position and current yaw angle are updated, the specific update expression is as follows:

[0130] ;

[0131] Repeat the above process, the expected position is unchanged in this process until the next decision execution time , the decision maker updates the expected position, the unmanned ship gets a new target position and continues to control and make corresponding movements according to the control process. Overall, in the embodiment of the application, the control of the unmanned ship can be basically regarded as a control process of first completing the heading angle control and then completing the distance control; but in the case that the heading angle is greatly affected in the distance control process, it will be temporarily switched to the yaw angle controller to correct the heading angle first.

[0132] It is worth mentioning that when the distance deviation of the unmanned ship is very small, that is, when the unmanned ship approaches the target point, the expected heading angle calculated by the calculation formula of the expected heading angle may oscillate to cause the oscillation of the heading angle deviation , so that the unmanned ship performs unstable oscillation movement under the action of the heading angle controller. Therefore, the calculation formula of the expected heading angle needs to be modified to avoid this situation, and the modified calculation formula of the expected heading angle is as follows: ;

[0133] ;

[0134] wherein is a distance threshold, which is set to 0.1 m. This modification can be regarded as prohibiting the use of the heading angle controller in the last stage of the unmanned ship approaching the target point, so as to prevent the unmanned ship from performing oscillation movement.

[0135] The thrust calculated by the controller needs to be clipped by setting a maximum thrust value to prevent the thrust from being too large. In addition, it also needs to be multiplied by a multiplier and then transmitted to the simulation platform to set the propeller strength. The maximum thrust value is set to 10 N in the application, and the propeller strength multiplier is set to 10000. Therefore, the modified calculation formula of the thrust is as follows:

[0136] ;

[0137] wherein represents a clipping operation, and when the input value is within the interval , the output value is ; when the input value is less than the lower boundary , the output value is the lower boundary, and when the input value is greater than the upper boundary , the output value is the upper boundary, so as to limit the input value​​ It achieved the function of limiting the amplitude.

[0138] According to the calculation formula for correcting thrust ( ) respectively for thrust Thrust is obtained by making corrections This will allow the corrected thrust to be used. The data is transmitted to the simulation platform to set the thruster intensity. The unmanned surface vessel (USV) control flow diagram and the internal flow diagram of the USV's PID controller are shown below. Figure 10 and Figure 11 As shown, Figure 11 The middle diamond represents a commonly used process judgment difference graph. If the condition within the diamond is met, the process is transmitted to the True direction; otherwise, it's transmitted to the False direction. If neither True nor False is found, the transmission is interrupted. For... Figure 11 The triangle pointing right represents a switch valve. When data is received from the upper or lower side, data can flow between the left and right sides, and data can be transmitted from the left to the right. If no data is received from the upper or lower side, data cannot be transmitted from the left to the right. Figure 10 The triangle pointing to the right in the center represents the waiting valve. If the value inside the triangle is 't', it means that after waiting for time 't', the data on the left will be transferred to the right according to the arrow.

[0139] (3-3) Data Transmission Design. After completing the controller design, the speed of the UAV is directly controlled, and each UAV is equipped with a position controller (including a heading controller and a distance controller). Specifically, the ObjectDeliverer and TCP Socket plugins need to be imported into the UE4 platform to create a control blueprint for TCP data transmission, set up a TCP port connection to transmit information of each UAV and UAV to the TCP port, and receive control commands from the TCP port to control each UAV and UAV. In this application, the TCP sender needs to obtain the position information of each UAV and UAV and send the following list to the TCP port:

[0140] ;

[0141] ;

[0142] in, Representing the first The x, y, z coordinates of the drone in the geodetic coordinate system Representing the first The unmanned surface vessel's x and y coordinates in the geodetic coordinate system are indicated by a trailing 0 to prevent garbled characters at the end from interfering with the acquisition of position information during program reception.

[0143] The TCP receiver receives the following control speed and thrust information from the controller:

[0144] ;

[0145] wherein, respectively represent the x, y, z direction velocity of the first unmanned aerial vehicle in the earth coordinate system, respectively represent the two control thrusts of the first unmanned surface vehicle, indicates the flag of restarting the scene, when the scene is not reset, when the scene is reset, each unmanned aerial vehicle and unmanned surface vehicle is set back to the initial position, and the heading angle, velocity, acceleration, and angular velocity of each unmanned aerial vehicle and unmanned surface vehicle are all set to 0. The first 123 and the last 321 are used to check the control list. (3-4) Controller experiment design and effect evaluation. In the embodiments of the present application, two experiments are designed to evaluate the accuracy of the control model to ensure that the control model can complete the control task required by the decision maker.

[0146] Combined with the five control velocities of the unmanned surface vehicle

[0147] , i.e. forward, left forward, right forward, backward, and stationary, the decision execution time is set to 1 m / s, and the initial position of a unmanned surface vehicle is . At this time, the control velocity of the unmanned surface vehicle can be expanded as follows:

[0148] ;

[0149] Experiment 1: Four displacement actions are artificially selected in the control velocity as a continuous control sequence in the order of , i.e. forward, left forward, right forward, and backward. The distance deviation curve (as shown in Figure 12 ) and the heading angle deviation curve (as shown in Figure 13 ) in the control process, as well as the expected trajectory and actual trajectory comparison graph (as shown in Figure 14 ) are recorded. Among them, Figure 12 the horizontal coordinate represents the control step step, and the vertical coordinate represents the distance between the actual position and the expected position at the current step; Figure 13 the horizontal coordinate represents the control step step, and the vertical coordinate represents the heading angle deviation between the actual position and the expected position at the current step; Figure 14 the horizontal coordinate represents the x coordinate of the environment, and the vertical coordinate represents the y coordinate of the environment, Figure 14 in which line a refers to the curve composed of the expected x coordinate and the expected y coordinate of the unmanned surface vehicle at each control step, and line b refers to the curve composed of the actual x coordinate and the actual y coordinate of the unmanned surface vehicle at each control step. ​​

[0150] The results of Experiment 1 show that the controller's control of the unmanned surface vessel roughly conforms to the aforementioned dual-loop alternating control process of the heading and distance controllers: heading angle control is completed first, followed by distance control. Furthermore, the distance between corresponding points on the expected and actual trajectories is calculated, and the mean square value is also calculated. Mean square value is used as an indicator for evaluating the error of a control model. The calculation formula is as follows:

[0151] ;

[0152] in, For the first on the expected trajectory One point, For the actual trajectory of the first One point, For the number of trajectory points, This represents the geometric distance between two points. Figures 12-14 It can be obtained that in Experiment 1, for The error index for the trajectory tracking task is .

[0153] Experiment 2: Based on the tasks to be completed by the unmanned surface vessel (USV), a decision-making model was trained in subsequent tasks using more trajectory points (i.e., more decision steps). The final effect of the controller was then evaluated using the trajectory of one of the USVs provided by the trained model. Here, the number of trajectory points... The value is 500. Similarly, record the distance deviation curve during the control process (e.g., ...). Figure 15 (as shown) and heading angle deviation curve (as shown) Figure 16 As shown), and a comparison chart of the expected trajectory and the actual trajectory (as shown). Figure 17 (As shown). Among them, Figure 15 , Figure 16 as well as Figure 17 The horizontal axis represents the x-coordinate of the environment, and the vertical axis represents the y-coordinate of the environment. Figure 15 The lines in the diagram refer to the curves formed by the expected x-coordinate and expected y-coordinate of the unmanned surface vessel at each control step. Figure 16 The middle line refers to the curve composed of the actual x-coordinate and the actual y-coordinate of the unmanned surface vessel at each control step. Figure 17 It is Figure 15 and Figure 16 The result of plotting two curves in the same graph.

[0154] Depend on Figures 15-17 It can be seen that in Experiment 2, for The error index for the trajectory tracking task is The trajectory comparison chart shows that, under the control of the controller, the unmanned surface vessel can continuously and stably track multiple trajectory points while maintaining a low error.

[0155] Compared with the first experiment and the second experiment, the error of the controller is within an acceptable range, and the controller can still maintain stable tracking with low error under continuous multi-trajectory point tracking, which shows that the controller designed in the embodiments of the application is better adapted to the Unreal Engine UE4 and can be used in the control of unmanned ships by deep reinforcement learning.

[0156] Step (4), migration of the particle model to the simulation environment:

[0157] (4-1) Formation model setting. According to the multi-formation control target of the application, the leader-follower model is selected as the formation model. The core idea of the leader-follower model is to guide other followers to complete the formation task through a small number of leaders (usually intelligent agents with global target information). In this method, the leader is responsible for path planning, target point allocation, and maintenance of the global formation structure, and the follower adjusts its position and motion state to maintain the relative relationship with the leader through observation and perception of the leader.

[0158] In order to maintain a reasonable formation between the leader and the follower, the followers should be evenly distributed around the leader. Set the three-dimensional position of the unmanned aerial vehicle in a sub-formation as wherein, are the coordinates of the unmanned aerial vehicle on the , , axis respectively. The two-dimensional projection point position of the unmanned aerial vehicle is defined as the projection on the plane, that is, the sea level on which the unmanned ship is located, and the expression of the two-dimensional projection point position of the unmanned aerial vehicle is as follows:

[0159] ;

[0160] The unmanned aerial vehicle in the formation corresponds to the target point , and the expression of the target point is as follows:

[0161] ;

[0162] For the unmanned ship cluster in the sub-formation, the unmanned ship corresponds to the sub-target point position (expected position) , and the expression of the sub-target point position is as follows:

[0163] ;

[0164] To maintain a reasonable formation between the leader and the followers, the followers should be evenly distributed around the leader. Specifically, in the horizontal plane, the followers should be distributed on a circle with the leader as the center and a radius of . Set the initial angular offset , the calculation formula of the expected angle of each follower is as follows:

[0165] ;

[0166] wherein, is the total number of unmanned ships in the sub-platoon, is the expected radial distance, which is set to in this application, indicating the expected distance between each follower and the leader, i.e. the platoon radius. The expected position coordinates of the follower are as follows:

[0167] ;

[0168] Since the control of the unmanned plane actually only uses the point mass model, the discussion of the unmanned plane only stays in two dimensions, and the control speed expansion expression of the unmanned ship is as follows:

[0169] ;

[0170] wherein, the unmanned plane is characterized by the position and speed of the projection point of the unmanned plane leader, so as to simplify the unmanned plane to the same two-dimensional space as the unmanned ship for discussion.

[0171] (4-2) Environment setting under the point mass model. Three platoons are set, each with 5 unmanned ships, so there are 3 unmanned planes ( ) and 15 unmanned ships ( ) in total, i.e. 18 intelligent agents. According to the description in step (2), the discrete action space is set for the unmanned plane and the unmanned ship. The unmanned plane is designed with five control speeds , representing five decision-making actions of going forward, going left, going right, going backward, and staying in place; the unmanned ship is also designed with five control speeds , representing five decision-making actions of going forward, going left forward, going right forward, going backward, and staying in place. The control speed of the unmanned ship has been given, and the control speed of the unmanned plane is similar, with a speed amplitude of 0.75 m / s to achieve more stable platoon control. The control speed expansion expression of the unmanned plane is as follows:

[0172]

[0173] ;

[0174] ​The amplitude of the control speed here is smaller than the actual situation, which is due to the inertia of the unmanned vehicle in the simulation environment is larger, and a longer decision execution time is set, and in order to not make the displacement of each decision too large, a smaller amplitude of the control speed is selected. In the actual situation, that is, the real ship collects data for training, the amplitude of the control speed and the decision execution time can be adjusted according to the actual inertia of the unmanned vehicle.

[0175] In the particle model, without considering the influence of external force, after the unmanned vehicle makes a decision action selection, it is defaulted to move at a uniform speed in a straight line according to the control speed, that is, the position of the unmanned vehicle at the next decision execution time is equal to the expected position given by the calculation formula of the expected position of the unmanned vehicle . It should be noted that the particle model is a feature of the particle model environment, which does not consider the effect of external force, and the agent is simplified as a particle and its shape is ignored.

[0176] And in the setting of the environment, the starting position and target position of the unmanned vehicle are set for the three formations:

[0177] ;

[0178] ;

[0179] ;

[0180] According to the starting position and target position given by the above formula , , , it can be seen that the initial state of the three formations is triangularly distributed, and along the earth coordinate system axis from the negative direction to the positive direction, formation 1 is in the middle, and formations 2 and 3 are equally distributed on the two sides behind formation 1. Their target positions are respectively for formations 1, 2 and 3 to perform the tasks of moving forward, left forward and right forward. Only the original formation radius is replaced by the initial formation radius .

[0181] As can be seen from the above formula , , , the shortest displacement distance of each formation needs to move is 500m, considering the speed amplitude of the unmanned vehicle is 0.75m / s, the speed amplitude of the unmanned vehicle is 1m / s, and the decision execution time is 10s, according to the lower limit of the number of decision steps calculated at the slower unmanned vehicle speed, the maximum displacement of each decision is 7.5m (excluding the case of staying in place), therefore the number of decision steps At least 500m / 7.5m≈67 is required. However, to make the entire decision-making process more diverse and allow the drone to choose a strategy of remaining stationary for adjustments, this invention selects a larger number of decision steps. .

[0182] (4-3) The model is trained under a point mass model (environment). This application uses the optimized MAPPO algorithm as the deep learning algorithm. The unmanned surface vessel (USV) detects obstacles by using detection rays of a certain length, evenly distributed at the center of the USV, to achieve obstacle detection. For example... Figure 18 As shown, unmanned surface vessel There are a total Strips are distributed at equal angles, with a maximum length of The detection rays, in which, are illustrated Figure 18 Three lines (referred to by the letter 'c') are shorter than the maximum length. That is, there is an obstacle in that direction, and the distance to the obstacle is... , Indicates the first A detection ray, It refers to the vector pointing from the position of unmanned surface vessel i at time t to the outermost position of its first detection ray at time t.

[0183] In the improved MAPPO algorithm, each formation of drones and unmanned surface vessels (USVs) gradually explores and learns the optimal policy through E rounds of exploration. For example, in common reinforcement learning algorithms, the agent selects actions based on the current environmental state, the environment provides corresponding reward signals, and the agent optimizes the policy to maximize long-term cumulative rewards. Commonly used algorithms such as the Double Q network alleviate Q-learning overestimation by decoupling action selection and value evaluation; the PPO algorithm ensures policy update stability by pruning the objective function; and the TD3 algorithm further improves the performance of continuous control tasks by combining objective policy smoothing and the Double Q network mechanism. These methods improve sample efficiency and convergence stability at different levels. Since this application mainly discusses the transfer of the model to the simulation environment, the algorithm and training steps under the mass point model are not elaborated in detail. Other effective algorithms can be substituted, and the application is not limited to the examples mentioned in this application. The embodiments of this application are not described in detail here, and it is understood that the embodiments of this application are not limited in this respect.

[0184] Assuming a well-trained deep reinforcement learning model has been established, it can satisfy the following input states: the position and velocity of each UAV / boat in the formation at a certain moment, and the length of each detection ray of the UAV. The output should be the control velocity expansion expression for the UAV and the discrete action space of the UAV and UAV in the control velocity expansion expression for the UAV. Combining steps (4-1) and (4-2) to set the initial formation position, target position, and the presence of a certain number of obstacles, the model can achieve formation maintenance and collision avoidance tasks within the mass model. This well-trained model is then used. Migrating to the UE4 high-fidelity environment for fine-tuning training can reduce the long training time overhead caused by the controller's requirement for a decision execution time of dT=10s. This allows for obtaining satisfactory results in the simulation environment with fewer exploration rounds, thus efficiently completing multi-UAV / ship cooperative control tasks in high-fidelity environments.

[0185] (4-4) Model transfer training.

[0186] Set up a simulation environment similar to the mass environment in the UE4 platform, such as... Figure 19 As shown, the obstacles are now simplified to simple cubes and spheres. Figure 19 (The white cubes and spheres in the middle) Figure 19 The line segments within the white dashed boxes represent the detection rays of each unmanned surface vessel (USV). Therefore, the algorithm needs to receive information from UE4. It's particularly important to note that, unlike the point mass environment, the velocities and positions of the agents in the algorithm's input depend on the information transmitted from UE4. That is, the data sent by the TCP receiver, rather than the control speed and expected position set by the mass-point model:

[0187] ;

[0188] ;

[0189] The first two digits, 123, and the last two digits, 321, are both check characters.

[0190] In transfer learning, the algorithm model is first initialized by loading the model trained in step (4-3) and fine-tuning training begins. During training, for the t-th step policy of a certain exploration round e, based on the current policy, the control sequence is given by the control model designed in step (3-3) and transmitted through the TCP port. The process involves controlling the corresponding controller in a simulation environment and then collecting the states of multiple agents from the TCP port. The input is used to make the next decision in the deep reinforcement learning algorithm, and the strategy and model are fine-tuned and optimized according to the algorithm design.

[0191] The parameter setting of the fine-tuning process in the transfer learning only sets a small number of exploration rounds E, and other parameters related to the exploration efficiency need to be additionally set, so that the exploration efficiency is relatively low. The parameters that would change with the exploration round e are directly set to the annealing value of the last exploration round of the training process to meet the design concept of fine-tuning.

[0192] It should be noted that the decision of the controller in the particle model (environment) does not need to wait for the decision execution time; and the controller in the simulation environment needs to wait for the decision execution time at each step, but the simulation environment is closer to the real situation (more situations and external forces are considered). Therefore, by combining the two, the decision model is first trained under the assumption of the particle model, the decision model is retained, and the decision model is copied to the simulation environment for further fine-tuning instead of retraining, which can greatly speed up the training process.

[0193] (4-5) Transfer model verification. In terms of algorithm effect, the present application mainly uses the average total number of collisions and the average reward convergence value for measurement. The total number of collisions in the e-th exploration round includes the collisions between the unmanned aerial vehicles / boats and the collisions between the unmanned aerial vehicles / boats and the obstacles, and the average total number of collisions is obtained by averaging the total number of collisions in each exploration round . Obviously, the fewer the number of collisions, the more effectively the unmanned aerial vehicles / boats can avoid collisions, and the more excellent the algorithm performs in the strategy exploration effect. The reward average value of a round e is the average value of the two-dimensional matrix composed of the reward average value of all unmanned aerial vehicles / boats at all times in the round, which is directly calculated in full dimension and multiplied by an amplification coefficient to obtain:

[0194] ;

[0195] From the calculation formula of , it can be seen that the amplification coefficient is 100. Since the design of the reward function is according to the task target, the faster the reward rises, the higher the convergence value , and the more excellent the algorithm performs in the strategy exploration effect. In order to verify the effect of transfer training, the present application lists the average total number of collisions and the reward convergence value under formal training and transfer training for comparison, which are shown in Table 3 as follows:

[0196] Table 3: Comparison table of effects of formal training and transfer training

[0197]

[0198] The decision effect diagram under the UE4 simulation environment is as follows: Figure 20As shown, curves d, e and f are the trajectory curves of the UAVs, and the remaining curves are the trajectory curves of the USVs, and the white cubes and spheres are obstacles. In the decision-making process of the three formations, the controllers can better adapt to the environment and decision makers, keep the formation around the UAV, plan the path and avoid collision. This is specifically reflected in that the path of the USV basically avoids the obstacles while the path of the UAV changes almost synchronously; the undulating nature of the horizontally moving formation path is due to the fact that the UAV can directly select the control speed in front, back, left and right to always choose to quickly approach the target point to the right, but the USV only has the speed selection of the right lower and right upper, so its path will present the situation of turning back. Since the UE4 simulation environment is more faithful and complex than the particle model, it can be seen that the model effect has decreased to a certain extent: the reward convergence value only decreased by about 6%, and although the collision frequency is relatively high, this is because the original collision frequency of the former in the training environment is already small and close to the boundary. In summary Figure 20 , transfer learning can achieve a training effect close to that of the ideal particle model with fewer exploration rounds. Therefore, the experiment proves that applying the controller and transfer learning method in the simulation environment using "modeling-decision-making-transfer" can greatly reduce the training time and increase the training efficiency. Through theoretical innovation and engineering verification, the present application promotes the leap of multi-agent cooperative control from algorithm design to simulation application, and provides reliable technical support for the task execution of intelligent unmanned systems in complex dynamic environments.

[0199] It should be noted that the present embodiment only briefly and schematically illustrates the overall process of the multi-UAV and USV formation control method based on the Unreal Engine, and detailed descriptions of each step can be referred to the related content in the foregoing embodiments, which will not be repeated here. It can be understood that the present application does not limit this.

[0200] In summary, the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application aims to solve the problem of poor training fidelity of the multi-agent formation control model based on deep reinforcement learning under the traditional particle model and the insufficient real-time decision response performance of the trained model in a dynamic environment. To this end, the embodiments of the present application can achieve a relatively realistic multi-unmanned aerial vehicle and unmanned ship formation control effect by constructing a high-fidelity simulation environment and a formation control model based on the Unreal Engine, and a particle model and a high-fidelity simulation environment of the Unreal Engine dual verification system. Specifically, after verifying the convergence of the deep reinforcement learning algorithm in the particle model, the strategy network parameters of the deep reinforcement learning algorithm are mapped to the simulation environment through transfer learning, and data interaction is realized in combination with the TCP protocol. The above experiments show that after introducing the simulation environment, the formation control model and transfer learning, the agent formation scene can maintain good decision-making effect, verifying the feasibility of the algorithm from the theoretical model to the engineering application. Through the technical closed loop of “modeling-decision-transfer”, the embodiments of the present application provide an extensible solution and a more optimal decision-making model for multi-unmanned aerial vehicle and unmanned ship cooperative control, and have important application value in complex scenes such as disaster rescue.

[0201] The multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application utilizes the high simulation and multi-platform cooperative test environment provided by the Unreal Engine (UE4) platform itself, and designs an alternating control of the heading angle controller and the distance controller according to the characteristics of the platform, which can maintain stable tracking with low error under continuous multi-trajectory point tracking, thereby building a necessary and solid bridge for transferring the particle model of the unmanned ship to the Unreal Engine simulation platform to achieve more realistic control effect. Specifically, the embodiments of the present application realize the transfer of the control strategy by constructing the transfer process of the particle model to the Unreal Engine simulation environment, improve the model training efficiency in the simulation environment, and at the same time retain high performance and precision of the model, thereby confirming the effectiveness and feasibility of the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application in the field of multi-unmanned aerial vehicle and unmanned ship cooperative control.

[0202] For reference Figure 21 The embodiments of the present application also provide a multi-unmanned aerial vehicle and unmanned ship formation control device 2100 based on the Unreal Engine, which can implement the above method. The device comprises the following modules:

[0203] A kinematic model construction module 2101 is configured to construct a kinematic model of a multi-agent formation. The multi-agent formation comprises a formation of unmanned aerial vehicles and a formation of unmanned ships.

[0204] An experimental simulation environment construction module 2102 is configured to construct a first experimental simulation environment of the multi-agent formation based on the Unreal Engine.

[0205] The unmanned ship control model construction module 2103 is configured to construct a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematics model, wherein the double-loop alternating unmanned ship control model comprises a heading controller and a distance controller.

[0206] The deep reinforcement learning model pre-training module 2104 is configured to construct a particle model environment and pre-train a deep reinforcement learning model in the particle model environment.

[0207] The model migration fine-tuning training module 2105 is configured to migrate the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tune the trained deep reinforcement learning model in combination with the kinematics model and the double-loop alternating unmanned ship control model, and obtain a multi-agent formation control model.

[0208] The multi-agent formation control model acquisition module 2106 is configured to deploy the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification, wherein the multi-agent formation control model is used to control the multi-agent formation.

[0209] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiments, the present device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0210] The present application also provides an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0211] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiments, the present device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0212] Please refer to Figure 22 , Figure 22 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0213] The processor 2201 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0214] The memory 2202 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 2202 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 2202 and are called and executed by the processor 2201 to implement the above-mentioned method of the embodiments of the present application.

[0215] The input / output interface 2203 is configured to realize information input and output.

[0216] The communication interface 2204 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0217] The bus 2205 is configured to transmit information between the components (for example, the processor 2201, the memory 2202, the input / output interface 2203, and the communication interface 2204) of the device.

[0218] The processor 2201, the memory 2202, the input / output interface 2203, and the communication interface 2204 are connected to each other through the bus 2205 to realize the communication connection between the components in the device.

[0219] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned method.

[0220] It can be understood that the above-mentioned method embodiments are applicable to the storage medium embodiments. The storage medium embodiments specifically implement the functions of the above-mentioned method embodiments, and achieve the same beneficial effects as the above-mentioned method embodiments.

[0221] The embodiments of the present application further provide a computer program product, which includes a computer program. The computer program is executed by a processor to implement the above-mentioned method.

[0222] It can be understood that the contents in the above method embodiments are all applicable to the present program product embodiments, the present program product embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0223] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0224] The method and device for controlling a multi- UAV and unmanned ship formation based on an Unreal Engine provided by the embodiments of the present application, by utilizing the high simulation and multi-platform collaborative test environment provided by the Unreal Engine platform itself, combining with the kinematic model, designs the alternate control of the heading controller and the distance controller for the characteristics of the Unreal Engine platform, can still maintain stable tracking with low error under continuous multi- trajectory point number trajectory tracking, builds a necessary and solid bridge for the point mass model of the unmanned ship to transfer to the Unreal Engine simulation platform to realize more realistic control effect; and through the migration process of the point mass model to the Unreal Engine simulation environment, the migration of the control strategy is realized, the model training efficiency of the deep reinforcement learning model in the simulation environment is improved, while the high performance and precision of the deep reinforcement learning model are reserved, and the good decision effect of the deep reinforcement learning model in the agent formation scene is improved.

[0225] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A method for controlling a multi-UAV and unmanned surface vehicle formation based on Unreal Engine, characterized in that, The method comprises the following steps: constructing a kinematics model of a multi-agent formation; wherein the multi-agent formation comprises a UAV formation and an unmanned ship formation; constructing a first experimental simulation environment of the multi-agent formation based on an Unreal Engine; constructing a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematics model; wherein the double-loop alternating unmanned ship control model comprises a heading controller and a distance controller; constructing a particle model environment and pre-training a deep reinforcement learning model in the particle model environment; migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tuning the trained deep reinforcement learning model in combination with the kinematics model and the double-loop alternating unmanned ship control model, and obtaining a multi-agent formation control model; deploying the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation; wherein the method further comprises the step of executing the control of the double-loop alternating unmanned ship control model, which comprises: during the process of the unmanned ship performing a task, calculating the expected position of the unmanned ship according to the decision control instruction generated by the multi-agent formation control model for the unmanned ship at the current time; wherein the decision control instruction comprises a decision speed and a decision execution time; calculating the expected heading angle between the expected position and the current position of the unmanned ship; correcting the expected heading angle and calculating the target heading angle deviation between the corrected expected heading angle and the actual heading angle of the unmanned ship at the current time; when the absolute value of the target heading angle deviation is greater than a preset heading angle deviation threshold, activating the heading controller to control the yaw angle of the unmanned ship; when the absolute value of the target heading angle deviation is less than the preset heading angle deviation threshold, activating the distance controller to control the distance of the unmanned ship.

2. The method of claim 1, wherein, The construction of the kinematics model of the multi-agent formation comprises: determining the UAV coordinate system and the UAV movement mode of the UAV formation; determining the unmanned ship coordinate system and the unmanned ship movement mode of the unmanned ship formation; constructing the kinematics model corresponding to the UAV formation according to the UAV coordinate system and the UAV movement mode; constructing the kinematics model corresponding to the unmanned ship formation according to the unmanned ship coordinate system and the unmanned ship movement mode.

3. The method of claim 1, wherein, The construction of the first experimental simulation environment of the multi-agent formation based on the Unreal Engine comprises: creating a new level in the Unreal Engine and importing a water and ocean scene in the level; creating a UAV blueprint class in the Unreal Engine and importing a UAV mesh model in the UAV blueprint class to obtain a UAV blueprint; creating an unmanned ship blueprint class in the Unreal Engine and importing an unmanned ship mesh model in the unmanned ship blueprint class to obtain an unmanned ship blueprint; Import the drone blueprint and the unmanned ship blueprint into the water marine scene, and complete construction of the first experimental simulation environment.

4. The method of claim 1, wherein, The deep reinforcement learning model trained in the particle model environment is migrated to the first experimental simulation environment, and the trained deep reinforcement learning model is fine-tuned in combination with the kinematics model and the double-loop alternating unmanned ship control model to obtain a multi-agent formation control model, including: The deep reinforcement learning model trained in the particle model environment is migrated to the first experimental simulation environment, and the control sequence generated by the double-loop alternating unmanned ship control model is obtained, and the motion state of the multi-agent formation in the kinematics model is obtained; The control sequence and the motion state are input into the trained deep reinforcement learning model for fine-tuning to obtain the multi-agent formation control model.

5. The method of claim 1, wherein, The multi-agent formation control model is deployed in a second experimental simulation environment different from the first experimental simulation environment for verification, including: A second experimental simulation environment different from the first experimental simulation environment is constructed based on the Unreal Engine; The multi-agent formation control model is deployed in the second experimental simulation environment, and the multi-agent formation control model is verified, and the average total number of collisions and the average reward convergence value in the verification process are obtained; According to the average total number of collisions and the average reward convergence value, it is determined whether the multi-agent formation control model passes the model verification, and if the multi-agent formation control model passes the model verification, the multi-agent formation control model that passes the verification is used to dynamically control the multi-agent formation performing a task.

6. A multi-UAV and unmanned surface vehicle formation control device based on Unreal Engine, for implementing the method of any one of claims 1 to 5, characterized in that, The device includes the following modules: A kinematics model construction module is configured to construct a kinematics model of a multi-agent formation; wherein the multi-agent formation includes a drone formation and an unmanned ship formation; An experimental simulation environment construction module is configured to construct a first experimental simulation environment of the multi-agent formation based on the Unreal Engine; An unmanned ship control model construction module is configured to construct a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematics model; wherein the double-loop alternating unmanned ship control model includes a heading controller and a distance controller; A deep reinforcement learning model pre-training module is configured to construct a particle model environment and pre-train a deep reinforcement learning model in the particle model environment; A model migration and fine-tuning training module is configured to migrate the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, and fine-tune the trained deep reinforcement learning model in combination with the kinematics model and the double-loop alternating unmanned ship control model to obtain a multi-agent formation control model; A multi-agent formation control model acquisition module is configured to deploy the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation; The device is further configured to: In the process that the unmanned ship executes a task, a decision control instruction for the unmanned ship at a current time is generated according to the multi-agent formation control model, and an expected position of the unmanned ship is calculated; the decision control instruction comprises a decision speed and a decision execution time; An expected heading angle between the expected position and a current position of the unmanned ship is calculated; The expected heading angle is corrected, and a target heading angle deviation between the corrected expected heading angle and an actual heading angle of the unmanned ship at the current time is calculated; When an absolute value of the target heading angle deviation is greater than a preset heading angle deviation threshold, a yaw angle controller is activated to control the unmanned ship; When the absolute value of the target heading angle deviation is less than the preset heading angle deviation threshold, a distance controller is activated to control the unmanned ship.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method in any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 5.

Images