Multi-unmanned aerial vehicle and unmanned ship formation control method and device based on unreal engine

Abstract

The invention discloses a multi-unmanned aerial vehicle and unmanned ship formation control method and device based on an unreal engine. The method comprises the following steps: constructing a kinematics model of multi-agent formation; constructing a first experiment simulation environment; constructing a double-ring alternating unmanned ship control model; the deep reinforcement learning model trained in the mass point model environment is migrated to a first experimental simulation environment, and fine tuning training is performed on the trained deep reinforcement learning model in combination with the kinematic model and the double-ring alternating unmanned ship control model to obtain a multi-agent formation control model; deploying the multi-agent formation control model in a second experimental simulation environment for verification; and the multi-agent formation control model is used for realizing control of multi-agent formation. According to the method, stable tracking with low errors can still be kept under trajectory tracking of continuous multi-trajectory points, the training efficiency of the model in a simulation environment is improved, high performance and precision of the model are kept, the decision effect is improved, and the method can be widely applied to the technical field of multi-agent control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent control, and in particular to a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method and device based on an Unreal Engine. BACKGROUND

[0002] At present, deep reinforcement learning is widely used in multi-agent control and decision-making scenarios. In the problem of multi-agent cooperative formation control, the core control objectives include the gathering of unmanned aerial vehicles, the formation of a formation, and the stable maintenance of the formation structure during movement. The multi-agent formation control method based on deep reinforcement learning not only can learn dynamic behaviors in complex environments through data-driven methods, but also can exhibit stronger adaptability and robustness when facing high-dimensional state spaces and diversified task requirements. These characteristics make the formation control model based on deep reinforcement learning have certain advantages in handling collaborative tasks of large-scale multi-agent systems. However, the multi-agent formation control model based on deep reinforcement learning has the problems of poor training fidelity under the traditional particle model and insufficient real-time decision response performance of the trained model in dynamic environments.

[0003] In summary, the technical problems in the related art need to be improved. SUMMARY

[0004] Embodiments of the present application aim to at least partially solve one of the technical problems in the related art. To this end, the main purpose of the embodiments of the present application is to propose a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method and device based on an Unreal Engine, which can maintain stable tracking with low error under continuous multi-trajectory point trajectory tracking, and at the same time, improve the model training efficiency in the simulation environment, retain high performance and accuracy of the model, and improve the decision-making effect of the model in the agent formation scenario.

[0005] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on an Unreal Engine, which comprises the following steps: constructing a kinematic model of a multi-agent formation; wherein the multi-agent formation includes an unmanned aerial vehicle formation and an unmanned surface vehicle formation; constructing a first experimental simulation environment of the multi-agent formation based on the Unreal Engine; constructing a double-loop alternating unmanned surface vehicle control model based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned surface vehicle control model includes a heading controller and a distance controller; constructing a particle model environment and pre-training a deep reinforcement learning model in the particle model environment; The deep reinforcement learning model trained in the particle model environment is migrated to the first experimental simulation environment, and the trained deep reinforcement learning model is fine-tuned in combination with the kinematic model and the double-loop alternating unmanned ship control model to obtain a multi-agent formation control model. The multi-agent formation control model is deployed in a second experimental simulation environment different from the first experimental simulation environment for verification, and the multi-agent formation control model is used to control the multi-agent formation.

[0006] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a multi-unmanned aerial vehicle and unmanned ship formation control device based on Unreal Engine, which comprises the following modules: A kinematic model construction module is configured to construct a kinematic model of a multi-agent formation, wherein the multi-agent formation comprises an unmanned aerial vehicle formation and an unmanned ship formation. An experimental simulation environment construction module is configured to construct a first experimental simulation environment of the multi-agent formation based on Unreal Engine. An unmanned ship control model construction module is configured to construct a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematic model, wherein the double-loop alternating unmanned ship control model comprises a heading controller and a distance controller. A deep reinforcement learning model pre-training module is configured to construct a particle model environment and pre-train a deep reinforcement learning model in the particle model environment. A model migration and fine-tuning training module is configured to migrate the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tune the trained deep reinforcement learning model in combination with the kinematic model and the double-loop alternating unmanned ship control model, and obtain a multi-agent formation control model. A multi-agent formation control model acquisition module is configured to deploy the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification, and the multi-agent formation control model is used to control the multi-agent formation.

[0007] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the above-mentioned method.

[0008] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned method.

[0009] To achieve the above object, another aspect of the embodiment of the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the above method.

[0010] The embodiment of the present application at least has the following beneficial effects: the present application provides a multi-UAV and unmanned ship formation control method and device based on Unreal Engine, which constructs a kinematic model of multi-agent formation; wherein the multi-agent formation includes a UAV formation and an unmanned ship formation; a first experimental simulation environment of the multi-agent formation is constructed based on Unreal Engine; a double-loop alternating unmanned ship control model is constructed based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned ship control model includes a heading controller and a distance controller; a particle model environment is constructed, and a deep reinforcement learning model is pre-trained in the particle model environment; the deep reinforcement learning model trained in the particle model environment is migrated to the first experimental simulation environment, and the trained deep reinforcement learning model is fine-tuned in combination with the kinematic model and the double-loop alternating unmanned ship control model to obtain a multi-agent formation control model; the multi-agent formation control model is deployed in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation. The embodiment of the present application uses the high-simulation and multi-platform collaborative test environment provided by the Unreal Engine platform itself, and designs the alternating control of the heading controller and the distance controller in combination with the kinematic model according to the characteristics of the Unreal Engine platform, which can still maintain stable tracking with low error under continuous multi-trajectory point tracking, and builds a necessary and solid bridge for the particle model of the unmanned ship to be transferred to the Unreal Engine simulation platform to achieve more realistic control effect. Moreover, through the migration process from the particle model to the Unreal Engine simulation environment, the migration of the control strategy is realized, the model training efficiency of the deep reinforcement learning model in the simulation environment is improved, the high performance and precision of the deep reinforcement learning model are reserved, and the good decision-making effect of the deep reinforcement learning model in the multi-agent formation scenario is improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a step flowchart of the multi-UAV and unmanned ship formation control method based on Unreal Engine provided by the embodiment of the present application; Figure 2 is a flowchart of the multi-UAV and unmanned ship formation control method based on Unreal Engine provided by the embodiment of the present application; Figure 3 is a schematic diagram of a UAV coordinate system provided by the embodiment of the present application; Figure 4 is a schematic diagram of a UAV control speed design provided by the embodiment of the present application; Figure 5is a plan view of a coordinate system of an unmanned ship provided by an embodiment of the present application; Figure 6 is a schematic diagram of an unmanned ship control speed design provided by an embodiment of the present application; Figure 7 is a right rear view of an unmanned aerial vehicle grid model provided by an embodiment of the present application; Figure 8 is a right rear view of an unmanned ship grid model provided by an embodiment of the present application; Figure 9 is a top view of an unmanned ship grid model provided by an embodiment of the present application; Figure 10 is a schematic diagram of an unmanned ship control flow provided by an embodiment of the present application; Figure 11 is a schematic diagram of an internal flow of an unmanned ship PID controller provided by an embodiment of the present application; Figure 12 is a distance deviation curve diagram in experiment one provided by an embodiment of the present application; Figure 13 is a heading angle deviation curve diagram in experiment one provided by an embodiment of the present application; Figure 14 is a comparison diagram of expected trajectory and actual trajectory in experiment one provided by an embodiment of the present application; Figure 15 is an expected trajectory diagram in experiment two provided by an embodiment of the present application; Figure 16 is an actual trajectory diagram in experiment two provided by an embodiment of the present application; Figure 17 is a comparison diagram of expected trajectory and actual trajectory in experiment two provided by an embodiment of the present application; Figure 18 is a schematic diagram of obstacle detection of an unmanned ship provided by an embodiment of the present application; Figure 19 is a schematic diagram of each formation and environment layout in Unreal Engine provided by an embodiment of the present application; Figure 20 is a decision trajectory diagram obtained after fine tuning training in Unreal Engine provided by an embodiment of the present application; Figure 21 is a structure schematic diagram of a multi-unmanned aerial vehicle and unmanned ship formation control device based on Unreal Engine provided by an embodiment of the present application; Figure 22 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0012] For the purpose of making the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with embodiments of the present application. They are merely examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0013] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0014] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0016] In recent years, deep reinforcement learning has been widely applied to multi-agent control and decision-making scenarios. In the problem of multi-agent cooperative formation control, the core control objectives include the gathering of unmanned aerial vehicles, the formation of a formation, and the stable maintenance of the formation structure during movement. Currently, the most widely used methods mainly include the leader-follower method, the artificial potential field method, and the kinematics-based control method.

[0017] The leader-follower method is a classic and widely used strategy in multi-agent cooperative formation control. The core idea is to guide other followers to complete the formation task through a small number of leaders (usually agents with global target information). In this method, the leader is responsible for path planning, target point assignment, and maintaining the global formation structure, while the follower adjusts its position and motion state to maintain the relative relationship with the leader through observation and perception of the leader. Usually, this method ensures that the formation structure always meets a specific geometric shape during motion by pre-setting fixed relative distance or angle constraints. The leader-follower method has the advantages of simple implementation and high computational efficiency, especially in scenarios with a small number of agents. However, it has high dependence on leaders, and once the leader fails or makes errors in a complex environment, the entire formation may not function properly. In addition, as the number of agents increases, the information interaction and collaboration requirements between followers significantly increase, limiting the applicability of the method in large-scale systems.

[0018] The artificial potential field method is a multi-agent cooperative control method based on virtual force fields, widely used in formation control, obstacle avoidance, and path planning tasks. The core idea of this method is to construct a virtual potential field for agents, where target points, obstacles, and other agents are modeled as attractive and repulsive sources to guide the motion behavior of agents. Specifically, the target point exerts an attractive force on the agent, causing it to move towards the target direction, while obstacles or other agents exert a repulsive force to avoid collisions or maintain appropriate formation structure. The resultant force of the agent is obtained by vector superposition of the attractive and repulsive forces, thereby determining its next motion direction and speed. The artificial potential field method is simple to implement and computationally efficient, especially in local environment modeling and single-agent obstacle avoidance. However, the parameters of the potential field function need to be carefully designed and adjusted to adapt to different tasks or environments. In addition, the traditional potential field method is prone to local optimal problems, especially in complex obstacle environments, where agents may be trapped in a certain position and unable to move forward. Furthermore, when the number of agents is large, the complexity of the repulsive force interaction increases significantly, which may lead to unstable formation structure or degraded control performance.

[0019] Kinematics-based control method is a kind of multi-agent cooperative control method with kinematics model as the core. By directly describing the kinematics variables such as position, velocity and acceleration of agents, the formation, maintenance and trajectory tracking tasks can be achieved. This method usually takes the preset trajectory or formation constraint as the target, decouples the position relationship and velocity coupling relationship between agents, and designs the control law to ensure that each agent moves along the expected path and maintains the geometric shape of the formation. The form of kinematics model is usually differential equation, which describes the change of agent's motion state with time and can accurately reflect the motion characteristics of agent. Kinematics-based control method has the characteristics of high control accuracy and rigorous theoretical basis, especially in static and simple dynamic environment. However, its high dependence on physical model of agent limits its adaptability to complex environment. When the number of agents increases or the complexity of environment increases, the system needs higher computing resources to update the kinematics constraint in real time, which affects the control efficiency. In addition, this method usually assumes that agents can accurately execute control instructions, but in actual scenarios, the dynamics limitations and environmental uncertainties of agents may cause deviations, which affects the stability of formation.

[0020] The leader-follower method, artificial potential field method, and kinematics-based control method of the above examples are traditional formation control techniques that have provided a solid theoretical foundation for subsequent research due to their excellent performance in precise formation control tasks. However, these classic methods are significantly effective in scenarios with a small number of agents, but their adaptability to large-scale formations is clearly limited. When the number of agents increases, the complexity of the system's dynamic modeling increases rapidly, and the computational load grows exponentially, leading to a significant decline in the feasibility and efficiency of real-time control. To address these issues, multi-agent formation control methods based on deep reinforcement learning (DRL) have gradually become a research hotspot in recent years. Compared to traditional methods, deep reinforcement learning not only learns dynamic behavior in complex environments through data-driven methods, but also exhibits greater adaptability and robustness when faced with high-dimensional state spaces and diverse task requirements. These characteristics make deep reinforcement learning-based formation control models advantageous in handling large-scale multi-agent system coordination tasks. Additionally, many deep reinforcement learning-based models are not completely independent of traditional methods, often incorporating classic control techniques such as the leader-follower model or artificial potential field method to improve control effectiveness. In terms of the environment for deploying agents, control of agents is primarily based on the point mass model, which assumes that after making a decision to act, agents move at a constant speed in a straight line, i.e., their position at the next decision execution time only depends on the execution time and speed. This assumption simplifies the environment, but it also ignores the effects of external forces, making it difficult to effectively model real-world situations. Therefore, in addition to the initial point mass model, algorithms need to complete multi-agent tasks in environments with higher realism to achieve better practical results. In terms of control, deep control models, i.e., deep learning control models, are a type of model widely used in the control system field in recent years. These models use deep neural networks to model the dynamic behavior of complex systems and use these models for system control. The core advantage of deep learning control is its powerful non-linear modeling capability, which can automatically learn system features from a large amount of data, providing solutions for complex systems where traditional control methods are ineffective. UE4 (Unreal Engine 4), which was originally used for game development, has integrated various environment models and dynamic models in recent years and is gradually being used for physical simulation and visualization. The high-fidelity environment built in UE4 can perform controller performance verification well and provide accurate kinematics for multi-agent cooperative control.However, the multi-agent formation control model based on deep reinforcement learning has the problems of poor training fidelity under the traditional particle model and insufficient real-time decision response performance of the trained model in a dynamic environment.

[0021] Therefore, in the embodiments of the present application, a multi-unmanned aerial vehicle and unmanned surface vehicle formation control method and device based on Unreal Engine are provided. The scheme uses the high simulation and multi-platform cooperative test environment provided by the Unreal Engine platform itself, and designs an alternating control of the heading controller and the distance controller in combination with the kinematic model for the characteristics of the Unreal Engine platform. The scheme can maintain stable tracking with low error under continuous multi-trajectory point trajectory tracking, and builds a necessary and solid bridge for transferring the particle model of the unmanned surface vehicle to the Unreal Engine simulation platform to achieve more realistic control effect. Furthermore, the scheme realizes the migration of the control strategy by constructing the migration process of the particle model to the Unreal Engine simulation environment, improves the model training efficiency of the deep reinforcement learning model in the simulation environment, retains the high performance and precision of the deep reinforcement learning model, and improves the good decision effect of the deep reinforcement learning model in the agent formation scenario.

[0022] The multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine provided in the embodiments of the present application relates to the technical field of multi-agent control. The multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto. The server end can be configured as a standalone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network. The software can be an application that implements the multi-unmanned aerial vehicle and unmanned surface vehicle formation control method based on Unreal Engine, and the like, but is not limited to the above forms.

[0023] The application is operable with numerous general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0024] Referring to Figure 1 , Figure 1 is an optional step flowchart of a method for multi-UAV and unmanned ship formation control based on Unreal Engine provided by an embodiment of the application, Figure 1 The method in the embodiment can include, but is not limited to, steps S101 to S106.

[0025] Step S101, constructing a kinematic model of a multi-agent formation; wherein the multi-agent formation includes a UAV formation and an unmanned ship formation; In some embodiments, step S101 can include: determining a UAV coordinate system and a UAV movement mode of the UAV formation; determining an unmanned ship coordinate system and an unmanned ship movement mode of the unmanned ship formation; constructing a kinematic model corresponding to the UAV formation according to the UAV coordinate system and the UAV movement mode; and constructing a kinematic model corresponding to the unmanned ship formation according to the unmanned ship coordinate system and the unmanned ship movement mode.

[0026] In the embodiment of the application, the kinematic model is the basis for multi-agent formation control, which accurately describes the movement law of the UAV and the unmanned ship in space, including the relationship between the position, velocity, acceleration and other parameters and time. By constructing the kinematic model of the multi-agent formation, the movement characteristics of each agent can be clearly understood, providing a theoretical basis for designing an effective control strategy, ensuring that the formation moves according to the expected trajectory and formation during the movement process, and ensuring the smooth execution of the formation task.

[0027] Step S102, constructing a first experimental simulation environment of the multi-agent formation based on Unreal Engine; In some embodiments, step S102 can include: creating a new level in the Unreal Engine and importing a water body marine scene in the level; creating a drone blueprint class in the Unreal Engine and importing a drone mesh model in the drone blueprint class to obtain a drone blueprint; creating an unmanned surface vehicle blueprint class in the Unreal Engine and importing an unmanned surface vehicle mesh model in the unmanned surface vehicle blueprint class to obtain an unmanned surface vehicle blueprint; importing the drone blueprint and the unmanned surface vehicle blueprint into the water body marine scene to complete construction of the first experimental simulation environment.

[0028] In the embodiments of the present application, by constructing the first experimental simulation environment of the multi-agent formation, a near-real experimental condition can be provided for the multi-agent formation, so that researchers can fully test and verify the control algorithm and strategy of the formation without the need to actually build an expensive and complex physical experimental platform, greatly reducing the experimental cost, improving the research and development efficiency, and at the same time, problems that can be encountered in the actual environment can be found and solved in advance.

[0029] Step S103, constructing a double-loop alternating unmanned surface vehicle control model based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned surface vehicle control model includes a heading controller and a distance controller. In some embodiments, the step of executing control of the double-loop alternating unmanned surface vehicle control model can include: during execution of a task by the unmanned surface vehicle, calculating an expected position of the unmanned surface vehicle according to a decision control instruction for the unmanned surface vehicle at a current time generated by the multi-agent formation control model; wherein the decision control instruction includes a decision speed and a decision execution time; calculating an expected heading angle between the expected position and a current position of the unmanned surface vehicle; correcting the expected heading angle and calculating a target heading angle deviation between the corrected expected heading angle and an actual heading angle of the unmanned surface vehicle at the current time; when an absolute value of the target heading angle deviation is greater than a preset heading angle deviation threshold, activating the heading controller to control a yaw angle of the unmanned surface vehicle; and when the absolute value of the target heading angle deviation is less than the preset heading angle deviation threshold, activating the distance controller to control a distance of the unmanned surface vehicle.

[0030] In the embodiments of the present application, in the control of the UAV, because the kinetic model is simplified, the speed given by the decision network (also known as a decision maker, i.e., a multi-agent formation control model) is directly used to control the UAV, without the need for additional design of a control model. In the control of the USV, considering that the decision space is five discrete control speeds, but the value and effect of the decision are mainly based on the position of the USV, only controlling the speed of the USV will introduce an error that is difficult to estimate in the decision evaluation part. Therefore, the present application chooses to directly control the position of the USV using a controller, and assumes that the USV moves at a constant speed in a straight line according to the decision speed within the decision execution time. Specifically, the present application uses a double-loop alternating control of a heading controller and a distance controller to enable the USV to complete the task of reaching the expected position.

[0031] In the embodiments of the present application, the double-loop alternating USV control model decomposes the control of the USV into two relatively independent and mutually cooperative subsystems, i.e., heading control and distance control. Through the alternating control of the heading controller and the distance controller, stable tracking with low error can be maintained under continuous multi-trajectory point tracking. The heading controller is responsible for ensuring that the USV travels according to the predetermined heading, ensuring the consistency of the direction of travel of the formation as a whole. The distance controller precisely controls the distance between the USV and other members in the formation or target points, maintaining the stability of the formation shape. This double-loop alternating control method can more flexibly and accurately cope with complex and variable marine environments, improve the control accuracy and robustness of the USV formation, and enable the formation to better adapt to various dynamic changes when performing tasks.

[0032] Step S104, constructing a particle model environment, and pre-training a deep reinforcement learning model in the particle model environment; In the embodiments of the present application, the particle model environment simplifies the physical properties of the multi-agent, treats the agent as a particle, and removes the influence of complex shapes, sizes, and other factors on training, thereby greatly reducing the computational complexity and accelerating the training speed of the deep reinforcement learning model. Pre-training in the particle model environment enables the model to quickly learn basic control strategies and behavior patterns, laying a good foundation for subsequent fine-tuning training in a more complex simulation environment, improving the overall training efficiency, and reducing training time and resource consumption.

[0033] Step S105, migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tuning the trained deep reinforcement learning model in combination with the kinematic model and the double-loop alternating USV control model, and obtaining a multi-agent formation control model; In some embodiments, step S105 can include: migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, obtaining the control sequence generated by the double-ring alternating unmanned ship control model, and obtaining the motion state of the multi-agent formation in the kinematic model; inputting the control sequence and the motion state into the trained deep reinforcement learning model for fine-tuning training to obtain the multi-agent formation control model.

[0034] In the embodiments of the present application, by migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment closer to the real environment for fine-tuning training, the general knowledge learned in the pre-training stage can be fully utilized, and at the same time, the detailed physical information and complex scenes in the first experimental simulation environment are combined to make the model further learn the control strategy suitable for the actual environment. Combined with the kinematic model and the double-ring alternating unmanned ship control model, the model can ensure accurate control of the formation heading and distance on the basis of following the motion law of the agent, and finally obtain a multi-agent formation control model with stronger adaptability and control precision, which can better cope with various challenges in actual tasks.

[0035] Step S106, deploying the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation.

[0036] In some embodiments, step S106 can include: constructing a second experimental simulation environment different from the first experimental simulation environment based on the Unreal Engine; deploying the multi-agent formation control model in the second experimental simulation environment to verify the multi-agent formation control model, and obtaining the average total number of collisions and the average reward convergence value in the verification process; determining whether the multi-agent formation control model passes the model verification according to the average total number of collisions and the average reward convergence value, and if the multi-agent formation control model passes the model verification, dynamically controlling the multi-agent formation performing the task with the verified multi-agent formation control model.

[0037] The main difference between the first experimental simulation environment for training and the second experimental simulation environment for verification includes but is not limited to: the number and distribution of obstacles in the experimental simulation environment. It should be noted that the setting of the experimental simulation environment can be set according to the actual application, and the embodiments of the present application do not limit this.

[0038] In the embodiments of the present application, the generalization ability and robustness of the multi-agent formation control model can be comprehensively tested by verifying in a second experimental simulation environment different from the first experimental simulation environment. Through testing under different scenarios and conditions, it can be ensured that the model can stably and reliably operate under various unknown environments and effectively control the multi-agent formation to complete the predetermined task. The model verification process is conducive to discovering potential problems that may exist in the model in actual application, providing a direction for further optimization and improvement of the model, and ultimately improving the success rate and reliability of the entire multi-agent formation system in actual tasks.

[0039] The steps S101 to S106 shown in the embodiments of the present application use the high simulation and multi-platform collaborative test environment provided by the Unreal Engine platform itself, combine the kinematic model, and design the alternate control of the heading controller and the distance controller according to the characteristics of the Unreal Engine platform, so that stable tracking with low error can be maintained under continuous multi-trajectory point tracking, thereby building a necessary and solid bridge for the point model of the unmanned ship to be transferred to the Unreal Engine simulation platform to achieve more realistic control effect. Moreover, by constructing the migration process of the point model to the Unreal Engine simulation environment, the migration of the control strategy is realized, the model training efficiency of the deep reinforcement learning model in the simulation environment is improved, the high performance and precision of the deep reinforcement learning model are retained, and the good decision-making effect of the deep reinforcement learning model in the agent formation scenario is improved.

[0040] To explain the principle of the technical scheme of the present application, the overall process of the present application will be described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present application and cannot be regarded as a limitation of the present application.

[0041] Please refer to Figure 2 For example Figure 2As shown, the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided by the embodiment of the application mainly includes four steps, which are: (1) design the unmanned aerial vehicle / ship kinematics model: establish the coordinate system of the unmanned aerial vehicle / ship, simplify the freedom of unmanned aerial vehicle / ship movement according to the actual situation, and design a reasonable control speed for the unmanned aerial vehicle / ship; (2) build a multi-unmanned aerial vehicle and unmanned ship formation environment: the environment is built in the Unreal Engine, and the wave model of the environment needs to be designed, as well as the use of three-dimensional modeling of the unmanned aerial vehicle / ship and the design of the mechanical model; (3) design the unmanned aerial vehicle / ship control model: design the double-loop alternating control of the heading controller and the distance controller of the unmanned ship, build the data interaction between the Unreal Engine and the controller, and ensure that the unmanned ship completes the task of reaching the expected position; (4) migrate the particle model to the simulation environment: design a simulation environment similar to the particle model environment in the Unreal Engine, fine-tune the deep reinforcement learning model trained in the particle model in the simulation environment, and finally verify the model in the simulation environment different from the fine-tuning training based on the Unreal Engine. In the embodiment of the application, the Unreal Engine used is UE4, and hereinafter the Unreal Engine can be described as UE4.

[0042] As shown in Figure 2 , the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided by the embodiment of the application builds a high simulation environment based on the Unreal Engine, designs the dynamics model of the unmanned aerial vehicle / ship and the corresponding controller, and realizes the migration of the deep reinforcement learning algorithm from the particle model to the simulation environment. The specific implementation process is as follows: Step (1), design the kinematics model of the unmanned aerial vehicle / ship: (1-1) The four-rotor unmanned aerial vehicle and the double-propeller unmanned ship are selected as the dynamics model of the unmanned aerial vehicle and the unmanned ship, respectively, in the embodiment of the application.

[0043] (1-2) The unmanned aerial vehicle coordinate system is selected, which is divided into the earth coordinate system E and the body coordinate system B.

[0044] Among them, the earth coordinate system E, also known as the inertial coordinate system, takes a fixed point on the ground as the origin , takes the positive east direction as the positive direction of the axis, takes the positive north direction as the positive direction of the axis, and according to the right-hand coordinate system principle, the vertical upward direction is the positive direction of the axis.

[0045] The body coordinate system B is fixed on the unmanned aerial vehicle, as shown in Figure 3 , the coordinate moves and rotates with the position and attitude of the unmanned aerial vehicle, and the origin is generally taken as the center of gravity of the unmanned aerial vehicle, the head direction is taken as the positive direction of the axis, and the left side direction is taken as the positive direction of the The positive direction of the axis, according to the principle of the right-hand coordinate system, is the vertically upward direction. The positive direction of the axis; where, , , and The coordinate system generated for each of the four rotors of the UAV relative to the body coordinate system. A thrust that is axial and upward.

[0046] (1-3) Selection of UAV motion mode. The quadcopter UAV adjusts the rotor speed by controlling the rotation speed of the four motors (the direction of the motors of the quadcopter UAV remains constant during operation), thereby achieving changes in the four lift and rotational torque, and thus controlling the basic motion of the six degrees of freedom of the quadcopter UAV.

[0047] like Figure 4 As shown, this application embodiment mainly considers the two degrees of freedom motion (forward and backward movement, and lateral movement) of the UAV in the xy plane, and designs five control speeds for the UAV. , , , and ), , , , as well as These represent five decision actions: forward, left, right, backward, and stationary, to facilitate the UAV in completing its tasks. Because the UAV's dynamic model has many input control variables (the rotational speeds of the four rotors), and its size and inertia are relatively small, its motion is highly flexible and varied in practical applications. Therefore, its controller is simplified, ignoring the control process from the initial state to the desired speed. The control of the UAV is considered as directly controlling its speed at the next moment based on the decision actions. It should be noted that the UAV's control method is the same in the simulation environment and the mass model environment.

[0048] (1-4) Selection of the coordinate system for the unmanned surface vessel (USV). The motion of the USV can also be considered as six-degree-of-freedom rigid body motion, typically described using two right-handed Cartesian coordinate systems. Similar to UAVs, one is the geodetic fixed coordinate system E(…). , , , The other is the fixed coordinate system B of the hull. , , , The xy-plane diagram is shown below. (Also known as the body coordinate system). Figure 5 As shown. The definition of the geodetic fixed coordinate system is similar to that of the Earth coordinate system in a UAV, the difference being... The axis points to the true west instead of the true east, at this time according to the principle of the right-hand coordinate system, The axis is vertically downward, because the water surface target generally does not jump out of the water surface, so the z axis is oriented downward, and the y axis also changes. As shown in Figure 5 The body coordinate system is fixed on the unmanned ship and moves together with the unmanned ship, the origin is located at the center of gravity of the unmanned ship, the x axis points to the bow of the ship, the y axis points to the right side of the ship, and the z axis points downward, Figure 5 refers to the heading angle, and v refers to the actual speed of the unmanned ship. Figure 5 The body coordinate system of the unmanned ship and the movement of the unmanned ship in the x-y plane are shown, which is combined with the yaw motion to realize the translational speed and motion in the x-y plane. Since the unmanned ship has only two propellers, it cannot realize the transverse motion in the y direction, so the displacement in the y direction needs to be realized by changing the heading angle.

[0049] (1-5) Selection of unmanned ship movement mode. As shown in Figure 6 The controller distribution and inertia of the unmanned ship are considered in the embodiment of the application. In the embodiment of the application, the movement of the unmanned ship is simplified to two degrees of freedom movement (forward and backward movement, yaw movement) in the x-y plane. Five control speeds (v , , , and v ) are designed for the unmanned ship, , , , and respectively represent five decision actions of moving forward, moving to the left front, moving to the right front, moving backward and staying in place, to facilitate the unmanned ship to complete the corresponding task. The pitch, heave and roll motions of the unmanned ship in the x-z plane and the y-z plane are ignored. At the same time, since the unmanned ship selects a double propeller ship model, it cannot directly control the transverse motion, so the transverse motion is also not considered in the application. It should be noted that in the embodiment of the application, the five control speeds (v , , , and v ) of the unmanned ship are represented by capital , and the five control speeds (v , , , and v ) of the unmanned ship are represented by small v.

[0050] Step (2), construct a multi-unmanned aerial vehicle and unmanned ship formation environment: ​(2-1) New level is created in UE4 platform, water body marine scene is imported, wave attenuation water depth D of water body marine component is set to 20.48 m, maximum wave height offset is set to 0, Gerstner wave is selected as wave type, which is a way of water surface wave form simulation and belongs to a kind of linear wave form superposition method. For Gerstner wave, wave number , minimum wavelength , maximum wavelength , wavelength drop . The rest is set according to the default setting, so that a relatively stable sea level environment is set.

[0051] (2-2) The unmanned aerial vehicle blueprint class is created, and the unmanned aerial vehicle grid model is imported. The unmanned aerial vehicle grid model is shown in Figure 7 . For the unmanned aerial vehicle, the application embodiment selects a quadcopter unmanned aerial vehicle model, but because the kinetic model is simplified, the unmanned aerial vehicle is controlled according to the speed given by the decision network, and the mass of the simulation physical part is not set.

[0052] (2-3) The unmanned surface vehicle blueprint class is created, and the unmanned surface vehicle grid model is imported. The structure of the unmanned surface vehicle grid model is shown in Figure 8 and Figure 9 . Two propellers with thrust direction along the x-axis direction of the hull are set at the position of the propeller at the tail of the hull, and six symmetrically distributed buoyancy components are set on both sides of the hull, and the radius of each buoyancy component is set to 1 m.

[0053] Among them, the angle resistance coefficient of the buoyancy component is changed to 1.5, and the rest is kept as the default setting. The specific parameter setting is shown in Table 1: Table 1: Buoyancy parameter setting table

[0054] The distribution positions of the buoyancy component and the propeller component are shown in Table 2: Table 2: Distribution position setting table of buoyancy component and propeller component

[0055] The rest of the simulation physical quantity is set as follows: mass 200 kg, linear damping 0.001 N·s / m, and angular damping coefficient is set to 0 N·m·s / rad.

[0056] (2-4) Repeat steps (2-2) and (2-3) until the required number of drones and unmanned surface vessels (USVs) is met, and drag the prepared blueprints of each drone and USV into the water / ocean scene. In this embodiment, a three-formation scene is set up with 3 navigator drones (m=3) and 5 follower USVs (n=5), that is, the total number of agents in the Block scene is set to 18; secondly, the yaw angle of each follower USV is set to 0, the Z-axis height is fixed at 50, and the X-axis and Y-axis coordinate values ​​are determined according to the number and total number of follower USVs.

[0057] Step (3), design the UAV / boat control model: (3-1) Control Model Selection. Regarding the control of the UAV, to simplify its dynamic model, the UAV is directly controlled based on the velocity given by the decision network (also known as the decision controller, i.e., the aforementioned multi-agent formation control model), without the need for additional control model design. Regarding the control of the unmanned surface vessel (USV), considering that its decision space consists of five discrete control velocities, but the value and effect of the decision are mainly based on the USV's position, controlling only the USV's velocity would introduce immeasurable errors in the decision evaluation process; therefore, this application chooses to use a controller to directly control the USV's position, and assumes the simple assumption that the USV moves at a uniform linear velocity according to the decision velocity during the decision execution time. The USV control flow diagram and the internal flow diagram of the USV PID controller are shown below. Figure 10 and Figure 11 As shown, the expected location of the unmanned surface vessel is... The calculation formula is as follows: ; in, Figure 10 In Represents decision shift, decision shift The calculation formula is as follows: .

[0058] (3-2) Controller design. For example... Figure 10 As shown, using a classic PID controller reduces the expected position. With current location deviation Step by step, the unmanned surface vessel was brought closer and eventually stopped at the intended location. Specifically, this application employs a dual-loop alternating control system consisting of a heading controller and a distance controller to enable the unmanned surface vessel (USV) to reach the intended position. The two controllers are manually tuned separately. The specific implementation process of the dual-loop alternating control system is as follows: Assuming in The unmanned surface vessel receives control commands (decision speed) based on the decision network. and The expected position can be calculated using the formula for calculating the expected position of an unmanned surface vessel. Specifically, the expected location is first calculated. With current location The expected heading angle between, the expected heading angle The calculation formula is as follows: ; in, and These are the expected positions. With current location Between the geodetic coordinate system and Axial deviation. Furthermore, to distinguish between forward and backward movements and to prevent the unmanned surface vessel from turning its entire bow backward while moving backward, [further measures are needed]. Use positive and negative signs as markers for forward and backward movement, respectively. ,mark The expression is as follows: ; in, This is a sign function, whose function is to take the sign of its value.

[0059] Subsequently, according to the markings For the expected heading angle Make corrections to obtain the corrected expected heading angle. Among them, the expected heading angle The corrected calculation formula is as follows: ; Next, the corrected expected heading angle is calculated. and the current moment of the unmanned surface vessel Actual heading angle The deviation of the heading angle between Among them, the corrected expected heading angle is calculated. and the current moment of the unmanned surface vessel Actual heading angle The deviation of the heading angle between The calculation formula is as follows: ; like Figure 11 As shown, when the corrected expected heading angle and the current moment of the unmanned surface vessel Actual heading angle The deviation of the heading angle between The absolute value is greater than the preset heading angle deviation threshold. When set to 0.1 rad, the yaw angle controller (i.e., the heading controller) is activated for yaw angle control. After manual tuning, the PID parameters of the heading controller can be determined as follows: , This simplifies the heading controller to a proportional controller. The torque control quantity is obtained through the proportional element. Torque control quantity The calculation formula is as follows: ; Then based on the torque control quantity Thrust distribution is performed to obtain the thrust from the two thrusters on the unmanned surface vessel. , At this time, the thrust on the two thrusters of the unmanned surface vessel ( , The calculation formula for ) is as follows: ; in, The distribution interval of the two thrusters can be found in Table 2. Take this place As an approximation for ease of calculation.

[0060] When the corrected expected heading angle and the current moment of the unmanned surface vessel Actual heading angle The deviation of the heading angle between Less than the preset heading angle deviation threshold When the distance controller is activated for distance control, the expected position needs to be calculated. Go to current location Distance deviation between The formula for calculating distance deviation is as follows: ; Among them, the mark The purpose of introducing this parameter is to assign direction to the distance deviation, preventing a consistently positive distance deviation from causing incorrect feedback direction during overshoot and thus preventing the control system from diverging. After manual tuning, the PID parameters of the distance controller can be determined as follows: , , This simplifies the distance controller into a proportional-derivative controller. The thrust control quantity is obtained through the proportional and derivative components. Thrust control amount The calculation formula is as follows: ; in, The distance deviation at the previous control moment is initially set to 0; To control the timing, the initial value is set to 0.01s, and it is adjusted in real time according to the transmission delay during subsequent control processes.

[0061] Then, based on the thrust control amount Thrust distribution is performed. Since the two thrusters are symmetrically distributed about the x-axis, the thrust on the two thrusters can be obtained directly by symmetrical distribution. The formula for calculating the thrust on the two thrusters of the unmanned surface vessel is as follows: ; After completing the above control process, the thrust calculated by the controller is... Disassembled into thrust and Then, regarding the thrust and After correction and Finally, the corrected thrust and The data is transmitted to the simulation platform, and then the next control moment is received back from the simulation platform. unmanned surface vessel location and heading angle The current position and current yaw angle are updated using the following expression: ; Repeat the above process, during which the expected position is... Unchanged until the next decision is implemented. The decision-maker updates the expected position, and the unmanned surface vessel (USV) obtains the new target position and continues to be controlled and move accordingly according to the control process. Overall, in the embodiments of this application, the control of the USV can be regarded as a control process that first completes the heading angle control and then completes the distance control; however, if the heading angle is greatly affected during the distance control process, it will briefly switch to the yaw angle controller to correct the heading angle first.

[0062] It is worth mentioning that when the distance deviation of the unmanned surface vessel... Very small, meaning that when the unmanned surface vessel approaches the target point, the expected heading angle is... The calculation formula ( The calculated expected heading angle Oscillations may occur, causing deviations in heading angle. The oscillations cause the unmanned surface vessel (USV) to undergo unstable oscillating motion under the control of the heading angle controller. Therefore, it is necessary to adjust the expected heading angle. The calculation formula ( To avoid this situation, corrections will be made to the expected heading angle. The calculation formula ( The corrected calculation formula is as follows: ; in, The distance threshold is set to 0.1m. This correction can be seen as prohibiting the use of the heading angle controller in the final stage of the unmanned surface vessel approaching the target point, in order to prevent the unmanned surface vessel from oscillating.

[0063] The thrust calculated by the controller A maximum thrust value needs to be set for trimming to prevent excessive thrust. Additionally, it needs to be multiplied by a factor. The data is then transmitted to a simulation platform to set the thruster strength. This application specifies the maximum thrust value. Set to 10N, thruster strength multiplier The value is set to 10000. Therefore, the formula for calculating the thrust correction is as follows: ; in, This indicates a cropping operation, when the input value... exist The output value is within the range. When input value Less than the lower boundary The output value is the lower boundary and greater than the upper boundary. The output value is then the upper boundary, thus affecting the input value. It achieved the function of limiting the amplitude.

[0064] According to the calculation formula for correcting thrust ( ) respectively for thrust Thrust is obtained by making corrections This will allow the corrected thrust to be used. The data is transmitted to the simulation platform to set the thruster intensity. The unmanned surface vessel (USV) control flow diagram and the internal flow diagram of the USV's PID controller are shown below. Figure 10 and Figure 11 As shown, Figure 11 The middle diamond represents a commonly used process judgment difference graph. If the condition within the diamond is met, the process is transmitted to the True direction; otherwise, it's transmitted to the False direction. If neither True nor False is found, the transmission is interrupted. For... Figure 11 The triangle pointing right represents a switch valve. When data is received from the upper or lower side, data can flow between the left and right sides, and data can be transmitted from the left to the right. If no data is received from the upper or lower side, data cannot be transmitted from the left to the right. Figure 10 The triangle pointing to the right in the center represents the waiting valve. If the value inside the triangle is 't', it means that after waiting for time 't', the data on the left will be transferred to the right according to the arrow.

[0065] (3-3) Data transmission design. After the completion of the controller design, the unmanned aerial vehicles are directly controlled by the controller, and each unmanned surface vehicle is equipped with a position controller (including a heading controller and a distance controller). Specifically, the ObjectDeliverer and TCP Socket plugins need to be imported into the UE4 platform, a control blueprint for TCP data transmission is created, a TCP port connection is set up to transmit information from the unmanned aerial vehicles and unmanned surface vehicles to the TCP port, and control instructions from the TCP port are accepted to control the unmanned aerial vehicles and unmanned surface vehicles. In this application, the TCP sending end needs to obtain the position information of the unmanned aerial vehicles and unmanned surface vehicles, and send the following list to the TCP port: ; ; wherein, represents the x, y, z coordinates of the first unmanned aerial vehicle in the geodetic coordinate system, represents the x, y coordinates of the first unmanned surface vehicle in the geodetic coordinate system, and the last 0 is used to prevent the program from receiving end-of-line code interference with the acquisition of position information.

[0066] The TCP receiving end accepts the following control speed and thrust information from the controller: ; wherein, represents the x, y, z direction velocity of the first unmanned aerial vehicle in the geodetic coordinate system, represents the two control thrusts of the first unmanned surface vehicle, indicates the flag to restart the scene, when the scene is not reset, when the scene is reset, and the initial position of each unmanned aerial vehicle and unmanned surface vehicle is set to 0, and their heading angle, speed, acceleration, and angular velocity are also set to 0. The first 123 and the last 321 are used to check the control list.

[0067] (3-4) Controller experiment design and effect evaluation. In this embodiment, two experiments are designed to evaluate the accuracy of the control model to ensure that the control model can complete the control task required by the decision maker.

[0068] Combined with the five control speeds of the unmanned surface vehicle ( ), i.e. forward, left forward, right forward, backward, and stationary, the decision execution time is set to 5 seconds, and the speed amplitude is 1 m / s. The initial position of a unmanned surface vehicle is . At this time, the control speed of the unmanned surface vehicle can be expanded as follows:​​​​ ; Experiment 1: Four actions with displacement are selected as continuous control sequence in the control speed artificially, in the order of , that is, forward, left forward, right forward, backward. The distance deviation curve (as shown in Figure 12 ) and the heading angle deviation curve (as shown in Figure 13 ) in the control process, and the expected trajectory and actual trajectory comparison diagram (as shown in Figure 14 ) are recorded. Among them, Figure 12 The abscissa represents the control step number step, and the ordinate represents the distance between the actual position and the expected position at the current step; Figure 13 The abscissa represents the control step number step, and the ordinate represents the heading angle deviation between the actual position and the expected position at the current step; Figure 14 The abscissa represents the x coordinate of the environment, and the ordinate represents the y coordinate of the environment, Figure 14 The line a in represents the curve composed of the expected x coordinate and the expected y coordinate of the unmanned ship at each control step, and the line b represents the curve composed of the actual x coordinate and the actual y coordinate of the unmanned ship at each control step.

[0069] From the results of experiment 1, it can be seen that the controller controls the unmanned ship in accordance with the above-mentioned double-loop alternating control process of heading controller and distance controller: first complete the heading angle control, and then complete the distance control. In addition, the distance between the corresponding points of the expected trajectory and the actual trajectory is calculated, and the mean square value is calculated as the control model error evaluation index. The calculation formula of the mean square value is as follows: ; Among them, is the point on the expected trajectory, is the point on the actual trajectory, is the number of trajectory points, represents the geometric distance between two points. From Figures 12-14 , it can be obtained that the error index of the trajectory tracking task in experiment 1 for is .

[0070] Experiment 2: According to the task to be completed by the unmanned ship, the decision model is trained under the task of using more trajectory points, that is, decision steps, and the trajectory of one of the unmanned ships given by the trained model is used to test the final effect of the controller. Here, the number of trajectory points is 500. Similarly, the distance deviation curve (as shown in Figure 15and the heading angle deviation curve (as shown in FIG. 6B) and the expected trajectory and actual trajectory comparison graph (as shown in FIG. 6C). Figure 16 , Figure 17 , Figure 15 , Figure 16 , Figure 17 The abscissa of the above figures represents the x coordinate of the environment, and the ordinate represents the y coordinate of the environment. Figure 15 The line in the above figure refers to the curve composed of the expected x coordinate and the expected y coordinate of the unmanned ship at each control step. Figure 16 The line in the above figure refers to the curve composed of the actual x coordinate and the actual y coordinate of the unmanned ship at each control step. Figure 17 is the result of drawing the two curves in Figure 15 and Figure 16 in the same figure.

[0071] As can be seen from Figures 15-17 , in the second experiment, the error index of the trajectory tracking task when is , and the trajectory comparison graph shows that the unmanned ship can continuously achieve stable trajectory tracking of multiple trajectory points under the control of the controller, and maintain a low error.

[0072] In summary, the error of the controller is within an acceptable range, and it can still maintain a low error in stable tracking of continuous multiple trajectory points, indicating that the controller designed in the embodiments of the present application is well adapted to Unreal Engine UE4 and can be used in the control of unmanned ships by deep reinforcement learning.

[0073] Step (4), migration of the particle model to the simulation environment: (4-1) Formation model setting. According to the multi-formation control target of the present application, the formation model is selected as the leader-follower model. The core idea of the leader-follower model is to guide other followers to complete the formation task through a small number of leaders (usually intelligent agents with global target information). In this method, the leader is responsible for path planning, target point allocation, and maintenance of the global formation structure, while the follower adjusts its position and motion state to maintain the relative relationship with the leader through observation and perception of the leader.

[0074] In order to maintain a reasonable formation between the leader and the follower, the followers should be evenly distributed around the leader. Set the three-dimensional position of the unmanned aerial vehicle in a sub-formation as , wherein are the coordinates of the unmanned aerial vehicle on the , , axes. The two-dimensional projection point position of the unmanned aerial vehicle is defined as in the plane, i.e. the projection of the sea level on which the UAV is located, the two-dimensional projection point position of the UAV The expression of the target point position (expected position) of the UAV is as follows: ; UAVs in the formation corresponding target point The expression of the target point position (expected position) of the UAV is as follows: ; For the UAV cluster in the sub-formation, wherein the UAV corresponding target point position (expected position) The expression of the target point position (expected position) of the UAV is as follows: ; In order to maintain a reasonable formation between the leader and the follower, the follower should be evenly distributed around the leader. Specifically, in the horizontal plane, the follower should be distributed at equal angular intervals on the circumference with the leader as the center and the radius of The initial angular offset is set, and the calculation formula of the expected angle of each follower is as follows: ; wherein, is the total number of UAVs in the sub-formation, is the expected radial distance, which is set to in the present application, indicating the expected distance between each follower and the leader, i.e. the formation radius. The expected position coordinates of the follower are as follows: ; Since the control of the UAV actually only uses the point mass model, the discussion of the UAV only stays in two dimensions, and the expansion expression of the control speed of the UAV is as follows: ; wherein, the UAV is characterized by the position and speed of the projection point of the UAV leader, so as to simplify the UAV to the same two-dimensional space as the UAV for discussion.

[0075] (4-2) Environment setting under the point mass model. Three formations are set, each formation has 5 UAVs, so there are 3 UAVs ( ) and 15 UAVs ( ) in total, i.e. 18 intelligent agents. According to the description in step (2), the discrete action space is set for the UAV and the UAV, the UAV is designed with five control speeds , which respectively represent five decision-making actions of going forward, going left, going right, going backward and staying in place; the UAV is also designed with five control speeds , respectively, represent five decision actions of going forward, going left forward, going right forward, going backward and staying in place. The control velocity of USV has been given, and the control velocity of UAV is similar, the control velocity amplitude is set to 0.75 m / s to achieve more stable formation control, and the control velocity expression of UAV is as follows:

[0076] ; The amplitude of the control velocity here is smaller than the actual situation, which is due to the inertia of the USV in the simulation environment being larger, a longer decision execution time is set, and in order to not make the displacement of each decision too large, a smaller control velocity amplitude is selected. In the actual situation, that is, collecting data in real ship and real field for training, the control velocity amplitude and decision execution time can be adjusted according to the actual inertia of the USV.

[0077] In the point mass model, without considering the influence of external force, after the UAV / USV makes a decision action selection, it is defaulted to move at a constant speed in a straight line according to the control velocity, that is, the position of the UAV / USV at the next decision execution time is equal to the expected position given by the calculation formula of the expected position of the USV It should be noted that the point mass model is a feature of the point mass model environment, which does not consider the effect of external force, and the agent is simplified as a point and its shape is ignored.

[0078] And in the setting of the environment, the starting position and target position of the UAV for the three formations are set as follows: ; ; ; According to the starting position and target position given by the above formula , , , it can be seen that the initial state of the three formations is triangularly distributed, along the earth coordinate system axis from the negative direction to the positive direction, formation 1 is in the middle, and formations 2 and 3 are distributed equidistantly on the two sides behind formation 1. Their target positions are respectively for formations 1, 2 and 3 to perform the tasks of going forward, going left forward and going right forward. Only the original formation radius is replaced by the initial formation radius .

[0079] From the above formula , , It can be seen that the minimum displacement distance required for each formation is 500m. Considering the speed range of the UAV is 0.75m / s and the speed range of the unmanned surface vessel is 1m / s, and the decision execution time is 10s, based on the slower speed of the UAV as the lower limit of the number of decision steps, each decision can displace a maximum of 7.5m (excluding the case of staying in place). Therefore, the number of decision steps is... At least 500m / 7.5m≈67 is required. However, to make the entire decision-making process more diverse and allow the drone to choose a strategy of remaining stationary for adjustments, this invention selects a larger number of decision steps. .

[0080] (4-3) The model is trained under a point mass model (environment). This application uses the optimized MAPPO algorithm as the deep learning algorithm. The unmanned surface vessel (USV) detects obstacles by using detection rays of a certain length, evenly distributed at the center of the USV, to achieve obstacle detection. For example... Figure 18 As shown, unmanned surface vessel There are a total Strips are distributed at equal angles, with a maximum length of The detection rays, among which, are shown Figure 18 Three lines (referred to by the letter 'c') are shorter than the maximum length. That is, there is an obstacle in that direction, and the distance to the obstacle is... , Indicates the first A detection ray, It refers to the vector pointing from the position of unmanned surface vessel i at time t to the outermost position of its first detection ray at time t.

[0081] In the improved MAPPO algorithm, each formation of drones and unmanned surface vessels (USVs) gradually explores and learns the optimal policy through E rounds of exploration. For example, in common reinforcement learning algorithms, the agent selects actions based on the current environmental state, the environment provides corresponding reward signals, and the agent optimizes the policy to maximize long-term cumulative rewards. Commonly used algorithms such as the Double Q network alleviate Q-learning overestimation by decoupling action selection and value evaluation; the PPO algorithm ensures policy update stability by pruning the objective function; and the TD3 algorithm further improves the performance of continuous control tasks by combining objective policy smoothing and the Double Q network mechanism. These methods improve sample efficiency and convergence stability at different levels. Since this application mainly discusses the transfer of the model to the simulation environment, the algorithm and training steps under the mass point model are not elaborated in detail. Other effective algorithms can be substituted, and the application is not limited to the examples mentioned in this application. The embodiments of this application are not described in detail here, and it is understood that the embodiments of this application are not limited in this respect.

[0082] Assuming that the deep reinforcement learning model has been trained, it can meet the input state including the positions and velocities of the UAVs / boats in each formation at a certain time, as well as the lengths of the detection rays of the USVs, and output the control velocity expansion expression of the USVs and the discrete action space of the USVs and UAVs in the control velocity expansion expression of the UAVs. Combined with steps (4-1) and (4-2) to set the initial position, target position and situation with a certain number of obstacles, the formation shape keeping and collision avoidance tasks can be realized in the particle model. Taking the trained model Migrating to the UE4 high-fidelity environment for fine-tuning training can reduce the long training time overhead caused by the requirement of the controller for decision execution time dT=10s. In this way, only a small number of exploration rounds are needed in the simulation environment to obtain a relatively ideal result, thereby efficiently completing the multi-UAV / boat cooperative control task in a high-fidelity environment.

[0083] (4-4) Model migration training.

[0084] A simulation environment similar to the particle environment is set up in the UE4 platform, as shown in Figure 19 At this time, the obstacles are simplified as simple cubes and spheres (white cubes and spheres in Figure 19 The object line segments in the white dashed box in Figure 19 are the detection rays of the USVs. Therefore, the algorithm needs to accept the information passed by UE4, and it is particularly noted that unlike the particle environment, the velocities and positions of the agents contained in the algorithm input depend on the information passed by UE4 , that is, the data sent by the TCP receiving end, rather than the control velocity and expected position set by the particle model: ; ; Among them, the first 123 and the last 321 are test symbols.

[0085] In the migration training, the algorithm model is first initialized, the trained model in step (4-3) is loaded, and fine-tuning training is started. During the training process, for the t-step strategy of a certain exploration round e, based on the current strategy, the control model designed in step (3-3) gives the control sequence, which is transmitted to the corresponding controller in the simulation environment through the TCP port , and the states of multiple agents are collected from the TCP port , which are input into the deep reinforcement learning algorithm for the next decision, and the strategy and model are fine-tuned and optimized according to the algorithm design.

[0086] The parameter settings of the fine-tuning process in the transfer learning only set a small number of exploration rounds E, and other parameters related to the exploration efficiency need to be additionally set, so that the exploration efficiency is relatively low. The parameters that would change with the exploration round e are directly set to the annealing value of the last exploration round of the training process to meet the design concept of fine-tuning.

[0087] It should be noted that the decision of the controller in the particle model (environment) does not need to wait for the decision execution time; and the controller in the simulation environment needs to wait for the decision execution time at each step, but the simulation environment is closer to the real situation (more situations and external forces are considered). Therefore, by combining the two, the decision model is trained under the assumption of the particle model first, the decision model is retained, and the decision model is copied to the simulation environment for further fine-tuning instead of retraining, which can greatly speed up the training process.

[0088] (4-5) Transfer model verification. In terms of algorithm effect, the present application mainly uses the average total number of collisions and the average reward convergence value for measurement. The total number of collisions in the e-th exploration round includes the collisions between the unmanned aerial vehicles / boats and the collisions between the unmanned aerial vehicles / boats and the obstacles, and the average total number of collisions is obtained by averaging the total number of collisions in each exploration round . Obviously, the fewer the number of collisions, the more effectively the unmanned aerial vehicles / boats can avoid collisions, and the more excellent the algorithm performs in the strategy exploration effect. The reward average value of a round e is the average value of the two-dimensional matrix composed of the reward average values of all unmanned aerial vehicles / boats at all times in the round, which is directly calculated in full dimension and multiplied by an amplification coefficient to obtain: ; According to the calculation formula of , the amplification coefficient is 100. Since the design of the reward function is according to the task target, the faster the reward rises, the higher the convergence value , and the more excellent the algorithm performs in the strategy exploration effect. In order to verify the effect of transfer training, the present application lists the average total number of collisions and the reward convergence value under formal training and transfer training for comparison, which are shown in Table 3 as follows: Table 3: Comparison table of effects of formal training and transfer training

[0089] The decision effect diagram under the UE4 simulation environment is as follows Figure 20As shown, curves d, e and f are the trajectory curves of the UAVs, and the remaining curves are the trajectory curves of the USVs, and the white cubes and spheres are obstacles. In the decision-making process of the three formations, the controllers can better adapt to the environment and decision makers, keep the formation around the UAV, plan the path and avoid collision. This is specifically reflected in that the path of the USV basically avoids the obstacles while the path of the UAV changes almost synchronously; the undulating nature of the horizontally moving formation path is due to the fact that the UAV can directly select the control speed in front, back, left and right to always choose to quickly approach the target point to the right, but the USV only has the speed selection of the right lower and right upper, so its path will present the situation of turning back. Since the UE4 simulation environment is more faithful and complex than the particle model, it can be seen that the model effect has decreased to a certain extent: the reward convergence value only decreases by about 6%, and although the collision frequency is relatively high, this is because the original collision frequency of the former in the training environment is already small and close to the boundary. In summary Figure 20 , the transfer learning can achieve a training effect close to that of the ideal particle model with fewer exploration rounds. Therefore, the experiment proves that applying the controller and transfer learning method in the simulation environment using "modeling-decision-making-transfer" can greatly reduce the training time and increase the training efficiency. Through theoretical innovation and engineering verification, the present application promotes the leap of multi-agent cooperative control from algorithm design to simulation application, and provides reliable technical support for the task execution of intelligent unmanned systems in complex dynamic environments.

[0090] It should be noted that the present embodiment only briefly and schematically illustrates the overall process of the multi-UAV and USV formation control method based on the Unreal Engine, and detailed descriptions of each step can be referred to the related content in the foregoing embodiments, which will not be repeated here. It can be understood that the present application does not limit this.

[0091] In summary, the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application aims to solve the problem of poor training fidelity of the multi-agent formation control model based on deep reinforcement learning under the traditional particle model and the insufficient real-time decision response performance of the trained model in a dynamic environment. To this end, the embodiments of the present application can achieve a relatively realistic multi-unmanned aerial vehicle and unmanned ship formation control effect by constructing a high-fidelity simulation environment and a formation control model based on the Unreal Engine, and a particle model and a high-fidelity simulation environment of the Unreal Engine dual verification system. Specifically, after verifying the convergence of the deep reinforcement learning algorithm in the particle model, the strategy network parameters of the deep reinforcement learning algorithm are mapped to the simulation environment through transfer learning, and data interaction is realized in combination with the TCP protocol. The above experiments show that after introducing the simulation environment, the formation control model and transfer learning, the agent formation scene can maintain good decision effect, verifying the feasibility of the algorithm from the theoretical model to the engineering application. Through the technical closed loop of “modeling-decision-transfer”, the embodiments of the present application provide an extensible solution and a more optimal decision model for multi-unmanned aerial vehicle and unmanned ship cooperative control, and have important application value in complex scenes such as disaster rescue.

[0092] The multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application utilizes the high simulation and multi-platform cooperative test environment provided by the Unreal Engine (UE4) platform itself, and designs an alternating control of the heading angle controller and the distance controller according to the characteristics of the platform, which can maintain stable tracking with low error under continuous multi-trajectory point tracking, thereby building a necessary and solid bridge for the particle model of the unmanned ship to be transferred to the Unreal Engine simulation platform to achieve more realistic control effect. Specifically, the embodiments of the present application realize the transfer of the control strategy by constructing the transfer process of the particle model to the Unreal Engine simulation environment, improve the model training efficiency in the simulation environment, and at the same time retain high performance and precision of the model, thereby confirming the effectiveness and feasibility of the multi-unmanned aerial vehicle and unmanned ship formation control method based on the Unreal Engine provided in the embodiments of the present application in the field of multi-unmanned aerial vehicle and unmanned ship cooperative control.

[0093] For reference Figure 21 The embodiments of the present application also provide a multi-unmanned aerial vehicle and unmanned ship formation control device 2100 based on the Unreal Engine, which can implement the above method. The device comprises the following modules: A kinematic model construction module 2101 is configured to construct a kinematic model of a multi-agent formation. The multi-agent formation comprises a formation of unmanned aerial vehicles and a formation of unmanned ships. An experimental simulation environment construction module 2102 is configured to construct a first experimental simulation environment of the multi-agent formation based on the Unreal Engine. The unmanned ship control model construction module 2103 is configured to construct a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematics model, wherein the double-loop alternating unmanned ship control model comprises a heading controller and a distance controller. The deep reinforcement learning model pre-training module 2104 is configured to construct a particle model environment and pre-train a deep reinforcement learning model in the particle model environment. The model migration fine-tuning training module 2105 is configured to migrate the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tune the trained deep reinforcement learning model in combination with the kinematics model and the double-loop alternating unmanned ship control model, and obtain a multi-agent formation control model. The multi-agent formation control model acquisition module 2106 is configured to deploy the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification, wherein the multi-agent formation control model is used to control the multi-agent formation.

[0094] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0095] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0096] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0097] Please refer to Figure 22 , Figure 22 The hardware structure of the electronic device of another embodiment is illustrated, which comprises: The processor 2201 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application. The memory 2202 can be implemented in the form of Read Only Memory (ROM), static storage device, dynamic storage device or Random Access Memory (RAM), etc. The memory 2202 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 2202 and are called and executed by the processor 2201 to perform the above-mentioned method of the embodiments of the present application; The input / output interface 2203 is configured to realize information input and output. The communication interface 2204 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.). The bus 2205 transmits information between various components (such as the processor 2201, the memory 2202, the input / output interface 2203 and the communication interface 2204) of the device. The processor 2201, the memory 2202, the input / output interface 2203 and the communication interface 2204 are connected and communicated with each other in the device through the bus 2205.

[0098] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned method.

[0099] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present storage medium embodiments, the functions specifically realized by the present storage medium embodiments are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those achieved by the above-mentioned method embodiments.

[0100] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to realize the above-mentioned method.

[0101] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present program product embodiments, the functions specifically realized by the present program product embodiments are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved by the present program product embodiments are also the same as those achieved by the above-mentioned method embodiments.

[0102] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include memory that is remotely located with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0103] The multi-unmanned aerial vehicle and unmanned ship formation control method, device, electronic equipment, storage medium and program product based on Unreal Engine provided by the embodiments of the present application can keep stable tracking with low error under continuous multi-trajectory point number trajectory tracking by utilizing the high simulation and multi-platform collaborative test environment provided by the Unreal Engine platform itself, designing the alternative control of the heading controller and the distance controller in combination with the kinematic model for the characteristics of the Unreal Engine platform, building a necessary and solid bridge for the point mass model of the unmanned ship to transfer to the Unreal Engine simulation platform to realize more realistic control effect, and realizing the migration of the control strategy through the migration process of the point mass model to the Unreal Engine simulation environment, improving the model training efficiency of the deep reinforcement learning model in the simulation environment, while retaining the high performance and precision of the deep reinforcement learning model, and improving the good decision-making effect of the deep reinforcement learning model in the agent formation scene.

[0104] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for controlling a multi-UAV and unmanned surface vehicle formation based on Unreal Engine, characterized in that, The method comprises the following steps: constructing a kinematic model of a multi-agent formation; wherein the multi-agent formation comprises a UAV formation and an unmanned ship formation; constructing a first experimental simulation environment of the multi-agent formation based on an Unreal Engine; constructing a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematic model; wherein the double-loop alternating unmanned ship control model comprises a heading controller and a distance controller; constructing a particle model environment and pre-training a deep reinforcement learning model in the particle model environment; migrating the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tuning the trained deep reinforcement learning model in combination with the kinematic model and the double-loop alternating unmanned ship control model, and obtaining a multi-agent formation control model; deploying the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation.

2. The method of claim 1, wherein, The method further comprises the step of executing control of the double-loop alternating unmanned ship control model, wherein the execution of the control of the double-loop alternating unmanned ship control model comprises: during the execution of a task by the unmanned ship, calculating an expected position of the unmanned ship according to a decision control instruction generated by the multi-agent formation control model for the unmanned ship at a current time; wherein the decision control instruction comprises a decision speed and a decision execution time; calculating an expected heading angle between the expected position and a current position of the unmanned ship; correcting the expected heading angle and calculating a target heading angle deviation between the corrected expected heading angle and an actual heading angle of the unmanned ship at the current time; when an absolute value of the target heading angle deviation is greater than a preset heading angle deviation threshold, activating the heading controller to control a yaw angle of the unmanned ship.

3. The method of claim 1, wherein, ​ ​ ​ ​ ​ 4. The method of claim 1, wherein, ​ ​ ​ ​ ​ When the absolute value of the target heading angle deviation is less than a preset heading angle deviation threshold, the distance controller is activated to perform distance control on the unmanned ship.

5. The method of claim 1, wherein, The deep reinforcement learning model trained in the particle model environment is migrated to the first experimental simulation environment, the control sequence generated by the double-loop alternating unmanned ship control model is obtained, and the motion state of the multi-agent formation in the kinematics model is obtained. The deep reinforcement learning model trained in the particle model environment is migrated to the first experimental simulation environment, the control sequence generated by the double-loop alternating unmanned ship control model is obtained, and the motion state of the multi-agent formation in the kinematics model is obtained. The control sequence and the motion state are input into the trained deep reinforcement learning model for fine-tuning to obtain the multi-agent formation control model.

6. The method of claim 1, wherein, The multi-agent formation control model is deployed in a second experimental simulation environment different from the first experimental simulation environment for verification, including: A second experimental simulation environment different from the first experimental simulation environment is constructed based on the Unreal Engine; The multi-agent formation control model is deployed in the second experimental simulation environment to verify the multi-agent formation control model, and the average total number of collisions and the average reward convergence value in the verification process are obtained; According to the average total number of collisions and the average reward convergence value, it is determined whether the multi-agent formation control model passes the model verification, and if the multi-agent formation control model passes the model verification, the multi-agent formation control model that passes the verification is used to dynamically control the multi-agent formation performing a task.

7. A multi-UAV and unmanned surface vehicle formation control device based on Unreal Engine, characterized in that, The device includes the following modules: A kinematics model construction module is configured to construct a kinematics model of a multi-agent formation; wherein the multi-agent formation includes a UAV formation and an unmanned ship formation; An experimental simulation environment construction module is configured to construct a first experimental simulation environment of the multi-agent formation based on the Unreal Engine; An unmanned ship control model construction module is configured to construct a double-loop alternating unmanned ship control model based on the first experimental simulation environment and the kinematics model; wherein the double-loop alternating unmanned ship control model includes a heading controller and a distance controller; A deep reinforcement learning model pre-training module is configured to construct a particle model environment and pre-train a deep reinforcement learning model in the particle model environment; A model migration and fine-tuning training module is configured to migrate the deep reinforcement learning model trained in the particle model environment to the first experimental simulation environment, fine-tune the trained deep reinforcement learning model in combination with the kinematics model and the double-loop alternating unmanned ship control model, and obtain a multi-agent formation control model. A multi-agent formation control model acquisition module is configured to deploy the multi-agent formation control model in a second experimental simulation environment different from the first experimental simulation environment for verification; wherein the multi-agent formation control model is used to control the multi-agent formation.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 6.

Images