A multi-unmanned ship game confrontation method and system based on a DRMRPG algorithm

Abstract

The application discloses a multi-unmanned ship game confrontation method and system based on a DRMRPG algorithm, and the method comprises the following steps: constructing a multi-unmanned ship intelligent game framework based on the DRMRPG algorithm; introducing a multi-level experience replay strategy, a dynamic soft update strategy and a residual connection strategy on the basis of a MADDPG algorithm to obtain the DRMRPG algorithm; constructing an initial multi-unmanned ship game decision model based on the DRMRPG algorithm; training the initial multi-unmanned ship game decision model by using the multi-unmanned ship intelligent game framework to obtain an intelligent decision model; and performing multi-unmanned ship game confrontation by using the intelligent decision model to obtain a game decision. The application has high reward value, high stability and high efficiency, and has certain effectiveness.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent game adversarial technology, specifically relating to a multi-unmanned surface vessel game adversarial method and system based on the DRMRPG algorithm. Background Technology

[0002] Unmanned surface vessels (USVs) possess all-weather mission capabilities, especially in performing dangerous, time-consuming, and labor-intensive tasks in harsh marine environments, offering broad application prospects in both military and civilian fields. Intelligent decision-making is a crucial supporting technology for USV applications, enabling them to replace human pilots in dangerous missions. Deep reinforcement learning combines the advantages of deep learning in perception and understanding with the excellent performance of reinforcement learning in decision-making, achieving accurate decision-making for USVs in various environments and in response to real-time situations.

[0003] On the other hand, reinforcement learning, as an important branch of artificial intelligence, currently has significant application value in multi-agent game-based adversarial problems involving unmanned surface vessels (USVs) and unmanned aerial vehicles (UAVs). In 2021, Li Bo et al. proposed an improved multi-agent deep deterministic policy gradient algorithm, applied to the research of cooperative tasks involving multiple UAVs, which can solve simple task decision-making problems. In 2023, Liu Jing et al. proposed a UAV swarm cooperative encirclement method combining game theory and Q-Learning, and the results showed that this method can effectively encircle and capture a single target. In 2022, Zhan proposed a multi-agent proximal policy optimization algorithm to realize distributed decision-making and cooperative task completion for heterogeneous UAVs. In contrast, research on game-based adversarial problems involving USVs is relatively limited both domestically and internationally, and is still in the development stage. In 2022, Su Zhen et al. conducted research on dynamic game-based adversarial problems involving USV swarms, proposing to use a deep deterministic policy gradient algorithm to design a policy solution method, and the trained agents can perform cooperative encirclement tasks well. In 2023, Xia Jiawei et al. used the MAPPO algorithm to complete a cooperative encirclement task against a single unmanned surface vessel (USV). By combining the background of the encirclement task, they established a state space with scalability and permutation invariance, and finally used a course-based learning training technique to train the encirclement strategy. However, research on USV swarm game adversarial research is still in its early stages and has considerable room for improvement. Currently, there is a lack of authoritative and realistic simulation platforms. Furthermore, for reinforcement learning algorithms, when facing high-dimensional state spaces and complex observation environments, existing multi-agent deep reinforcement learning algorithms still face problems such as difficulty in convergence, poor training stability, and low sampling efficiency. Summary of the Invention

[0004] This invention aims to address the shortcomings of existing technologies and provides the following solutions:

[0005] A multi-unmanned surface vessel (USV) game adversarial method based on the DRMRPG algorithm includes the following steps:

[0006] Construct a multi-unmanned surface vessel intelligent game framework based on the aforementioned DRMRPG algorithm;

[0007] Based on the MADDPG algorithm, a multi-level experience replay strategy, a dynamic soft update strategy, and a residual connection strategy are introduced to obtain the DRMRPG algorithm.

[0008] Based on the DRMRPG algorithm, an initial multi-unmanned surface vessel game decision model is constructed.

[0009] The initial multi-unmanned surface vessel (USV) game decision model is trained using the aforementioned multi-USV intelligent game framework to obtain an intelligent decision model.

[0010] The intelligent decision-making model is used to conduct multi-unmanned surface vessel (USV) game competition to obtain game decisions.

[0011] Preferably, the method for constructing the multi-unmanned surface vessel intelligent game framework includes:

[0012] Construct the state space of the multi-unmanned surface vessel (USV) game environment based on the DRMRPG algorithm, and obtain the state information of the red and blue USV agents during the game.

[0013] Based on the state information, an action space for a multi-unmanned surface vessel (USV) game environment based on the DRMRPG algorithm is constructed to obtain the set of actions that can be performed by the red and blue USV agents during the game.

[0014] Based on the set of actions, a segmented reward function is constructed to guide the red and blue unmanned surface vessel agents to continuously optimize their strategies during the game, thus completing the construction of the multi-unmanned surface vessel intelligent game framework.

[0015] Preferably, methods for constructing piecewise reward functions include:

[0016] The reward function for the Red Team's unmanned surface vessel agent is:

[0017] Y end =r adv +r pos +r col

[0018]

[0019]

[0020] Where, r end This represents the final reward obtained by the Red Team's unmanned surface vessel. adv Indicates the location of the other party's unmanned surface vessel, pos desIndicates the location of the target location, pos i This represents the location of the i-th unmanned surface vessel (USV) in the red team's USV swarm, range map r represents the extent of the map. adv r represents the negative reward that the target provides to the agent. pos r represents the positive reward obtained by the agent when it approaches the target landmark. col The term "collide" indicates the penalty caused by a collision.

[0021] The reward function for the blue team's unmanned surface vessel agent is:

[0022] r end =r blue +r col

[0023]

[0024] Where, r blue This indicates a positive reward for the intelligent agent.

[0025] Preferably, the method for obtaining the DRMRPG algorithm includes:

[0026] The multi-level experience replay strategy is introduced to replace the single-layer random experience replay pool in the MADDPG algorithm.

[0027] The dynamic soft update strategy is introduced into the MADDPG algorithm to guide the parameter updates of the Actor network and the Critic network:

[0028]

[0029] Where τ represents the current soft update parameter, γ represents the weight factor, f(x) represents the decay function, τ0 represents the soft update parameter at the previous time point, and τ e Indicates the truncation parameter;

[0030] The residual connection strategy is introduced into the MADDPG algorithm to extract the state space features of the agent and ensure gradient loss.

[0031] Preferably, the method for constructing the initial multi-unmanned surface vessel game decision model includes:

[0032] Set the initial positions and heading angles of the red and blue unmanned surface vessels;

[0033] The experience data is divided into layers using a multi-level experience replay pool and then divided into layers according to a ratio to obtain the initial multi-unmanned surface vessel game decision model.

[0034] Preferably, the method for obtaining the intelligent decision-making model includes:

[0035] Within the framework of intelligent game theory among multiple unmanned surface vessels (USVs), the USVs of the red and blue sides engage in combat in the initial scenario, generating adversarial data.

[0036] Based on the adversarial data, the data is split into a time sequence of fixed length T, and after being processed by max-min normalization, it is input into the initial multi-unmanned surface vessel game decision model, and the strategy and value of the current state information are output.

[0037] Using the strategy and the value calculation loss, the parameters of the initial multi-unmanned surface vessel game decision model are updated through gradient descent optimization to obtain the intelligent decision model.

[0038] The present invention also provides a multi-unmanned surface vessel game adversarial system based on the DRMRPG algorithm. The system applies the method described in any of the above-mentioned methods and includes: a framework construction module, an algorithm improvement module, a model construction module, a model training module, and a decision module.

[0039] The framework construction module is used to construct a multi-unmanned surface vessel intelligent game framework based on the DRMRPG algorithm.

[0040] The algorithm improvement module is used to introduce a multi-level experience replay strategy, a dynamic soft update strategy, and a residual connection strategy on the basis of the MADDPG algorithm to obtain the DRMRPG algorithm.

[0041] The model building module constructs an initial multi-unmanned surface vessel game decision-making model based on the DRMRPG algorithm.

[0042] The model training module uses the multi-unmanned surface vessel intelligent game framework to train the initial multi-unmanned surface vessel game decision model to obtain an intelligent decision model.

[0043] The decision-making module uses the intelligent decision-making model to conduct multi-unmanned surface vessel (USV) game competition and obtain game decisions.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] This invention utilizes a residual connection method to extract environmental features, thereby increasing the agent's cumulative reward. It also employs a dynamic soft update mechanism to improve the agent's sampling efficiency and stability, and a multi-level experience replay pool to enhance the utilization of historical experience, thus improving the algorithm's convergence performance. Overall, this invention achieves high reward values, high stability, and high efficiency, demonstrating its effectiveness. Attached Figure Description

[0046] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0048] Figure 2 This is a flowchart illustrating the method according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of a multi-unmanned surface vessel game framework based on the DRMRPG algorithm according to an embodiment of the present invention;

[0050] Figure 4 This is a simulation engine architecture diagram according to an embodiment of the present invention;

[0051] Figure 5 This is an information interaction diagram of the intelligent framework for multi-unmanned surface vessels based on the Unity engine, according to an embodiment of the present invention.

[0052] Figure 6 This is a schematic diagram of a three-degree-of-freedom model of an unmanned surface vessel according to an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of the MADDPG algorithm network architecture containing residual connections in an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of the multi-level experience playback pool structure according to an embodiment of the present invention;

[0055] Figure 9 This is a schematic diagram of the simulation engine environment in an embodiment of the present invention;

[0056] Figure 10 This is a schematic diagram of an unmanned surface vessel (USV) model according to an embodiment of the present invention, wherein a is a schematic diagram of the red team's USV model and b is a schematic diagram of the blue team's USV model;

[0057] Figure 11 This is a diagram of the naval warfare decision-making training structure based on the DRMRPG algorithm according to an embodiment of the present invention.

[0058] Figure 12 The image shows the training trajectory of the red and blue sides in naval battles according to an embodiment of the present invention. Among them, a is the combat trajectory of the red and blue sides within 200 time steps when using the original MADDPG algorithm, b is the combat trajectory of the red and blue sides within 200 time steps when using the MRDRPG algorithm, and c is the combat trajectory of the red and blue sides within 400 time steps when using the MRDRPG algorithm.

[0059] Figure 13 This is a simulation reward change diagram of the red agent in the MADDPG and DRMRPG algorithms according to an embodiment of the present invention.

[0060] Figure 14 This is a simulation reward change diagram of the blue-side intelligent agent using the MADDPG and DRMRPG algorithms in an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Example 1

[0064] In this embodiment, as Figure 1 , Figure 2 As shown, a multi-unmanned surface vessel (USV) game adversarial method based on the DRMRPG algorithm includes the following steps:

[0065] S1. Construct a multi-unmanned surface vessel intelligent game framework based on the DRMRPG algorithm.

[0066] The method for constructing a multi-unmanned surface vessel (USV) intelligent game framework includes: constructing a state space for a multi-USV game environment based on the DRMRPG algorithm to obtain the state information of the red and blue USV agents during the game; based on the state information, constructing an action space for the multi-USV game environment based on the DRMRPG algorithm to obtain the set of actions that the red and blue USV agents can execute during the game; and based on the action set, constructing a piecewise reward function to guide the red and blue USV agents to continuously optimize their strategies during the game, thus completing the construction of the multi-USV intelligent game framework.

[0067] In this embodiment, the naval warfare game framework based on deep reinforcement learning is as follows: Figure 3 As shown, the agent's state is integrated to describe the situation of the naval battle environment. During the game, the agent makes decisions based on a reinforcement learning model. In the interaction process, the agent changes the environmental state by performing actions and receives reward signals from the environment as feedback. The agent's goal is to find a strategy that maximizes the total reward obtained after performing a series of actions.

[0068] The naval warfare game framework comprises three fundamental elements: state space, action space, and reward function. This embodiment uses the Unity engine as the simulation engine. An environment object is virtualized within the Python reinforcement learning training code, and the TCP protocol suite is used to implement information exchange between the simulation engine and the virtual environment. The simulation engine generates real-time overall environment observations and serializes them into a JSON format byte stream, which is then transmitted via TCP to the virtual environment at the reinforcement learning end for parsing. After parsing the environment observations, the virtual environment assigns local observations to the agent, calculates the agent's state online, and collects the agent's instructions, sending them back to the simulation engine, thus realizing information exchange between the simulation engine and the virtual environment.

[0069] The simulation engine architecture diagram of this embodiment is as follows: Figure 4 As shown, a simulation environment architecture for a multi-unmanned surface vessel (USV) game problem based on the DRMRPG algorithm is proposed using the Unity engine. Overall, the simulation environment consists of a rendering module, entity objects, an object manager pool, TCP clients, and environment observation objects. The rendering module includes a skybox, lighting model, cloud and fog effects, and water rendering effects. The assets used are sourced from the Unity Store and are open-source and free. The environment rendering style and entity objects are as follows: Figure 9 As shown, the red and blue unmanned surface vessel models used are as follows: Figure 10 As shown, the object manager pool maintains three objects: the message manager, the environment manager, and the state information manager. The message manager parses the entity control information received by the TCP client process and sends control commands to the entity objects to control entities in the environment. The environment manager parses the environment control information received by the TCP client process and controls basic functions such as starting, stopping, and accelerating the environment. The state information manager serializes messages generated by the environment observation singleton object; it encapsulates and serializes the overall environment observations generated by the environment singleton object into a JSON-formatted TCP data stream for parsing by the virtual environment in Python. Entity objects control all possible environmental entities, such as agents and obstacles. Each entity object carries a dedicated control script to parse the commands transmitted by the message manager and control the entity. The TCP client process is located at the bottom of the simulation environment and is responsible for sending and receiving all communication information with Python. It is implemented using a client-server architecture and the TCP / IP protocol suite, and a dedicated private protocol is designed. The environment observation singleton is a singleton object created when the environment starts and is responsible for generating overall environment observation information.

[0070] Based on this, the information interaction diagram of the multi-unmanned surface vessel game framework based on the Unity engine in this embodiment of the invention is as follows: Figure 5As shown, the implementation logic is to virtualize a reinforcement learning environment on the Python side. It is actually a always-on server that waits for the simulation environment to connect in a blocking manner. When it receives a connection request, it starts the reinforcement learning process, collects and parses the observation information of the Unity simulation environment, and the agent generates control information based on the current local observations. This information is transmitted through the virtual environment, and the agent controls the entity and changes the state of the agent in the simulation environment.

[0071] ① Constructing a naval warfare state space

[0072] To better reflect the current combat situation, the state space must include not only its own state information, but also the state information of friendly forces, the state information of enemy forces, the relative relationship between the two sides, and the state of non-intelligent entities in the environment.

[0073] In this environment, the state space of a cooperative agent includes its own position, its own heading angle, its own vector velocity, the target location, and the distances of other agents from the target location. However, the state space of an adversarial agent does not include the target location. The collaborative state space of a single agent is shown below. In this embodiment, the state space has been normalized, which is very important in most cases.

[0074] The state space of the Red Team's unmanned surface vessel intelligent agent is:

[0075] state colla =[obs self obs other obs ext obs des ]

[0076] obs self =[Xpos self Ypos self Xvel self Yvel self yaw self ]

[0077] obbs ohter =[∑ i≠self Xpos i , ∑ i≠self Ypos i ]

[0078] obs ext =[∑ i dis(agt, des)]

[0079] obs des = [Xdes, Ydes]

[0080] The state space of the Blue Team's unmanned surface vessel intelligent agent is:

[0081] state adv =[obs self obs other obs ext ]

[0082] obs self =[Xpos self Ypos self Xvel self Yvel self yaw self ]

[0083] obs ohter =[∑ i≠self Xpos i , ∑ i≠self Ypos i ]

[0084] obs ext =[∑ i dis(agt, des)]

[0085] Among them, obs self Xpos represents the status information of the unmanned surface vessel itself. self and Ypos self Xvel represents the spatial coordinates of the unmanned surface vessel itself. self and Yvel self The yaw indicates the speed of the unmanned surface vessel itself. self This indicates the unmanned surface vessel's own path, obs other Xpos indicates the status information of other unmanned surface vessels. i and Ypos i Obs represents the spatial coordinates of other unmanned surface vessels. ext This indicates the relative position information between other unmanned surface vessels (USVs) and the target landmark. `dis()` represents the relative position information, `agt` represents other USVs, `des` represents the target landmark, and `obs` represents the target landmark. des This represents the location information of the target landmark entity in the environment. Xdes and Ydes represent the spatial location information of the target landmark.

[0086] ② Constructing a naval combat space

[0087] To simulate the control problem of ships under real-world conditions, a responsive model is used to describe the ship's motion. In this problem, the ship's motion is dynamically modeled using three degrees of freedom: forward, drift, and roll. The motion parameters are as follows: Figure 6 As defined.

[0088] Where x, y, and r are the x-axis position, y-axis position, and z-axis rotational velocity in the navigation coordinate system, respectively; Ψ is the bow direction; u and v are the x-axis velocity and y-axis velocity in the appendage coordinate system, respectively; and T(Ψ) is the transition matrix, defined as follows:

[0089]

[0090] The transformation relationship between the velocity vectors of the navigation coordinate system and the attached coordinate system is as follows:

[0091]

[0092] With the help of Unity Engine's physics engine, the above three-degree-of-freedom dynamic model of the unmanned surface vessel can be easily realized, and it can be controlled using a four-dimensional motion space, as shown in Table 1.

[0093] Table 1

[0094]

[0095] It should be noted that in the actual implementation, the details of the motion engine are hidden, and the motion control of the intelligent agent is achieved using upper-level speed calculation and steering torque calculation. Therefore, the actual control equations of the unmanned surface vessel are as follows:

[0096] F acc =max(0.1,F-(f) ship +f resis ))

[0097] vel cur =Lerp(vel) cur vel tar ,l1)

[0098] rot cur =Lerp(rot) cur rot tar l2)

[0099] rot cur (angle)→rot cur (rad)

[0100] rot axis =[u x u y u z ]

[0101]

[0102] vel rig =Q(rot) cur rot axis )*velcur

[0103] Among them, F acc For the resultant force, F is the driving force, f ship with f resis These are hull resistance and environmental resistance, respectively. tar The desired velocity scalar is given by F. acc Confirmed, vel cur rot represents the current velocity scalar calculated using Lerp interpolation, where l is the interpolation parameter. tar For the desired rotational scalar, rot cur Described in the form of quaternions, and decomposed into vectors, rot axis The new rotation quaternion, vel, is calculated. rig The final rigid body velocity vector is given by vel. cur and rot cur This is jointly determined. It is necessary to point out that, in the interaction between the algorithm and the environment, in order to simulate the synchronization of instructions in real-world situations, the discrete action instructions of the algorithm will be implemented by the environment as a problem of continuous control of the intelligent agent.

[0104] ③ Construct the reward function

[0105] In general, this embodiment uses a reward function design based on proximity rewards and collision penalties. The relative distance between the agent and the target landmark, as well as the relative distance between the agent and other agents, are used as indicators to estimate water surface confrontation. All agents in the environment use the same reward function. When a hostile agent approaches the target landmark, the friendly agent receives a negative reward, while when the friendly agent approaches the target landmark, it receives a positive reward. In addition, a collision penalty is designed for all agents. When two agents get too close, they receive a large penalty. The reward function is designed as follows: The reward function for the red unmanned surface vessel agent is:

[0106] r end =r adv +r pos +r col

[0107]

[0108] Where, r end This represents the final reward obtained by the Red Team's unmanned surface vessel. adv Indicates the location of the other party's unmanned surface vessel, pos des Indicates the location of the target location, pos i This represents the location of the i-th unmanned surface vessel (USV) in the red team's USV swarm, range map r represents the extent of the map. advr represents the negative reward that the target provides to the agent. pos r represents the positive reward obtained by the agent when it approaches the target landmark. col The reward function for the blue unmanned surface vessel (USV) agent is: (The function represents the penalty caused by a collision, where `collide` signifies the occurrence of a collision.)

[0109] r end =r adv +r col

[0110]

[0111] Where, r blue This indicates a positive reward for the intelligent agent.

[0112] S2. Based on the MADDPG algorithm, a multi-level experience replay strategy, a dynamic soft update strategy, and a residual connection strategy are introduced to obtain the DRMRPG algorithm.

[0113] The methods for obtaining the DRMRPG algorithm include: introducing a multi-level experience replay strategy to replace the single-layer random experience replay pool in the MADDPG algorithm; introducing a dynamic soft update strategy in the MADDPG algorithm to guide the parameter updates of the Actor network and Critic network; and introducing a residual connection strategy in the MADDPG algorithm to extract the state space features of the agent and ensure gradient loss.

[0114] In this embodiment, the DRMRPG algorithm is proposed. By introducing a multi-level experience replay strategy on the basis of MADDPG, the agent is guided to learn and update. A dynamic soft update strategy is used to guide the parameter update of the Actor network and Critic network. A residual connection strategy is used to extract the state space features of the agent cluster and ensure gradient loss. Figure 11 This is a diagram of the training structure for naval warfare decision-making based on the DRMRPG algorithm.

[0115] ① Residual connection method

[0116] This embodiment proposes a method for extracting features for all agents during execution using residual connections. In this method, each agent's observations are first passed through a multilayer perceptron. The first multilayer perceptron processes the local observations of agent i. Encoded as Subsequently, the local observations of agent i Will be connected via residuals and Therefore, the final observation value obtained by the Actor network of agent i is... It is a tensor whose features have been extracted once and added to the original state. This will be passed as a new state feature into the multilayer perceptron where the Actor network resides. The Actor network will then... Get an action These and They will be concatenated into a unified observation vector, and the Critic network will provide a score Q. i (s, a), then agent i will be scored by the Critic network Q. i (s, a) Update its own policy π i During the update process of the Actor and Critic networks, all layers are optimized using ReLU, and the loss function is defined by the mean squared error (MSE). The network architecture of the MADDPG algorithm with residual connections is as follows: Figure 7 As shown.

[0117] ② Dynamic soft update strategy

[0118] While the vanishing / exploding gradient problem is alleviated by introducing residual connections for feature extraction of state information, deep neural networks will inevitably experience slower training speeds due to the increase in the number of network layers, which is undesirable given limited computational resources. Therefore, a dynamic soft update strategy is introduced to increase the update frequency of the target network, accelerate sampling efficiency, and ultimately speed up algorithm convergence.

[0119] In updating the value network parameters, the ultimate goal is to make Q... ω (s, a) gradually approximates r+max a′ Q ω (s′, a′) When using a single network, training becomes highly unstable due to the updating of the TD error. Therefore, a two-layer network design is introduced, using both an online network and a target network. During training, the target network is initially fixed, and the parameters of the target network are updated asynchronously by updating the online network. This can alleviate the training instability problem to some extent. The soft update strategy further achieves this goal. In algorithms using the soft update strategy, the target network is updated slowly based on the soft update parameter τ, with the formula ω. - ←τω+(1-τ)ω - Typically, τ is a small number. This strategy ensures that the network parameters are updated slowly, thus maintaining the stability of the training.

[0120] However, this strategy also has its drawbacks. Traditional soft update strategies use smaller update steps during training. While this helps maintain the stability of the strategy, it also leads to a slower learning process, requiring more time to reach the desired performance. Therefore, this embodiment selects a dynamic adaptive parameter update method, the formula of which is shown below:

[0121]

[0122] Where τ represents the current soft update parameter, γ represents the weight factor, τ0 represents the soft update parameter at the previous time point, and τ e Indicates the truncation parameter. This represents the decay function, which increases continuously during training, causing g(x) = γ(τ0 / f(x)) to decrease rapidly in the early stages of training. To prevent the network from failing to update in the later stages of algorithm training, a cutoff parameter τ is set. e This allows the policy network to continue updating at a smaller magnitude in the later stages, ensuring the effectiveness of training. In summary, this embodiment dynamically adjusts the soft update strategy, enabling the policy network to update at a higher magnitude in the early stages of training, achieving rapid algorithm convergence, while updating at a smaller magnitude in the later stages maintains continuous algorithm updates, facilitating policy iteration and algorithm convergence. Experiments show that this dynamic update strategy is very effective in simple environments and also has some effect in complex environments.

[0123] ③ Multi-level experience replay pool strategy

[0124] Although the convergence speed of the algorithm is improved by incorporating a dynamic soft update strategy, this strategy may negatively impact the agent's policy construction in complex environments (usually due to the agent's dependence on the initial policy under the soft update strategy). However, to further improve the agent's sampling efficiency in the environment, a redesign of the widely used stochastic experience replay method is proposed, employing a multi-level experience replay pool strategy. The multi-level experience replay pool is as follows: Figure 8 As shown.

[0125] In this set of experience replay pools, the basic experience replay pool is responsible for storing all generated historical experience data. At the end of each episode (Eposide), the algorithm calculates the maximum cumulative experience sum for that episode. If this cumulative experience sum is greater than the cumulative experience of the previous episode, the preferred experience replay pool will absorb the experience data for that episode; otherwise, the secondary experience replay pool will receive the experience data for that episode. This means that the historical experience of each episode is divided into two quality experience replay pools through multi-level experience replay pools. In the early stages of training, the agent learns historical experience from the basic experience replay pool containing all experience data and updates the algorithm. This allows the agent to explore more extensively in the early stages of training, avoiding premature entrapment in local optima. This is the first level of experience replay. When the data size in the second-level experience pool reaches a certain level, data will be randomly filtered from the second-level experience replay pool, namely the preferred experience replay pool and the secondary experience replay pool. The principles of experience filtering are shown in Table 2.

[0126] Table 2

[0127]

[0128] S3. Based on the DRMRPG algorithm, construct an initial multi-unmanned surface vessel game decision model.

[0129] The method for constructing the initial multi-unmanned surface vessel (USV) game decision-making model includes: setting the initial positions and heading angles of the USVs on both sides; using a multi-level experience replay pool to divide the experience data into layers according to a ratio to obtain the initial multi-USV game decision-making model.

[0130] S4. The initial multi-unmanned surface vessel (USV) game decision-making model is trained using a multi-USV intelligent game framework to obtain an intelligent decision-making model.

[0131] The method for obtaining the intelligent decision-making model includes: in the framework of multi-unmanned surface vessel (USV) intelligent game, the red and blue USVs engage in adversarial combat in the initial scenario to generate adversarial data; based on the adversarial data, the data is split into a time series of fixed length T, and after being processed by max-min normalization, it is input into the initial multi-USV game decision-making model, outputting the strategy and value of the current state information; using the strategy and value to calculate the loss value, the parameters of the initial multi-USV game decision-making model are optimized and updated through gradient descent to obtain the intelligent decision-making model.

[0132] In this embodiment:

[0133] ① Initialize model parameters

[0134] In each training session, model parameters were set for both the red and blue teams. The environmental parameters are shown in Table 3, and the hyperparameters are shown in Table 4. To ensure the generalization ability of the agents and improve the diversity of strategies, the initial position and heading angle of the agents were randomly selected at the start of each game. The initial parameter settings are shown in Table 5.

[0135] Table 3

[0136]

[0137] Table 4

[0138]

[0139] Table 5

[0140]

[0141] ② Generate naval warfare data based on a naval warfare game theory framework

[0142] In the initial scenario, red and blue unmanned surface vessels (USVs) are based on a naval warfare game framework and use the proposed MADDPG algorithm network model to generate strategies for confrontation. Simulated naval warfare data is generated in real time through a simulation environment and input into the naval warfare strategy generation model.

[0143] ③ Utilize the DRMRPG algorithm to update the model and achieve autonomous generation of naval warfare intentions and strategies.

[0144] The sequence data is input into the proposed DRMRPG algorithm network model, and the current state information S is output. t Strategy π θ (a t |s t ) and value V(s t The DRMRPG algorithm uses policy and value to calculate loss value, and optimizes and updates model parameters θ through gradient descent to obtain an intelligent decision model.

[0145] S5. Use an intelligent decision-making model to conduct multi-unmanned surface vessel (USV) game competition and obtain game decisions.

[0146] Example 2

[0147] In this embodiment, to verify the effectiveness of the multi-unmanned surface vessel game algorithm based on the DRMRPG algorithm proposed in this invention, comparative simulation training experiments and ablation training experiments were conducted based on the DRMRPG algorithm proposed in this invention. Figure 12 It is a training trajectory map of naval battles between the red and blue forces. Figure 12This study analyzes the policies of agents trained using the MADDPG and DRMRPG algorithms in a cooperative and adversarial unmanned surface vessel (USV) environment. In this environment, due to the inherent physical properties of the USV, it cannot move arbitrarily but must adhere to the physical laws of the real world. Figure 12 In (a), the cooperative drone strategy is as follows: In the early stages of the game, drone A, whose spawn point is closer to the target landmark, accelerates towards it, while drone B, whose spawn point is farther away, cruises slowly. This creates a situation where the drones are positioned one in front of the other, making it impossible for the opposing drone C to identify the target landmark's location. However, as the game progresses, drone A shows no sign of slowing down and accelerates away even after reaching the vicinity of the target landmark. After A leaves the target landmark, drone B accelerates forward to approach it, securing rewards for the team. This strategy causes drone C to approach drone B's location in the late game, thus reducing the cooperative drones' rewards, which is not a desirable strategy. Figure 12 (b) and Figure 12 In (c), the cooperative unmanned surface vessel (USV) strategy shows some surprising improvements: In the short term, in the early stages of the game, USV A, which spawns closer to the target landmark, will still accelerate towards it. However, after approaching the target landmark, it does not blindly head towards the distance but instead performs a turning maneuver to guide USV C. Meanwhile, USV B faithfully sails towards the target location at a constant speed (slower than A). This strategy makes USV C unable to identify the location of the target landmark and thus it stays in place, unable to approach the target landmark. As a result, the cooperative agent's cumulative reward is greatly improved. The difference between the two USV strategies proves that the DRMRPG algorithm has a greater improvement in agent strategy improvement than the MADDPG algorithm. Figure 13 This is a simulation reward change diagram of the red agent in the MADDPG and DRMRPG algorithms according to an embodiment of the present invention; Figure 14 This is a simulation reward variation graph of the blue-side agent using the MADDPG and DRMRPG algorithms according to an embodiment of the present invention. The algorithm was trained in this environment for 5000 games, with each 5000 games taking approximately 25 hours. It can be seen that in a relatively complex control environment, the agent trained using the DRMRPG algorithm has a significantly higher average cumulative reward than the agent trained using the MADDPG algorithm, and also exhibits better algorithm stability. For the red-side agent, the agent trained using the DRMRPG algorithm approaches convergence around 1500 moves, and its stability after convergence is significantly higher than that of the agent trained using the MADDPG algorithm. For the blue-side agent, the agent trained using the DRMRPG algorithm also approaches convergence around 1500 moves, and its stability is again significantly higher than that of the agent trained using the MADDPG algorithm. This demonstrates the effectiveness and superiority of the DRMRPG algorithm.

[0148] To further verify the effectiveness of each optimization module in the multi-unmanned surface vessel (USV) game adversarial framework based on the DRMRPG algorithm proposed in this invention, ablation experiments were conducted on the DRMRPG algorithm. These experiments were based on the MADDPG algorithm, and a series of different algorithm variants were constructed by gradually adding or removing these three improvement strategies. Table 6 details the various algorithm variants used in the ablation experiments. The "√" symbol in the table indicates that the algorithm includes the corresponding improvement mechanism, while the "×" symbol indicates that it does not. Here, DMRPG refers to the improved MADDPG method that includes the dynamic soft update strategy and the multi-level experience replay pool strategy; RMRPG refers to the improved MADDPG method that includes the residual connection method and the multi-level experience replay pool strategy; and DRPG refers to the improved MADDPG method that includes the dynamic soft update strategy and the residual connection method.

[0149] Table 6

[0150]

[0151] Specifically: (a) The RMRPG algorithm, lacking a dynamic soft update strategy, performed poorly in terms of convergence. RMRPG converged slowly, and the algorithm exhibited significant fluctuations during training, demonstrating the positive impact of the dynamic soft update strategy on the convergence speed. (b) The DRPG algorithm, lacking a multi-level experience replay pool strategy, showed unacceptable stability. With large datasets, the multi-level experience replay pool strategy significantly improved the algorithm's convergence performance. (c) The DMRPG algorithm, lacking a residual connection method, showed a significant decrease in cumulative reward and poor convergence performance against adversaries, demonstrating the effectiveness of the residual connection method. These experimental results confirm the correctness and feasibility of the DRMRPG algorithm.

[0152] Through comparative and ablation experiments, the average reward value and improvement rate of each algorithm when it converges (or reaches the end of the training period if it does not converge) are finally obtained, as shown in Table 7.

[0153] Table 7

[0154]

[0155] Data shows that, while maintaining algorithm convergence, the DRMRPG algorithm significantly improves the reward value of agents. Compared with the MADDPG algorithm, the cumulative reward of collaborator agents trained by the DRMRPG algorithm is increased by more than ten times.

[0156] Example 3

[0157] In this embodiment, a multi-unmanned surface vessel (USV) game adversarial system based on the DRMRPG algorithm includes: a framework construction module, an algorithm improvement module, a model construction module, a model training module, and a decision-making module.

[0158] The framework construction module is used to build a multi-UAV intelligent game framework based on the DRMRPG algorithm. The algorithm improvement module introduces multi-level experience replay strategy, dynamic soft update strategy, and residual connection strategy into the MADDPG algorithm to obtain the DRMRPG algorithm. The model construction module builds an initial multi-UAV game decision model based on the DRMRPG algorithm. The model training module trains the initial multi-UAV game decision model using the multi-UAV intelligent game framework to obtain an intelligent decision model. The decision-making module uses the intelligent decision model to conduct multi-UAV game adversarial play to obtain game decisions.

[0159] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-unmanned surface vessel (USV) game adversarial method based on the DRMRPG algorithm, characterized in that, Includes the following steps: Construct a multi-unmanned surface vessel intelligent game framework based on the aforementioned DRMRPG algorithm; Based on the MADDPG algorithm, a multi-level experience replay strategy, a dynamic soft update strategy, and a residual connection strategy are introduced to obtain the DRMRPG algorithm. Based on the DRMRPG algorithm, an initial multi-unmanned surface vessel game decision model is constructed. The initial multi-unmanned surface vessel (USV) game decision model is trained using the aforementioned multi-USV intelligent game framework to obtain an intelligent decision model. The intelligent decision-making model is used to conduct multi-unmanned surface vessel (USV) game competition to obtain game decisions. The method for constructing the multi-unmanned surface vessel intelligent game framework includes: Construct the state space of the multi-unmanned surface vessel (USV) game environment based on the DRMRPG algorithm, and obtain the state information of the red and blue USV agents during the game. Based on the state information, an action space for a multi-unmanned surface vessel (USV) game environment based on the DRMRPG algorithm is constructed to obtain the set of actions that can be performed by the red and blue USV agents during the game. Based on the set of actions, a segmented reward function is constructed to guide the red and blue unmanned surface vessel agents to continuously optimize their strategies during the game, thus completing the construction of the multi-unmanned surface vessel intelligent game framework. Methods for constructing piecewise reward functions include: The reward function for the Red Team's unmanned surface vessel agent is: r end ＝r adv +r pos +r col Where, r end This represents the final reward obtained by the Red Team's unmanned surface vessel. adv Indicates the location of the other party's unmanned surface vessel, pos des Indicates the location of the target location, pos i This represents the location of the i-th unmanned surface vessel (USV) in the red team's USV swarm, range map r represents the extent of the map. adv r represents the negative reward that the target provides to the agent. pos r represents the positive reward obtained by the agent when it approaches the target landmark. col The term "collide" indicates the penalty caused by a collision. The reward function for the blue team's unmanned surface vessel agent is: r end ＝r blue +r col Where, r blue This indicates a positive reward for the agent. The method for obtaining the DRMRPG algorithm includes: The multi-level experience replay strategy is introduced to replace the single-layer random experience replay pool in the MADDPG algorithm. The dynamic soft update strategy is introduced into the MADDPG algorithm to guide the parameter updates of the Actor network and the Critic network: Where τ represents the current soft update parameter, γ represents the weight factor, f(x) represents the decay function, τ0 represents the soft update parameter at the previous time point, and τ e Indicates the truncation parameter; The residual connection strategy is introduced into the MADDPG algorithm to extract the state space features of the agent and ensure gradient loss.

2. The multi-unmanned surface vessel (USV) game adversarial method based on the DRMRPG algorithm according to claim 1, characterized in that, The method for constructing the initial multi-unmanned surface vessel game decision model includes: Set the initial positions and heading angles of the red and blue unmanned surface vessels; The experience data is divided into layers using a multi-level experience replay pool and then divided into layers according to a ratio to obtain the initial multi-unmanned surface vessel game decision model.

3. The multi-unmanned surface vessel (USV) game adversarial method based on the DRMRPG algorithm according to claim 1, characterized in that, The methods for obtaining the intelligent decision-making model include: Within the framework of intelligent game theory among multiple unmanned surface vessels (USVs), the USVs of the red and blue sides engage in combat in the initial scenario, generating adversarial data. Based on the adversarial data, the data is split into a time sequence of fixed length T, and after being processed by max-min normalization, it is input into the initial multi-unmanned surface vessel game decision model, and the strategy and value of the current state information are output. Using the strategy and the value calculation loss, the parameters of the initial multi-unmanned surface vessel game decision model are updated through gradient descent optimization to obtain the intelligent decision model.

4. A multi-unmanned surface vessel (USV) game-playing adversarial system based on the DRMRPG algorithm, wherein the system applies the method described in any one of claims 1-3, characterized in that, include: The framework consists of a framework building module, an algorithm improvement module, a model building module, a model training module, and a decision-making module. The framework construction module is used to construct a multi-unmanned surface vessel intelligent game framework based on the DRMRPG algorithm. The algorithm improvement module is used to introduce a multi-level experience replay strategy, a dynamic soft update strategy, and a residual connection strategy on the basis of the MADDPG algorithm to obtain the DRMRPG algorithm. The model building module constructs an initial multi-unmanned surface vessel game decision-making model based on the DRMRPG algorithm. The model training module uses the multi-unmanned surface vessel intelligent game framework to train the initial multi-unmanned surface vessel game decision model to obtain an intelligent decision model. The decision-making module uses the intelligent decision-making model to conduct multi-unmanned surface vessel (USV) game competition and obtain game decisions.

Images