Vessel formation-oriented MADDPG super-network parameter optimization method based on differential evolution

Abstract

The invention discloses a naval vessel formation-oriented MADDPG super-network parameter optimization method based on differential evolution, and the method comprises the steps: constructing a naval vessel formation cooperative combat environment, setting a naval vessel as an intelligent agent, and setting task information for the intelligent agent; loading task information, and generating an initial strategy; according to the initial strategy, the intelligent agent interacts with the environment and collects data; training super-network parameters of the MADDPG model according to the data, and optimizing the super-network parameters by adopting a differential evolution algorithm to obtain an optimal parameter combination; loading the optimal parameter combination into the MADDPG model, and evaluating the performance of the intelligent agent to obtain an evaluation result; and the initial strategy is optimized according to the evaluation result, the intelligent agent interacts with the environment again, new data is collected, the super-network parameters are adjusted according to the new data, and the super-network parameters are iteratively optimized until a final training model is obtained. According to the method, coevolution optimization of the multi-agent strategy network is realized, and the convergence speed and stability of the strategy network are improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a MADDPG hypernetwork parameter optimization method based on differential evolution for a naval fleet. Background Art

[0002] With the rapid development of intelligent naval warfare, coordinated operations of naval fleets have placed higher demands on the autonomous decision-making capabilities of multi-agent systems (MAS). The Multi-Agent Deep Reinforcement Learning (MADRL) algorithm, exemplified by the Multi-Agent Deep Deterministic Policy Gradient (MADDPG), has become a core technology for intelligent command of naval fleets due to its advantages in distributed collaborative decision-making. However, complex naval combat scenarios, such as dynamic battlefield environments and enemy countermeasures, exhibit nonlinearity, high-dimensional state spaces, and sparse rewards. These drawbacks of MADDPG include: ① Hyperparameters rely on manual experience, making them difficult to adapt to battlefield dynamics; ② Initially, agents explore blindly, resulting in slow policy convergence and difficulty coping with collaborative tasks under sparse rewards; and ③ Traditional optimization methods are computationally expensive and cannot meet the demands of real-time combat.

[0003] At present, the distributed collaborative decision-making algorithms used in the MADRL field, such as the MADDPG algorithm, use a "centralized training-distributed execution" model to achieve collaborative optimization of policy networks using a shared experience pool and a joint state-action space. Its core mechanisms include: ① Actor-Critic (AC) network architecture, which is responsible for policy generation (Actor) and value evaluation (Critic) respectively; ② Experience replay mechanism to alleviate the correlation of training data; ③ Exploration noise injection to balance policy exploration and utilization. In the invention of hyperparameter optimization, the existing technology mainly adopts two types of methods: one is gradient-based local optimization, such as the Adam optimizer to iteratively update the network weights; the other is traditional global optimization, such as grid search, random search and other methods, which discretely sample hyperparameters such as learning rate, number of network layers, and exploration noise intensity.

[0004] However, gradient-based local optimization methods rely on information about the objective function's gradient, making them prone to falling into local optima or saddle points in non-convex solution spaces. Furthermore, the optimization results are highly sensitive to hyperparameters such as initialization values ​​and learning rates: A too-small learning rate leads to slow convergence, while a too-large one can cause oscillation or even divergence, making them difficult to adapt to the real-time adjustments required in dynamic battlefield environments. While traditional global optimization methods (such as grid search and random search) can explore the hyperparameter space, the computational cost of the former increases exponentially with the parameter dimension, while the latter lacks systematic guidance and relies on manually preset parameter ranges. Both methods struggle to capture the dynamic relationship between battlefield characteristics and hyperparameters, resulting in insufficient scenario adaptability of the optimization results. Summary of the Invention

[0005] The present invention aims to solve the technical problems of the MADDPG algorithm in naval fleet coordinated operations, such as the non-convex solution space being prone to falling into local optimality and the lack of dynamic adaptability of hyperparameters. The present invention provides a MADDPG hypernetwork parameter optimization method based on differential evolution for naval fleets. The technical solution adopted is:

[0006] A MADDPG hypernetwork parameter optimization method based on differential evolution for ship formations includes the following steps:

[0007] S1. Build a collaborative combat environment for a fleet of ships, set the ships as intelligent agents, and set task information for the intelligent agents;

[0008] S2. Load the task information and generate an initial strategy;

[0009] S3. Based on the initial strategy, the agent interacts with the environment and collects data;

[0010] S4. Training the hypernetwork parameters of the MADDPG model according to the data, and optimizing the hypernetwork parameters using a differential evolution algorithm to obtain an optimal parameter combination;

[0011] S5: Loading the optimal parameter combination into the MADDPG model, and evaluating the performance of the agent to obtain an evaluation result;

[0012] S6: Optimize the initial strategy based on the evaluation results, let the agent interact with the environment again and collect new data, adjust the hypernetwork parameters based on the new data, and repeat steps S4 to S6 to iteratively optimize the hypernetwork parameters until the final training model is obtained.

[0013] In one embodiment of the present invention, step S2 includes:

[0014] S21. Abstracting the process of interaction between the agent and the environment into a partially observable Markov decision process, wherein the partially observable Markov decision process is represented by a five-tuple: <state space, action space, transition probability function, reward function, discount factor of cumulative reward>;

[0015] S22. Setting relevant parameters, including MADDPG hypernetwork parameters, training parameters, and evaluation parameters for measuring the agent;

[0016] S23, loading the task information;

[0017] S24. Create a MADDPG algorithm based on the relevant parameters, initialize the network architecture, load the state space and the action space, and generate an initial strategy based on the current state of the agent.

[0018] In one embodiment of the present invention, step S21 includes:

[0019] The state space represents all states of the agent, including battlefield environment information and ship state information;

[0020] The action space represents the set of all actions that the agent can perform, including tactical maneuvers, combat-related actions, and collaborative actions;

[0021] The transition probability function describes the probability that the agent will transition to the next state after performing an action in the current state;

[0022] The reward function calculates and returns the reward obtained by the agent based on the change in the state of the environment after the agent performs the action;

[0023] The discount factor of the cumulative reward is used to calculate the expected cumulative benefit.

[0024] In one embodiment of the present invention, step S3 includes:

[0025] In each step of the interaction between the agent and the environment, the agent perceives the current state of the environment s through sensors and selects an optimal action a according to the initial strategy π(a|s);

[0026] After the action is executed, the agent observes the change in the environment state s' and the reward r obtained, and collects relevant data;

[0027] The data includes state s, action a and reward r.

[0028] In one embodiment of the present invention, step S4 includes:

[0029] S41, the MADDPG model is based on the Actor-Critic network structure, and the intelligent agent uses the Actor-Critic network structure to reconstruct the dynamic battlefield situation;

[0030] S42. Introduce the differential evolution algorithm to optimize the Actor-Critic hypernetwork parameters and obtain the optimal parameter combination.

[0031] In one embodiment of the present invention, step S41 includes:

[0032] S411. Parameterized Actor Network Strategy Function π θ (a|s) represents the probability distribution or probability density of the action to be performed in the current state. The optimization goal is to maximize the expected cumulative reward, which is achieved through the policy gradient theorem:

[0033]

[0034] In formula (1), represents the gradient of the parameter θ, J(θ) represents the objective function, and the strategy π θ The expected cumulative reward of represents the gradient of the objective function J(θ) with respect to the parameter θ; s~ρ π Represents the state distribution ρ of state s under strategy π π Sampling under a~π θ represents the policy π for action a according to parameter θ θ (a|s) sampling, represents the expectation after joint sampling of s and a; Q π (s,a) is the value function of the state-action pair (s,a) under the policy π, which evaluates how “good” the action a is in the state s; π θ (a|s) represents the probability of taking action a in state s with a policy parameter θ; logπ θ (a|s) represents the log probability of the strategy, Represents the gradient of the logarithmic probability with respect to the parameter θ, which measures the direction and magnitude of the impact of parameter changes on the action probability;

[0035] The Q value is estimated using the time difference method, and the gradient update direction is Denotes the strategy π θThe logarithmic gradient of (s,a) with respect to the parameter θ; r represents the immediate reward, which is the immediate reward obtained by the agent from the environment after performing action a in state s; γ represents the discount factor of the cumulative reward (0≤γ≤1), which is used to balance the importance of current rewards and future rewards. The closer γ is to 0, the more the agent pays attention to the current reward, and the closer γ is to 1, the more attention is paid to long-term rewards; V(s′) represents the value of the transferred state s' estimated by the Critic network, which represents the expected future cumulative reward of following the current strategy starting from state s'; V(s) represents the value of the current state s estimated by the Critic network; r+γV(s′)-V(s) represents the error of the temporal difference method, which is an approximation of the advantage function; r+γV(s′) is the target value estimated by the temporal difference method. After subtracting V(s), it measures the deviation of the value estimate of state s. A positive deviation indicates that action a performs better than the average in state s and the probability should be increased, while a negative deviation is the opposite;

[0036] The gradient update direction guides the parameter θ to update in the direction of increasing the probability of high-value actions by multiplying the logarithmic gradient of the policy with the error of the temporal difference method, optimizing the policy to maximize the long-term cumulative return;

[0037] S412. Parameterize the state value V of the Critic network φ (s) or state-action value Q φ (s,a);

[0038] The loss function of the state value is the square of the error of the temporal difference method, expressed as:

[0039] L(φ)=E s,a,s′,r～π [(r+γV φ (s′)-V φ (s)) 2 ] (2)

[0040] In formula (2), φ is the parameter, weight or bias of the critic network; V φ (s) represents the value estimate of the critic network with parameter φ for state s; r is the immediate reward obtained after executing action a in state s; γ represents the discount factor of the cumulative return (0≤γ≤1); s′ represents the next state transferred to after executing action a; r+γV φ (s′) represents the target of the temporal difference method, the estimate of the value of the next state; r+γV φ (s′)-V φ (s) represents the error of the time difference method; E s,a,s′,r～π [·] represents the expected value of the square of the error of the temporal difference method after jointly sampling the state s and the action a, and the average result in the long-term average sense;

[0041] The loss function is the expectation of the square of the error of the time difference method, and the goal is to make the state value estimate V φ (s) Approaching the time difference method target r+γV φ (s′);

[0042] The gradient update formula is:

[0043]

[0044] In formula (3), β is the learning rate of the critic network; is the state value function V φ (s) The gradient of the parameter φ, indicating the direction of parameter update; r+γV φ (s′)-V φ (s) is the error of the temporal difference method, which serves as the weight of the gradient and determines the update amplitude and direction;

[0045] The parameters φ of the critic network follow the gradient The direction of the update is determined by the time difference method error and the learning rate β.

[0046] In one embodiment of the present invention, step S42 includes:

[0047] S421, encoding the weights and biases of the hypernetwork into a set of vectors, each complete vector representing a hypernetwork parameter model and also serving as an individual in the evolutionary algorithm population for iteration;

[0048] S422, using the mutation strategy DE / rand / 1 to generate mutated network parameter individuals to obtain a mutated population;

[0049] S423. A binomial crossover method is used on the population to obtain a new population, which is expressed as:

[0050]

[0051] In formula (4), x i Represents the i-th individual in the original population, with dimension D, x i =[x i,1 ,x i,2 ,...,x i,D ];v i represents the i-th candidate individual generated by the mutation operation; u i represents the new individual generated after the crossover; j represents the dimension index, 1≤j≤D, representing the jth component in the individual vector; rand j represents a random number uniformly distributed in the interval [0,1], and each dimension j is generated independently; CR represents the crossover rate, which ranges from [0,1] and controls the strength of the crossover; jrand Represents a randomly selected dimension index;

[0052] S424. Calculate the population fitness value of the cumulative return after the crossover step, and compare it with the fitness value obtained from the initial population to select excellent individuals as the initial population of the next generation.

[0053] In one embodiment of the present invention, in step S421, the individual is constructed as follows:

[0054] x i ={w1_weight,w1_bias,w2_weight,w2_bias,...,b 22 _weight,b 22 _bias} (5)

[0056] In formula (5), x i represents the individuals in the population, w j _weight, w j _bias represents the super network weight and bias value for generating the j-th layer weight of the Actor-Critic network; b j _weight, b j _bias represents the hypernetwork weight and bias value for generating the j-th layer bias of the Actor-Critic network, j∈[1,2]; since the bias W2 of the Actor-Critic network is calculated by a two-layer linear network, the hypernetwork parameter hyper_b2 is divided into two layers, b 21 with b 22 They represent two linear networks that generate the bias of the second layer of the Actor-Critic network.

[0057] In one embodiment of the present invention, in step S422, the network parameter individual is represented as:

[0058] v i (t+1)=x r1 (t)+F(x r2 (t)-x r3 (t)) (6)

[0059] In formula (6), F is the scaling factor, v i (t+1) represents the i-th mutant individual in the t+1 generation population, x r1 (t), x r2 (t), x r3 (t) represents three random individuals in the population of generation t, where r1≠r2≠r3.

[0060] In one embodiment of the present invention, in step S424, the fitness function is set to the average reward of the multi-agent system based on the MADDPG algorithm over n training steps, which is expressed as:

[0061]

[0062] In formula (7), fitness represents the fitness value, that is, the average cumulative reward of all agents; i represents the index of the agent, in the interval [1, n], n is the number of agents, and reward represents the reward of the current agent;

[0063] The reward is composed of:

[0064] reward=detect_reward+destroy_reward-be_killed_reward-border_reward (8)

[0066] In formula (8), detect_reward represents the reward for detecting the enemy, destroy_reward represents the reward for destroying the enemy, be_killed_reward represents the reward for the Red agent being killed, and border_reward represents the reward for the Red agent touching the border of the combat area.

[0067] Beneficial effects of the present invention:

[0068] The present invention is a MADDPG hypernetwork parameter optimization method based on differential evolution for ship formations. It innovatively integrates the global search advantages of the differential evolution algorithm (DE) and the distributed strategy learning framework of MADDPG to construct a DE-MADDPG collaborative evolution optimization architecture adapted to naval combat scenarios. By designing a hyperparameter encoding mechanism driven by battlefield characteristics and utilizing the adaptive mutation-crossover-selection operator of DE, the collaborative evolution optimization of the multi-agent strategy network is realized. This breaks through the local convergence limitations of traditional gradient optimization methods in non-convex solution spaces, improves the convergence speed and stability of the strategy network, establishes a dynamic mapping relationship between battlefield characteristics and hyperparameters, enhances the collaborative decision-making efficiency in complex environments, and meets the autonomous adaptation needs of real-time collaborative combat of ship formations. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Flowchart of a MADDPG hypernetwork parameter optimization method for a naval fleet based on differential evolution provided by an embodiment of the present invention;

[0070] Figure 2 This is a comparison chart of the winning rates of the MADDPG hypernetwork parameter optimization method based on differential evolution for a naval fleet provided by an embodiment of the present invention;

[0071] Figure 3 This is a comparison chart of the returns of the MADDPG hypernetwork parameter optimization method based on differential evolution for a ship formation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] Existing gradient-based local optimization methods rely on information about the objective function's gradient, making them prone to falling into local optima or saddle points in non-convex solution spaces. Furthermore, optimization results are highly sensitive to hyperparameters such as initialization values ​​and learning rates: A too-small learning rate leads to slow convergence, while a too-large one can cause oscillation or even divergence, making them difficult to adapt to the real-time adjustments required in dynamic battlefield environments. While traditional global optimization methods (such as grid search and random search) can explore the hyperparameter space, the computational cost of the former increases exponentially with the parameter dimension, while the latter lacks systematic guidance and relies on manually preset parameter ranges. Both methods struggle to capture the dynamic relationship between battlefield characteristics and hyperparameters, resulting in insufficient scenario-specific adaptability of optimization results.

[0074] This paper uses the swarm intelligence search characteristics of the differential evolution (DE) algorithm to encode the super-network parameters of the neural network into population individuals. Through mutation, crossover, and selection operations, it realizes efficient search of the global parameter space, breaking through the local optimal limit. Compared with the traditional parameter adjustment method that relies on manual experience and trial and error, it shows dual evolutionary advantages:

[0075] On the one hand, through the population's mutation and crossover mechanisms, local optimal constraints can be overcome, enabling optimization over a wider range of hyperparameter spaces. On the other hand, by maintaining an evolutionary population composed of multiple candidate parameter groups, the algorithm enables simultaneous evaluation and optimization of multiple parameter combinations, significantly improving optimization efficiency. This "exploration-exploitation" dual-driven evolutionary mechanism not only significantly improves the agent's efficiency in exploring the environment in the early stages of training, but also provides a more evolutionarily advantageous decision-making solution for the intelligent game of ship formations by improving global rewards. The following describes in detail the MADDPG hypernetwork parameter optimization method for ship formations based on differential evolution, provided by the present invention.

[0076] The present invention provides a MADDPG hypernetwork parameter optimization method based on differential evolution for ship formations. By designing a hypernetwork parameter encoding mechanism driven by battlefield characteristics and utilizing the parallel global search capability of the DE algorithm to dynamically optimize the Actor-Critic network hyperparameter combination, a collaborative evolution framework adapted to naval battle scenarios is constructed.

[0077] Refer to the attached Figure 1The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formations includes the following steps:

[0078] S1. Build a collaborative combat environment for the fleet, set the ships as intelligent agents, and set task information for the intelligent agents;

[0079] S2, load task information and generate initial strategy;

[0080] S3, based on the initial strategy, the agent interacts with the environment and collects data;

[0081] S4. Train the hypernetwork parameters of the MADDPG model based on the data, and use the differential evolution algorithm to optimize the hypernetwork parameters to obtain the optimal parameter combination;

[0082] S5: Load the optimal parameter combination into the MADDPG model and evaluate the agent’s performance to obtain the evaluation results;

[0083] S6: Optimize the initial strategy based on the evaluation results, let the agent interact with the environment again and collect new data, adjust the hypernetwork parameters based on the new data, and repeat steps S4 to S6 to iteratively optimize the hypernetwork parameters until the final training model is obtained.

[0084] This example describes in detail the complete process of intelligent agent simulation in a naval combat simulation environment. First, a highly realistic naval combat simulation environment is constructed. This environment is a three-dimensional space that includes key elements such as the opposing ships, terrain, weather conditions, and the electromagnetic environment. Within this environment, the ships are configured as intelligent agents with a continuous action space. These agents can perceive the environmental state, select and execute actions, and learn based on environmental feedback.

[0085] After the environment is set up, the agent's state space and action space are defined. The state space contains information such as the agent's current position, speed, heading, weather conditions, and electromagnetic environment. The action space includes continuous actions such as heading adjustment and speed control. Next, the agent's mission objectives are set. In the simulation environment, the agent's primary task is to maximize the formation's collaborative mission objectives in a complex electromagnetic environment. To achieve this task, the agent needs to continuously perceive the environmental state, select the optimal action strategy based on the current state, and execute the selected action.

[0086] Step S2 of the present invention comprises:

[0087] S21. The process of interaction between the agent and the environment is abstracted into a partially observable Markov decision process, which is represented by a five-tuple: <state space, action space, transition probability function, reward function, discount factor of cumulative reward>.

[0088] S22. Setting relevant parameters, including MADDPG hypernetwork parameters, training parameters, and evaluation parameters for measuring the agent;

[0089] S23, loading task information;

[0090] S24. Create the MADDPG algorithm based on relevant parameters, initialize the network architecture, load the state space and action space, and generate the initial strategy based on the current state of the agent.

[0091] In order to more effectively model and analyze the combat process in complex ship formation combat scenarios, based on the OODA theory, which is a rapid cycle of four steps: observation, orientation, decision, and action, the combat process is abstracted into a partially observable multi-agent Markov decision process, using a five-tuple.<S,A,T,R,γ> To describe.

[0092] The state space is represented by S, which contains all possible states of the agent, including battlefield environment information (such as enemy ship distribution, sea conditions, weather, etc.) and ship status information (such as position, heading, speed, weapon equipment status, fuel reserves, etc.), which can be expressed as S = {s env ,s ship1 ,s ship2 ,…,s shipn}.

[0093] The action space is represented by A, which is the set of all actions that the agent can perform, including tactical maneuvers (such as changing course, adjusting speed, etc.), combat-related actions (such as launching weapons, turning on electronic countermeasures equipment, etc.) and collaborative actions (such as formation reorganization, information sharing, etc.), which can be expressed as A = {a maneuver ,a combat ,a cooperation}.

[0094] The transition probability function is represented by T, denoted as T(s′|s,a), which describes the probability of transitioning to the next state s′∈S after executing action a∈A in the current state s∈S. It is affected by many complex factors such as enemy countermeasures, changes in sea conditions and meteorological conditions, and needs to be estimated through a large amount of historical data and simulation experiments.

[0095] The single-step reward is represented by R, denoted as R(s,a). It is the immediate feedback that the agent receives from the environment after performing action a in state s. According to the combat mission objectives, a positive reward is obtained for successfully completing a beneficial task, and a negative reward is obtained for damage to the agent or failure to complete the task.

[0096] The discount factor of the cumulative reward is denoted by γ, γ∈[0,1], which is used to weigh the importance of current and future rewards. When γ is close to 1, the agent focuses on long-term combat effects, and when it is close to 0, it tends to pursue immediate rewards. The main goal is to learn an optimal policy π* so that the expected cumulative reward Maximize, where π is the policy, s t is the state at time t, a t is the action performed at time t, E represents the expected operation, and π* is found through continuous learning and optimization strategies to achieve better results in ship formation combat.

[0097] The present invention constructs a Markov decision process based on the OODA loop, integrates multi-source battlefield data in real time, and forms a closed-loop feedback mechanism by optimizing the reinforcement learning decision network, which significantly improves the formation's response speed to sudden threats.

[0098] To ensure the stability and consistency of program execution, a series of program parameters must be set. These parameters include model parameters, such as the hypernetwork parameters of the MADDPG model, which are optimized during training to find the optimal solution; training parameters, including the number of iterations, learning rate, and batch size, which control the efficiency and progress of model training; and evaluation parameters used to measure agent performance, such as survival time and task completion. These parameters are important for optimizing and improving agent strategies.

[0099] In one embodiment of the present invention, the specific method for loading task information is:

[0100] First, the task name being reset is printed to indicate the currently loaded environment. Next, a pre-set scenario file in the scenario format is opened. This file contains all task information, such as the environment's configuration details, observation space, and action space. The file is read and parsed to extract relevant information about the current task, such as the environment's configuration, description, and examples, in order to load the current task.

[0101] Create a MADDPG algorithm instance based on the specified parameters and initialize the Actor-Critic network architecture. Load the state space (position, heading, etc.) and action space (maneuver control, collaborative instructions) parameters defined in step S2, randomly initialize the network weights, and generate an initial policy based on the agent's current state, providing a basis for subsequent decision-making in the agent's interactions.

[0102] The intelligent agent interacts with the environment, integrating data from the ship's radar, optoelectronics, and navigation systems in real time. Radar provides long-range target detection information, optoelectronics provide precise target images and optical signatures, and the navigation system provides the ship's position and heading. Under strong electromagnetic interference, radar signals are intermittent. The intelligent agent captures brief radar gaps in the full-band interference. It also combines historical data to analyze enemy ship movement patterns and habits, predicting their position and intentions as accurately as possible.

[0103] Step S3 of the present invention involves: During each interaction between the agent and the environment, the agent senses the current environmental state s through sensors and selects an optimal action a based on the initial policy π(a|s). After the action is executed, the agent observes the change in the environmental state s' and the reward r received, and collects relevant data. This data includes the state s, action a, reward r, etc. This data is used for subsequent model training and policy optimization.

[0104] When the data stored in the experience replay pool reaches a threshold, MADDPG model training begins. A batch of samples are randomly sampled from the collected data to train the model's hypernetwork parameters. The hypernetwork parameters are then optimized using a differential evolution algorithm, which iterates repeatedly to find the hyperparameter combination with the highest fitness value. In each iteration, the agent's performance under different parameter combinations is evaluated based on the fitness function, and the best performing parameter combination is selected as the starting point for the next iteration.

[0105] Specifically, step S4 of the present invention includes:

[0106] The S41 and MADDPG models are based on the Actor-Critic network structure, and the intelligent agent uses the Actor-Critic network structure to reconstruct the dynamic battlefield situation.

[0107] The intelligent agent uses an actor-critic network to process partially observable states and dynamically reconstruct the battlefield situation. In actual combat, the intelligent agent often only has partial battlefield information. The actor network determines actions based on the current partial observations, and the critic network evaluates the value of the action. When communication is interrupted, the intelligent agent infers the battlefield situation based on its own sensor data and previous situation information. It then automatically switches to a silent coordinated formation, with each ship reducing electromagnetic radiation and communication activity, and operating autonomously according to predetermined rules and tactics to reduce the probability of enemy detection.

[0108] S411. Parameterized Actor Network Strategy Function π θ (a|s) represents the probability distribution or probability density of the action to be performed in the current state. The optimization goal is to maximize the expected cumulative reward, which is achieved through the policy gradient theorem:

[0109]

[0110] In formula (1), Indicates the gradient of the parameter θ, J(θ) represents the objective function, that is, the strategy π θ The expected cumulative reward of represents the gradient of the objective function J(θ) with respect to the parameter θ; s~ρ π Represents the state distribution ρ of state s under strategy π π Sampling under a~π θ represents the policy π for action a according to parameter θ θ (a|s) sampling, represents the expectation after joint sampling of s and a; Q π (s,a) is the value function of the state-action pair (s,a) under the policy π, which evaluates how “good” the action a is in the state s; π θ (a|s) represents the policy with parameter θ, that is, the probability of taking action a in state s; logπ θ (a|s) represents the log probability of the strategy. It represents the gradient of the log probability with respect to the parameter θ, and measures the direction and magnitude of the impact of parameter changes on the action probability.

[0111] When using the temporal-difference (TD) method to estimate the Q value, the gradient update direction is Denotes the strategy π θ The logarithmic gradient of (s, a) with respect to the parameters θ; r represents the immediate reward, i.e., the reward the agent receives from the environment after performing action a in state s; γ represents a discount factor for cumulative rewards (0 ≤ γ ≤ 1), which balances the importance of current rewards with future rewards. The closer γ is to 0, the more the agent prioritizes current rewards; the closer it is to 1, the more it prioritizes long-term rewards. V(s′) represents the value of the post-transition state s' estimated by the critic network, representing the expected cumulative future reward from following the current policy starting from state s'; V(s) represents the value of the current state s estimated by the critic network; r+γV(s′)-V(s) represents the time-delay error, an approximation of the advantage function; r+γV(s′) is the target value estimated by the time-delay method (combining the immediate reward r and the next-state value V(s')). Subtracting V(s) from this value measures the deviation in the estimated value of state s. A positive deviation indicates that action a performs better than average in state s (as represented by V(s)), and its probability should be increased; a negative deviation indicates the opposite. This gradient update direction guides the parameter θ to update in the direction of increasing the probability of high-value actions by multiplying the logarithmic gradient of the policy with the TD error, thereby optimizing the policy to maximize the long-term cumulative reward.

[0112] S412. Parameterize the state value or state-action value of the critic network.

[0113] Critic network parameterized as V φ (s)(state value) or Q φ (s,a) (state-action value). Taking state value as an example, its loss function is the square of TD error:

[0114] L(φ)=E s,a,s′,r～π [(r+γV φ (s′)-V φ (s)) 2 ] (2)

[0115] In formula (2), φ is the parameter (weight or bias) of the critic network; v φ (s) represents the value estimate of the critic network with parameter φ for state s; r is the immediate reward obtained after executing action a in state s; γ represents the discount factor of the cumulative return (0≤γ≤1); s′ represents the next state transferred to after executing action a; r+γV φ (s′) represents the TD target, i.e., the estimate of the value of the next state; r+γV φ (s′)-V φ (s) represents TD error; E s,a,s′,r～π [·] means that after sampling the state s and action a together, the expected value of the square of the TD error is calculated, that is, the average result in the long-term average sense. This loss function is the expectation of the square of the TD error, and the goal is to make the state value estimate V φ (s) Approaching TD target r+γV φ (s′).

[0116] The gradient update formula is:

[0117]

[0118] In formula (3), β is the learning rate of the critic network; is the state value function V φ (s) The gradient of the parameter φ, indicating the direction of parameter update; r+γV φ (s′)-V φ (s) is the TD error, which serves as the weight of the gradient and determines the update amplitude and direction. The parameters φ of the Critic network are along the gradient The direction of the update is determined by the TD error and the learning rate β.

[0119] S42. In order to improve the learning efficiency and performance of intelligent agents, the differential evolution algorithm is introduced to optimize the Actor-Critic hypernetwork parameters and obtain the optimal parameter combination.

[0120] The differential evolution algorithm first initializes a population, with each individual representing a set of hyperparameter combinations. Evolution then iterates through mutation, crossover, and selection. Each individual's fitness is evaluated, with the cumulative reward of the agent in a simulated combat environment serving as the fitness value. After multiple iterations, the optimal hyperparameter combination is found and applied to the agent's learning process, updating the learning strategy and network parameters.

[0121] S421. When using the differential evolution algorithm to optimize the Actor-Critic hypernetwork parameters, the weights and biases of the hypernetwork are encoded into a set of vectors. Each complete vector represents a hypernetwork parameter model and also serves as an individual in the evolutionary algorithm population for iteration.

[0122] The construction of an individual is expressed as:

[0123] x i ={w1_weight,w1_bias,w2_weight,w2_bias,...,b 22 _weight,b 22 _bias} (5)

[0125] In formula (5), w j _weight, w j _bias represents the super network weight and bias value for generating the j-th layer weight of the Actor-Critic network; b j _weight, b j _bias represents the hypernetwork weight and bias value for generating the j-th layer bias of the Actor-Critic network, j∈[1,2]; since the bias W2 of the Actor-Critic network is calculated by a two-layer linear network, the hypernetwork parameter hyper_b2 is divided into two layers, b 21 with b 22 They represent two linear networks that generate the bias of the second layer of the Actor-Critic network.

[0126] S422. Use the mutation strategy DE / rand / 1 to generate mutated network parameter individuals to obtain a mutated population.

[0127] The individual network parameters are expressed as:

[0128] v i (t+1)=x r1 (t)+F(x r2 (t)-x r3 (t)) (6)

[0129] In formula (6), F is the scaling factor, v i (t+1) represents the i-th mutant individual in the t+1 generation population, x r1 (t), x r2 (t), x r3 (t) represents three random individuals in the t-th generation population, where r1≠r2≠r3. Through the above steps, a mutated population can be obtained.

[0130] S423. Use binomial crossover on the population to get a new population, expressed as:

[0131]

[0132] In formula (4), x i Represents the i-th individual (vector) in the original population, with a dimension of D, i.e. x i =[x i,1 ,x i,2 ,...,x i,D ];v i represents the i-th candidate individual (mutation vector) generated by the mutation operation; u i represents the new individual (test vector) generated after crossover; j represents the dimension index (1≤j≤D), indicating the jth component in the individual vector; rand j represents a random number uniformly distributed in the interval [0,1], and each dimension j is generated independently; CR represents the crossover rate, which ranges from [0,1] and controls the strength of the crossover; j rand Represents a randomly selected dimension index.

[0133] S424. Calculate the fitness value of the population after the crossover step, and compare it with the fitness value obtained from the initial population, so as to select excellent individuals as the initial population of the next generation.

[0134] In evolutionary algorithms, the fitness of individuals is particularly important. The algorithm distinguishes good individuals from bad ones based on their fitness, promoting evolution through a survival-of-the-fittest process. The calculation of the fitness function directly affects the genetic algorithm's convergence rate and the search for optimal solutions. Therefore, the construction of the fitness function determines the evolutionary direction of the population. Setting a reasonable fitness function is key to optimizing reinforcement learning algorithms.

[0135] Since the calculation time of the fitness function accounts for a large proportion of the algorithm, its design should be simple to minimize the time complexity of the calculation. The environment used in this study is a self-built assumption based on the Mozi platform, so the fitness function is set to the average reward of the multi-agent system based on the MADDPG algorithm in n training steps, expressed as:

[0136]

[0137] In formula (7), fitness represents the fitness value, that is, the average cumulative reward of all agents; i represents the index of the agent, in the interval [1, n], n is the number of agents, and reward represents the reward of the current agent. The composition of reward is expressed as:

[0138] reward=detect_reward+destroy_reward-be_killed_reward-border_reward (8)

[0140] In formula (8), detect_reward represents the reward for detecting the enemy, destroy_reward represents the reward for destroying the enemy, be_killed_reward represents the reward for the Red agent being killed, and border_reward represents the reward for the Red agent touching the border of the combat area.

[0141] The present invention adopts the differential evolution algorithm through a multi-strategy co-evolution mechanism to adaptively adjust the search parameters, effectively avoid local optimal solutions, and greatly improve the strategy search efficiency and global optimization capabilities.

[0142] The optimal parameter combination obtained in the above steps is loaded into the MADDPG model, and the performance of the agent in the deduction environment is evaluated through the reward function to obtain the evaluation results.

[0143] Finally, the agent's strategy is optimized based on the evaluation results, the initial strategy is optimized based on the evaluation results, the agent is again allowed to interact with the environment and collect new data, the hypernetwork parameters are adjusted based on the new data, and steps S4 to S6 are repeated to iteratively optimize the hypernetwork parameters until the final training model is obtained.

[0144] Through continuous iteration and optimization, the combat capability and adaptability of the agent can be gradually improved. During the iteration process, the complexity and difficulty of the task can be gradually increased to further improve the generalization ability and robustness of the agent.

[0145] The pseudo code of the DE-MADDPG algorithm of the present invention is shown in Table 1.

[0146] Table 1

[0147]

[0148]

[0149] To verify the tactical coordination capabilities of reinforcement learning algorithms in a strong electromagnetic interference environment, a 3v3 fleet confrontation simulation scenario was constructed to simulate the typical "small swarm, strong interference" encounters of modern naval warfare. The Red and Blue sides each deployed three Type 037-IG missile frigates with similar parameters. The combat area was set within a 150x150 nautical mile rectangular area. The Blue side implemented full-band jamming through its shipboard electronic systems, reducing the radar detection range of the Red side's ships. Focusing on a non-fixed command ship architecture, the experiment required the intelligent agent to verify core capabilities through autonomous learning, including dynamic formation reconfiguration, accurate recognition of real targets in a jammed environment, and tactical coordination effectiveness.

[0150] This experiment uses the Mozi joint combat simulation platform as the simulation environment for ship formation operations, integrating the following core modules:

[0151] (1) Multi-agent decision-making system: The Red Army uses the DE-MADDPG algorithm to control three Type 037-IG missile frigates, supporting dynamic optimization of hypernetwork parameters;

[0152] (2) Electromagnetic interference model: The blue side activates full-band interference to cover the combat sea area;

[0153] (3) Ship dynamics model: including maneuvering constraints (maximum speed 30 knots), fuel consumption (continuous combat time 3 hours) and weapon system (anti-ship missile range 40 nautical miles).

[0154] The core mechanism of the Red Army's decision-making system is based on a multi-agent deep deterministic policy gradient algorithm. This system achieves collaborative decision-making through centralized training and distributed execution, relying on the OODA loop (Observe-Adjust-Decide-Act) to continuously optimize tactics. Throughout the operational process, the OODA loop is a continuously iterative process, with each link closely interconnected and influencing each other.

[0155] The evaluation index is consistent with formula (7), and still uses the average return obtained by the red agent in n training steps and the winning rate counted every evaluate_per_epoch training step. The winning rate win_rate calculation method is shown in formula (9).

[0156]

[0157] In formula (9), win_flag represents the number of wins within the evaluate_per_epoch training step. The specific parameters of the DE-MADDPG algorithm and other algorithms are shown in Table 2.

[0158] Table 2

[0159]

[0160] The comparison of the win rate and return of DE-MADDPG, MADDPG, COMA, and MAPPO algorithms in the 3v3 ship formation confrontation simulation scenario is shown in the attached figure. Figure 2 and attached Figure 3 .

[0161] The win rate curves show significant differences in performance among the algorithms during the initial stages of training (episodes 0-5000): DE-MADDPG and COMA both maintain high win rates of 0.55-0.6, MAPPO stabilizes around 0.5, and MADDPG fluctuates only between 0.2-0.3. This demonstrates that DE-MADDPG, through dynamic hypernetwork parameter optimization, can rapidly adapt to strong electromagnetic interference environments and achieve effective tactical coordination. While achieving similar exploration efficiency to COMA's centralized critic mechanism, it significantly outperforms MADDPG's independent policy learning and MAPPO's proximal policy optimization.

[0162] Entering the mid-term training (episodes 5000-17500), MADDPG achieved a significant increase in win rate (around 0.6) through experience accumulation, while DE-MADDPG, COMA, and MAPPO saw relatively modest improvements. During this phase, MADDPG's strategy convergence accelerated, but its performance ceiling was limited by its fixed network architecture and could not surpass DE-MADDPG's dynamic parameter adjustment capabilities.

[0163] In the later stages of training (after 17,500 episodes), DE-MADDPG further improved to around 0.7 and remained stable, significantly widening the gap with other algorithms. This advantage stems from the differential evolution mechanism's efficient optimization of the hypernetwork parameters in the early stages of training, which builds a more adaptable policy expression space for the agent. The stable performance gain in the later stages stems from the hypernetwork structure's ability to continuously respond to environmental changes. Specifically, DE-MADDPG initializes the policy network with parameters found in the early stages of differential evolution, enabling it to generalize more effectively in the face of interference. Traditional algorithms (such as MADDPG, COMA, and MAPPO), due to their fixed architectures or single optimizers, struggle to adapt their policy representations to dynamic changes in interference patterns, leading to stagnant performance after 17,500 episodes. This result demonstrates the synergistic effect of differential evolution pre-optimization and the dynamic adaptation of the hypernetwork: initializing parameter directions through global search in the early stages of training, and adapting to environmental uncertainty through the network's inherent flexibility in the later stages.

[0164] The reward curves show similar trends: Initially, DE-MADDPG's average reward is significantly higher than MADDPG, COMA, and MAPPO. In the middle, MADDPG's reward rises to around 200, but DE-MADDPG's exceeds 200. Later, DE-MADDPG stabilizes at 280, outperforming MADDPG, COMA, and MAPPO. This result validates DE-MADDPG's dual advantages in reward acquisition efficiency and long-term strategy stability.

[0165] The improved MADDPG hypernetwork parameter optimization method based on differential evolution proposed in this paper dynamically optimizes the MADDPG hypernetwork parameters through the DE algorithm, realizing the bidirectional evolution of hyperparameters and policy networks.

[0166] The ship agent of the present invention interacts with the battlefield in a simulation environment. First, the algorithm hypernetwork parameters are encoded. Specifically, the method initializes the MADDPG neural network model and its hypernetwork parameters. Subsequently, the agent executes actions in the simulation environment and updates its experience replay buffer based on environmental feedback (i.e., state transitions and rewards). Next, a differential evolution algorithm is used to iteratively optimize the MADDPG hypernetwork parameters. Through operations such as mutation, crossover, and selection, the hyperparameter combination that best suits the agent's response to the environment is continuously selected.

[0167] During the optimization process, a fitness function is used to evaluate the agent's performance under different parameter combinations. The fitness function is calculated based on the cumulative rewards the agent receives in the deduction environment. Higher rewards indicate better parameter combinations.

[0168] After multiple iterations reach the preset number of iterations, the optimal hypernetwork parameter combination is output. These parameters are then applied to the MADDPG neural network, enabling the agent to perform optimally in the simulation environment.

[0169] In addition, the term "agent" used in the present invention refers to an entity with the basic characteristics of autonomy, sociality, responsiveness and pre-action. It can be corresponding software or an entity (such as a person, drone, robot, etc.), which is embedded in the environment, perceives the environment through sensors, acts on the environment autonomously through effectors and meets the design requirements. The term "reward" used in the present invention refers to the reward calculated by the reward function after the agent performs a certain execution action, wherein the reward function is a function set when constructing a multi-agent decision-making model for characterizing the reward. The term "action strategy" used in the present invention refers to a strategy for determining the execution action required to change from the current state to the next state.

[0170] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A MADDPG hypernetwork parameter optimization method based on differential evolution for ship formations, characterized by: Including steps: S1. Build a collaborative combat environment for a fleet of ships, set the ships as intelligent agents, and set task information for the intelligent agents; S2. Load the task information and generate an initial strategy; S3. Based on the initial strategy, the agent interacts with the environment and collects data; S4. Training the hypernetwork parameters of the MADDPG model according to the data, and optimizing the hypernetwork parameters using a differential evolution algorithm to obtain an optimal parameter combination; S5: Loading the optimal parameter combination into the MADDPG model, and evaluating the performance of the agent to obtain an evaluation result; S6: Optimize the initial strategy based on the evaluation results, let the agent interact with the environment again and collect new data, adjust the hypernetwork parameters based on the new data, and repeat steps S4 to S6 to iteratively optimize the hypernetwork parameters until the final training model is obtained.

2. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 1 is characterized in that: Step S2 includes: S21. Abstracting the process of interaction between the agent and the environment into a partially observable Markov decision process, wherein the partially observable Markov decision process is represented by a five-tuple: <state space, action space, transition probability function, reward function, discount factor of cumulative reward>; S22. Setting relevant parameters, including MADDPG hypernetwork parameters, training parameters, and evaluation parameters for measuring the agent; S23, loading the task information; S24. Create a MADDPG algorithm based on the relevant parameters, initialize the network architecture, load the state space and the action space, and generate an initial strategy based on the current state of the agent.

3. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 2 is characterized in that: Step S21 includes: The state space represents all states of the agent, including battlefield environment information and ship state information; The action space represents the set of all actions that the agent can perform, including tactical maneuvers, combat-related actions, and collaborative actions; The transition probability function describes the probability that the agent will transition to the next state after performing an action in the current state; The reward function calculates and returns the reward obtained by the agent based on the change in the state of the environment after the agent performs the action; The discount factor of the cumulative reward is used to calculate the expected cumulative benefit.

4. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 3 is characterized in that: Step S3 includes: In each step of the interaction between the agent and the environment, the agent perceives the current state of the environment s through sensors and selects an optimal action a according to the initial strategy π(a|s); After the action is executed, the agent observes the change in the environment state s' and the reward r obtained, and collects relevant data; The data includes state s, action a and reward r.

5. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 4 is characterized in that: Step S4 includes: S41, the MADDPG model is based on the Actor-Critic network structure, and the intelligent agent uses the Actor-Critic network structure to reconstruct the dynamic battlefield situation; S42. Introduce the differential evolution algorithm to optimize the Actor-Critic hypernetwork parameters and obtain the optimal parameter combination.

6. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 5 is characterized in that: Step S41 includes: S411. Parameterized Actor Network Strategy Function π θ (a|s) represents the probability distribution or probability density of the action to be performed in the current state. The optimization goal is to maximize the expected cumulative reward, which is achieved through the policy gradient theorem: In formula (1), represents the gradient of the parameter θ, J(θ) represents the objective function, and the strategy π θ The expected cumulative reward of represents the gradient of the objective function J(θ) with respect to the parameter θ; s~ρ π Represents the state distribution ρ of state s under strategy π π Sampling under a~π θ represents the strategy π for action a according to parameter θ θ (a|s) sampling, represents the expectation after joint sampling of s and a; Q π (s,a) is the value function of the state-action pair (s,a) under the policy π, which evaluates how good or bad the action a is in the state s; π θ (a|s) represents the probability of taking action a in state s with a policy parameter θ; logπ θ (a|s) represents the log probability of the strategy, Represents the gradient of the logarithmic probability with respect to the parameter θ, which measures the direction and magnitude of the impact of parameter changes on the action probability; The Q value is estimated using the time difference method, and the gradient update direction is Denotes the strategy π θ The logarithmic gradient of (s,a) with respect to the parameter θ; r represents the immediate reward, which is the immediate reward obtained by the agent from the environment after performing action a in state s; γ represents the discount factor of the cumulative reward (0≤γ≤1), which is used to balance the importance of current rewards and future rewards. The closer γ is to 0, the more the agent pays attention to the current reward, and the closer γ is to 1, the more attention is paid to long-term rewards; V(s′) represents the value of the transferred state s' estimated by the Critic network, which represents the expected future cumulative reward of following the current strategy starting from state s'; V(s) represents the value of the current state s estimated by the Critic network; r+γV(s′)-V(s) represents the error of the temporal difference method, which is an approximation of the advantage function; r+γV(s′) is the target value estimated by the temporal difference method. After subtracting V(s), it measures the deviation of the value estimate of state s. A positive deviation indicates that action a performs better than the average in state s and the probability should be increased, while a negative deviation is the opposite; The gradient update direction guides the parameter θ to update in the direction of increasing the probability of high-value actions by multiplying the logarithmic gradient of the policy with the error of the temporal difference method, optimizing the policy to maximize the long-term cumulative return; S412. Parameterize the state value V of the Critic network φ (s) or state-action value Q φ (s,a); The loss function of the state value is the square of the error of the temporal difference method, expressed as: L(φ)=E s,a,s′,r～π [(r+γV φ (s′)-V φ (s)) 2 ] (2) In formula (2), φ is the parameter, weight or bias of the critic network; V φ (s) represents the value estimate of the critic network with parameter φ for state s; r is the immediate reward obtained after executing action a in state s; γ represents the discount factor of the cumulative return (0≤γ≤1); s′ represents the next state transferred to after executing action a; r+γV φ (s′) represents the target of the temporal difference method, the estimate of the value of the next state; r+γV φ (s′)-V φ (s) represents the error of the time difference method; E s,a,s′,r～π [·] represents the expected value of the square of the error of the temporal difference method after jointly sampling the state s and the action a, and the average result in the long-term average sense; The loss function is the expectation of the square of the error of the time difference method, and the goal is to make the state value estimate V φ (s) Approaching the time difference method target r+γV φ (s′); The gradient update formula is: In formula (3), β is the learning rate of the critic network; is the state value function V φ (s) The gradient of the parameter φ, indicating the direction of parameter update; r+γV φ (s′)-V φ (s) is the error of the temporal difference method, which serves as the weight of the gradient and determines the update amplitude and direction; The parameters φ of the critic network follow the gradient The direction of the update is determined by the time difference method error and the learning rate β.

7. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 5 is characterized in that: Step S42 includes: S421, encoding the weights and biases of the hypernetwork into a set of vectors, each complete vector representing a hypernetwork parameter model and also serving as an individual in the evolutionary algorithm population for iteration; S422, using the mutation strategy DE / rand / 1 to generate mutated network parameter individuals to obtain a mutated population; S423. A binomial crossover method is used on the population to obtain a new population, which is expressed as: In formula (4), x i Represents the i-th individual in the original population, with dimension D, x i =[x i,1 ,x i,2 ,...,x i,D ];v i represents the i-th candidate individual generated by the mutation operation; u i represents the new individual generated after the crossover; j represents the dimension index, 1≤j≤D, representing the jth component in the individual vector; rand j represents a random number uniformly distributed in the interval [0,1], and each dimension j is generated independently; CR represents the crossover rate, which ranges from [0,1] and controls the strength of the crossover; j rand Represents a randomly selected dimension index; S424. Calculate the fitness value of the population after the crossover step, and compare it with the fitness value obtained from the initial population, so as to select excellent individuals as the initial population of the next generation.

8. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 7 is characterized in that: In step S421, the individual is constructed as follows: x i ={w1_weight,w1_bias,w2_weight,w2_bias,...,b 22 _weight,b 22 _bias} (5) In formula (5), x i represents the individuals in the population, w j _weight, w j _bias represents the super network weight and bias value for generating the j-th layer weight of the Actor-Critic network; b j _weight, b j _bias represents the hypernetwork weight and bias value for generating the j-th layer bias of the Actor-Critic network, j∈[1,2]; since the bias W2 of the Actor-Critic network is calculated by a two-layer linear network, the hypernetwork parameter hyper_b2 is divided into two layers, b 21 with b 22 They represent two linear networks that generate the bias of the second layer of the Actor-Critic network.

9. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 7 is characterized in that: In step S422, the network parameter individual is represented as: v i (t+1)=x r1 (t)+F(x r2 (t)-x r3 (t)) (6) In formula (6), F is the scaling factor, v i (t+1) represents the i-th mutant individual in the t+1 generation population, x r1 (t), x r2 (t), x r3 (t) represents three random individuals in the population of generation t, where r1≠r2≠r3.

10. The MADDPG hypernetwork parameter optimization method based on differential evolution for ship formation according to claim 7 is characterized in that: In step S424, the fitness function is set to the average reward of the multi-agent system based on the MADDPG algorithm in n training steps, which is expressed as: In formula (7), fitness represents the fitness value, that is, the average cumulative reward of all agents; i represents the index of the agent, in the interval [1, n], n is the number of agents, and reward represents the reward of the current agent; The reward is composed of: reward=detect_reward+destroy_reward-be_killed_reward-border_reward (8) In formula (8), detect_reward represents the reward for detecting the enemy, destroy_reward represents the reward for destroying the enemy, be_killed_reward represents the reward for the Red agent being killed, and border_reward represents the reward for the Red agent touching the border of the combat area.

Images