An adaptive particle swarm optimization method and system based on multi-agent reinforcement learning

Abstract

The application discloses a particle swarm optimization method based on multi-agent deep reinforcement learning. In view of defects such as easy falling into local optimum and high-dimensional degeneration of a traditional algorithm, the application embeds multi-agent reinforcement learning into an optimization cycle. Each particle is regarded as an independent agent, and the state is perceived through a shared policy network; when the particle is stagnant, the agent outputs actions to adaptively select random jumping, Levy flight, speed reversal or dimension shift disturbance to correct the speed. Meanwhile, a hybrid reward function is designed to guide the network to update in real time, and offline pre-training is not needed, so that learning is realized while optimization is performed. The application effectively overcomes the traditional defects, and provides an efficient and adaptive new way for solving complex high-dimensional, multi-modal and actual engineering design optimization problems.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and optimization algorithms, specifically relating to an adaptive particle swarm optimization method based on multi-agent reinforcement learning, and an optimization system for implementing this method. This method seamlessly embeds a multi-agent deep reinforcement learning decision layer into the main loop of the classic particle swarm optimization algorithm. By assigning a shared policy network to each particle, the algorithm can autonomously learn and adjust its search strategy based on accumulated search experience, thereby significantly improving the algorithm's efficiency and solution quality in high-dimensional, multimodal complex optimization problems. Background Technology

[0002] With the rapid development of science and technology, more and more real-world optimization problems are exhibiting characteristics such as high dimensionality, inherent complexity, and multimodality, such as in engineering design, parameter identification, and energy scheduling. In the context of single-objective optimization, the mathematical model of these problems can usually be expressed as: minimizing an objective function defined on a high-dimensional Euclidean space under the condition of satisfying a series of equality and inequality constraints.

[0003] Particle Swarm Optimization (PSO), a classic metaheuristic algorithm for swarm intelligence, is widely used in various engineering fields due to its simple concept, ease of implementation, and relatively few parameters. Its basic idea is to simulate the swarm movement of birds or schools of fish, iteratively updating the position in the solution space by tracking the particle's own historical optimal position and the swarm's global optimal position to approximate the global optimum. However, the standard PSO algorithm has the following significant drawbacks: First, it is essentially memoryless, treating each optimization run as an independent event, failing to retain and utilize trajectory data from previous runs, leading to a significant waste of valuable search experience; second, in complex multimodal high-dimensional search spaces, the algorithm is prone to getting trapped in local optima, experiencing premature convergence, and losing its global exploration capability; third, its perturbation strategies, such as random restarts and Lévy flight, are usually fixed and heuristic, lacking adaptive feedback capabilities to the current search state, making it difficult to achieve a dynamic balance between exploration and exploitation.

[0004] To improve the performance of metaheuristic algorithms, researchers have attempted to combine reinforcement learning (RL) with evolutionary computation (EC). RL, by interacting with the environment and learning decision policies based on reward signals, can provide evolutionary algorithms with an adaptive balance mechanism between exploration and development. Existing work shows that using RL to guide the evolutionary process can accelerate convergence and improve the quality of solutions. However, most existing methods share a common limitation: they require a separate "offline pre-training" phase. That is, data needs to be collected and the RL policy model trained before optimization can begin. This process is not only time-consuming but also limits the immediate applicability and generality of the algorithm. Users must prepare training data for different problems, resulting in a high barrier to practical application. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and provide an adaptive particle swarm optimization method (MARLPSO) based on multi-agent reinforcement learning, and its corresponding system. The core innovation of this method lies in directly and in real-time integrating a multi-agent deep reinforcement learning framework, particularly multi-agent proximal policy optimization (MAPPO), into the iterative loop of PSO without any offline pre-training stage. By endowing each particle in the swarm with a shared "AI brain" (i.e., a policy network), it can perceive its own and the swarm's search state and learn when and how to apply intelligent perturbations to standard PSO updates. This invention transforms traditional PSO from a memoryless random search process into an accumulated, experience-driven intelligent optimization process, converting redundant computations from multiple independent runs into reusable search experience, significantly improving the algorithm's sample efficiency and solution performance without increasing pre-training costs.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] An adaptive particle swarm optimization method based on multi-agent reinforcement learning includes the following steps.

[0008] (1) Initialization and State Awareness: The position and velocity of each particle in the particle swarm are randomly initialized. At the beginning of each iteration, a normalized state observation vector is constructed for each particle. This vector comprehensively reflects the spatial relationship of the particle with respect to the global optimum and the individual optimum, the current velocity, the fitness ranking in the swarm, and its own stagnation level, providing a comprehensive and consistent snapshot of the search environment for the RL policy network.

[0009] (2) Learning strategy synthesis: Each particle inputs the observed state vector into a shared multi-agent reinforcement learning policy network. The network outputs a two-dimensional continuous action vector, which is linearly mapped to the perturbation trigger threshold and perturbation mode parameters. This step transforms the heuristic rules that originally relied on manual tuning into a state-aware adaptive control logic based on learning.

[0010] (3) Hybrid Velocity-Position Update: First, the baseline velocity is calculated according to the standard PSO formula. Second, it is determined whether the current particle's stagnation level is greater than the trigger threshold output by the policy network. If the condition is met, a perturbation policy is dynamically selected and parameterized from the four preset perturbation mode libraries according to the parameters, and superimposed on the baseline velocity to obtain the final velocity. The four perturbation policies are: random jump, used to make large-amplitude uniform displacements to jump out of the local area; Lévy flight, using a heavy-tailed Cauchy distribution for long-step jumps; velocity reversal, repositioning the search direction by reversing the velocity direction's momentum; and dimensional shift, breaking the degenerate motion mode by cyclically shifting the dimensional components in the velocity vector. Subsequently, the particle position is updated using the final velocity, and the position is clipped within the effective solution space.

[0011] (4) State transition and memory update: Fitness evaluation is performed on the new position. The stagnation counter is updated based on the relative fitness improvement. The individual historical best and the global best of the group are updated according to the greedy policy. During the training phase, a hybrid reward signal is calculated, which integrates information on the global best improvement and the individual historical best improvement, to guide the updating and optimization of the policy network.

[0012] Furthermore, in this invention, the reward function is designed as a weighted hybrid form: the global reward component accounts for 80%, providing a strong positive reward when the global optimal fitness improves; the individual reward component accounts for 20%, providing a smaller positive reward when the individual particle's historical optimal fitness improves, and a slight penalty when it stagnates. This design prioritizes collective progress while encouraging individual local exploration.

[0013] Furthermore, in this invention, the policy network is trained using the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, where multiple particle agents collaboratively learn in a shared environment to jointly optimize a common objective. Furthermore, this invention also provides an optimization system for implementing the above method, the system comprising: The state awareness module is used to construct a normalized state observation vector for each particle.

[0014] The policy decision module includes the shared multi-agent reinforcement learning policy network, which is used to output perturbation control parameters based on the state vector.

[0015] The hybrid update module is used to calculate the baseline velocity based on the standard PSO and select and apply a perturbation strategy based on the output of the decision module to finally update the particle position.

[0016] The reward and learning module is used to calculate the mixed reward signal and drive the online learning and parameter updates of the policy network.

[0017] The beneficial effects of this invention are: Breaking through the bottleneck of "offline pre-training": Deep reinforcement learning is tightly coupled with PSO to achieve online co-evolution without any pre-training stage, which greatly improves the plug-and-play nature and versatility of the algorithm.

[0018] It has the ability to accumulate learning and reuse experience: Through a shared policy network, the algorithm can utilize historical search experience across runs to transform redundant computations into actionable intelligence, significantly improving sample efficiency.

[0019] Provides state-aware adaptive perturbation: The triggering timing and specific modality of the perturbation strategy are dynamically decided by the deep neural network based on real-time, multi-dimensional state information, realizing highly intelligent and adaptive search capabilities.

[0020] Superior solution performance and scalability: Experiments show that on the CEC2017 standard test set, this method ranks first on 69.0% of the test functions in the 100-dimensional high-dimensional setting, significantly outperforming traditional and many advanced metaheuristic algorithms, and maintaining superior performance and stability in the expansion from low-dimensional to high-dimensional.

[0021] Applicable to complex constrained engineering problems: In practical engineering constrained optimization problems such as cantilever beam design, pressure vessel design, and reducer design, this method can find high-quality feasible solutions, verifying its effectiveness in solving complex practical engineering problems. Attached Figure Description

[0022] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0023] Figure 1 is a schematic diagram of the learning perturbation mechanism in an embodiment of the present invention, showing how a particle agent selects one of four perturbation strategies for velocity correction based on the current stagnant state and the output of the policy network.

[0024] Figure 2 is a schematic diagram of the MARL and PSO interaction mechanism in an embodiment of the present invention, showing the data flow and processing logic from input (fitness value, position, velocity, ranking, stagnation count) to output (final particle position) in an iterative loop.

[0025] Figure 3 is an overall flowchart of the MARLPSO algorithm in an embodiment of the present invention, covering the main steps such as initialization, state awareness, policy synthesis, hybrid update, state transition and memory update.

[0026] Figure 4 is a structural schematic diagram of the present invention when applied to practical engineering optimization problems, such as cantilever beam design problems.

[0027] Figure 5 is a schematic diagram of the convergence curve of the present invention in the cantilever beam design problem.

[0028] Figure 6 is a schematic diagram of the learning (episode) reward curve of the present invention on the cantilever beam design problem.

[0029] It should be understood that the accompanying drawings in this invention are merely illustrative and are not necessarily drawn to scale, nor do they limit the specific embodiments of this invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] In the description of this invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0032] Example 1: Applied to engineering optimization problems.

[0033] This embodiment illustrates the application of the MARLPSO method to practical engineering constrained optimization problems, such as cantilever beam design problems (e.g., Figure 4 (As shown).

[0034] The objective of this problem is to minimize the total weight of a cantilever beam, with design variables being the heights x1, x2, x3, x4, and x5 of five square hollow sections. This problem involves a complex nonlinear constraint. To handle this constraint, this embodiment employs a penalty function method to transform the original constrained optimization problem into an unconstrained problem for solution.

[0035] The MARLPSO algorithm from Example 1 was used for the solution, with the particle swarm size N = 30 and the maximum number of iterations tmax = 500. The algorithm was run 30 independent experiments.

[0036] Experimental results show that MARLPSO can find the optimal or near-optimal design scheme that satisfies all constraints, and its convergence curve (e.g.) Figure 5 As shown in the figure, the algorithm can quickly and stably descend to near the optimal solution. Meanwhile, the learning reward curve of the agent (as shown in the figure) demonstrates that the algorithm can quickly and stably descend to near the optimal solution. Figure 6 As shown in the figure, the average return increases with the number of iterations (or episodes), verifying that the policy network does indeed learn effectively during the optimization process.

[0037] The optimization results of this embodiment are compared with those of various existing advanced algorithms (such as MSAO, AOA, SMA, GWO, JADE, etc.). MARLPSO has a significant advantage in the quality of the solutions found, which proves the effectiveness and robustness of the method of this invention in practical complex engineering problems.

[0038] In summary, the method proposed in this invention achieves intelligent optimization that is pre-trained-free and adaptively applied online by deeply integrating multi-agent deep reinforcement learning with particle swarm optimization. This method not only provides a powerful new tool for solving complex high-dimensional multimodal optimization problems, but also opens up a new technical path for combining evolutionary computation and reinforcement learning.

Claims

1. An adaptive particle swarm optimization method based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Initialize a particle swarm, where each particle has position, velocity, historical best position, and historical best fitness value, and determine the global best position and global best fitness value of the swarm. In each iteration, construct a normalized state vector for each particle. The state vector includes at least: the distance component of the particle's current position relative to the global best position, the distance component of the particle's current position relative to its historical best position, a normalized velocity component, the particle's fitness ranking component in the swarm, and a normalized stagnation component, where the stagnation component is the smaller of the stagnation counter divided by 10 and 1. Input the state vector into a shared multi-agent reinforcement learning policy network, which outputs an action vector, and maps the action vector to a perturbation trigger threshold and a... The perturbation mode parameters are defined as follows: The baseline velocity of the particle is calculated using the standard particle swarm optimization algorithm; the normalized stagnation index of the particle is calculated, and it is determined whether the stagnation index is greater than the perturbation trigger threshold; if so, a perturbation strategy is selected from a preset perturbation strategy library based on the value of the perturbation mode parameters, and the corresponding perturbation vector is calculated; the perturbation vector is superimposed on the baseline velocity to obtain the final update velocity of the particle; the particle position is updated based on the final update velocity; the fitness of the updated particle position is evaluated, the stagnation counter, the individual historical best position, and the global best position of the population are updated, and a reward signal is calculated based on the change in fitness; the reward signal is used to perform online training and updating of the shared multi-agent reinforcement learning policy network using a multi-agent proximal policy optimization algorithm.

2. The method according to claim 1, characterized in that, The perturbation strategy library includes the following four strategies: A random jump strategy is used to activate when the perturbation mode parameter is between 0 and 0.25; The Levy flight strategy is activated when the disturbance mode parameters are between 0.25 and 0.

5. A velocity reversal strategy is activated when the perturbation mode parameter is between 0.5 and 0.

75. A dimension shifting strategy is used to activate when the perturbation mode parameter is 0.75 or higher.

3. The method according to claim 2, characterized in that, The perturbation vector corresponding to the random jump strategy is calculated as follows: Multiply the difference between the upper and lower bounds of the search space by the jump strength parameter, and then multiply by a uniformly distributed random vector in the interval -1 to 1. The jump intensity parameter is calculated based on the disturbance mode parameters.

4. The method according to claim 2, characterized in that, The perturbation vector corresponding to the Levi flight strategy is calculated as follows: multiply the difference between the upper and lower bounds of the search space by 1, add the value of the stagnation index, and then multiply by a random vector that follows a Cauchy distribution. This random vector needs to be truncated to the interval between -3 and 3. The scale parameter of the Cauchy distribution is calculated based on the perturbation mode parameter.

5. The method according to claim 2, characterized in that, The perturbation vector corresponding to the speed reversal strategy is calculated as follows: multiply the base speed by -1 and 1, add the value of the stagnation index and the reversal factor parameter; wherein, the reversal factor parameter is calculated based on the perturbation mode parameter.

6. The method according to claim 2, characterized in that, The perturbation vector corresponding to the dimension shifting strategy is calculated by multiplying the baseline velocity after cyclic shifting by the perturbation mode parameter; wherein, the number of cyclic shifting steps is calculated based on the perturbation mode parameter and the problem dimension, that is, taking the maximum value between the floor value of the product of the parameter value and the dimension and 1.

7. The method according to claim 1, characterized in that, The reward signal is a weighted mixed reward, calculated as follows: the reward signal equals 0.2 times the individual reward component plus 0.8 times the global reward component; whereby the global reward component is used to measure the improvement of the global optimal fitness of the population; and the individual reward component is used to measure the improvement of the individual particle's historical optimal fitness.

8. The method according to claim 7, characterized in that, The calculation logic for the global reward component is as follows: if the current global optimal fitness is better than the previous global optimal fitness, then the fitness difference is normalized using the hyperbolic tangent function and used as the reward value; otherwise, the reward value is 0.

9. The method according to claim 7, characterized in that, The calculation logic for the individual reward component is as follows: if the current particle's fitness is better than the particle's historical best fitness in the previous round, then the fitness difference is normalized using the hyperbolic tangent function and used as the reward value; otherwise, the reward value is -0.01.

Images