Intelligent agent optimization method and system based on reinforcement learning strategy

Abstract

The invention discloses an intelligent agent optimization method and system based on a reinforcement learning strategy, which can effectively improve the learning stability through limiting the step length of strategy updating by a near-end strategy, reduce the problem of unstable training or performance reduction caused by excessive strategy updating, and improve the learning efficiency. The deep neural network is combined with the reinforcement learning algorithm to improve the generalization ability and the complex environment coping ability of the intelligent agent, the self strategy is optimized by adopting the MADDPG algorithm to consider the behaviors of other intelligent agents, and the cooperation mechanism enables the intelligent agent to realize the globally optimal or better cooperation effect in the shared environment. Dependence on real world interaction can be reduced through the simulation environment, meanwhile, the semi-supervised learning method utilizes a small amount of annotated data and a large amount of unannotated simulation data, so that the intelligent agent can be continuously improved and optimized by itself in the learning process, the environmental adaptability of the intelligent agent is improved, and the learning efficiency of the intelligent agent is improved. And the dependence on a large amount of annotated data is reduced, and the data collection and processing cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agent optimization technology, and in particular to an intelligent agent optimization method and system based on a reinforcement learning strategy. Background Art

[0002] As an important branch of artificial intelligence, reinforcement learning (RL) focuses on learning how to optimize behavior strategies to maximize cumulative rewards through continuous trial and error through the interaction between the agent and the environment. Although reinforcement learning has shown great potential in many fields, such as game strategies, robot control, and autonomous driving, its traditional methods still face many challenges and limitations in practical applications.

[0003] First, learning efficiency is a prominent issue. Traditional reinforcement learning algorithms often require a large number of trial and error times to converge to an effective strategy. Especially in complex and high-dimensional environments, this process is particularly time-consuming and resource-intensive. In addition, for tasks in continuous action spaces or with long-term dependencies, the algorithm may fall into a local optimal solution and be unable to fully explore and find the global optimal strategy.

[0004] Secondly, the stability problem limits the widespread application of reinforcement learning. In practical applications, slight changes in the environment may lead to significant fluctuations in the behavior of the intelligent agent, thus affecting the reliability and robustness of the overall system. This requires the algorithm to not only have efficient learning capabilities, but also be able to maintain stable performance in various dynamic environments.

[0005] Furthermore, the lack of strategy generalization ability is also a major flaw of traditional reinforcement learning. When a trained model faces a new environment that is different from the training environment, its performance is often greatly reduced or even completely ineffective, which seriously hinders the effective application of reinforcement learning technology in actual complex and changeable scenarios.

[0006] Finally, data efficiency is also a limiting factor. Traditional reinforcement learning usually relies on a large amount of real-world interaction data to train models, which is impractical in many application scenarios. The scarcity and acquisition cost of data limit the widespread promotion and application of the algorithm. Summary of the invention

[0007] In view of this, the present invention proposes an intelligent agent optimization method and system based on reinforcement learning strategy, which can solve the defects of the prior art such as poor learning efficiency, poor stability, lack of strategy generalization ability and poor data efficiency.

[0008] The technical solution of the present invention is achieved in this way:

[0009] An agent optimization method based on reinforcement learning strategy, specifically including:

[0010] Reinforcement learning algorithms that use proximal strategies to optimize agents;

[0011] Combine the optimized reinforcement learning algorithm with the deep neural network as the input processing of the intelligent agent;

[0012] The MADDPG algorithm is used to optimize the collaborative strategy of intelligent agents;

[0013] The environmental adaptability of the intelligent agent is optimized by using simulated environment and semi-supervised learning methods.

[0014] As a further optional solution of the agent optimization method based on the reinforcement learning strategy, the reinforcement learning algorithm using the proximal strategy to optimize the agent specifically includes:

[0015] Setting an initial strategy parameter, wherein the initial strategy parameter is used to define the behavior strategy of the agent;

[0016] Run the current strategy in the environment and collect data through the interaction between the agent and the environment. The data includes the actions taken by the agent, the state changes of the environment, and the immediate rewards obtained.

[0017] Using the collected data, calculate the advantage function for each state-action pair;

[0018] Construct the objective function based on the advantage function;

[0019] Optimize the objective function based on the gradient ascent algorithm;

[0020] Update the agent’s strategy parameters based on the optimized objective function;

[0021] Put the updated strategy into the environment again for data collection, and repeat the above steps for iterative learning until the stopping condition is met.

[0022] As a further optional solution of the agent optimization method based on the reinforcement learning strategy, the optimized reinforcement learning algorithm combined with the deep neural network as the input processing of the agent specifically includes:

[0023] Preprocess the input data to adapt to the input requirements of the neural network;

[0024] Design the corresponding neural network architecture according to the task characteristics and data characteristics;

[0025] In the neural network architecture, embed the value function network of the reinforcement learning algorithm;

[0026] Define the loss function according to the goal of reinforcement learning;

[0027] Using the preprocessed data as input, a neural network embedded in a value function network is trained to obtain a deep neural network combined with an optimized reinforcement learning algorithm, and the deep neural network is processed as input to the intelligent agent.

[0028] As a further optional solution of the intelligent agent optimization method based on reinforcement learning strategy, the value function network of the reinforcement learning algorithm is embedded in the neural network architecture, specifically including:

[0029] Analyze the neural network architecture to determine the number and configuration of its input, output and hidden layers;

[0030] Determine the embedding point of the value function network based on the number and configuration of the input and output layers and hidden layers of the neural network architecture;

[0031] At the selected embedding point, a value function network layer is designed, and the value function network is embedded in the value function network layer.

[0032] As a further optional solution of the agent optimization method based on reinforcement learning strategy, the collaborative strategy of optimizing the agent using MADDPG algorithm specifically includes:

[0033] Clarify the roles and responsibilities of each agent in the environment;

[0034] Design communication mechanisms between agents;

[0035] The MADDPG algorithm is based on centralized training and decentralized execution, so that the strategy of each agent takes into account the strategies of other agents during training;

[0036] During the training process, the synergy between agents is regularly evaluated;

[0037] Based on the collaborative effect evaluation results, the agent's strategy is fine-tuned.

[0038] As a further optional solution of the intelligent agent optimization method based on the reinforcement learning strategy, the method of using a simulated environment and semi-supervised learning to optimize the environmental adaptability of the intelligent agent specifically includes:

[0039] Constructing a simulation environment, wherein the simulation environment is used to simulate the problems encountered by the intelligent agent in actual applications;

[0040] Run the agent in a simulation and record data as it performs tasks;

[0041] Use the collected simulation data and the labeled data to perform semi-supervised learning and obtain a semi-supervised learning model;

[0042] The semi-supervised learning model is deployed in the intelligent agent to optimize the environmental adaptability of the intelligent agent.

[0043] As a further optional solution of the agent optimization method based on the reinforcement learning strategy, the collected simulation data is combined with the labeled data to perform semi-supervised learning to obtain a semi-supervised learning model, which specifically includes:

[0044] Designing an initial semi-supervised learning model, wherein the initial semi-supervised learning model can use unlabeled simulated data and labeled data for learning;

[0045] Combine unlabeled simulated data and labeled data into a training set;

[0046] Use the training set to train the initial semi-supervised learning model to obtain the final semi-supervised learning model;

[0047] The method of using the training set to train the initial semi-supervised learning model to obtain the final semi-supervised learning model specifically includes:

[0048] Using the labeled data to train an initial semi-supervised learning model, and initializing parameters of the initial semi-supervised learning model;

[0049] Introduce unlabeled simulated data into the training process, use the initial semi-supervised learning model after initialization parameters to predict the unlabeled data and generate pseudo labels;

[0050] The pseudo labels are used as supervision information together with the labeled data for further training of the initial semi-supervised learning model after initializing the parameters. Through an iterative training process, the final semi-supervised learning model is obtained.

[0051] An intelligent agent optimization system based on reinforcement learning strategy, comprising:

[0052] The first optimization module is used to optimize the agent's reinforcement learning algorithm using a proximal strategy;

[0053] The second optimization module is used to combine the optimized reinforcement learning algorithm with the deep neural network as the input processing of the intelligent agent;

[0054] The third optimization module is used to optimize the collaborative strategy of the intelligent agent using the MADDPG algorithm;

[0055] The fourth optimization module is used to optimize the environmental adaptability of the intelligent agent by using a simulated environment and semi-supervised learning method.

[0056] A computing device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned intelligent agent optimization method based on reinforcement learning strategy when executing the computer program.

[0057] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned intelligent agent optimization method based on reinforcement learning strategy.

[0058] The beneficial effects of the present invention are as follows: by limiting the step size of the strategy update through the proximal strategy, the stability of learning can be effectively improved, and the problem of training instability or performance degradation caused by excessive strategy updates can be reduced, which enables the intelligent agent to converge to a better strategy more smoothly during the training process, thereby improving the overall training efficiency and performance. The deep neural network can process high-dimensional input data. After combining with the reinforcement learning algorithm, the intelligent agent can handle more complex tasks and environments. This combination enables the intelligent agent to extract features from the raw data more effectively, and then make more accurate decisions, thereby improving the generalization ability of the intelligent agent and the ability to cope with complex environments. By adopting the MADDPG algorithm to consider the behavior of other intelligent agents to optimize This collaborative mechanism enables the agent to better cooperate or compete with other agents in a shared environment to achieve global optimal or better collaborative effects. In complex multi-agent systems, this optimization can significantly improve the performance and efficiency of the overall system. The simulation environment provides a controllable and repeatable training environment for the agent, reducing its dependence on real-world interactions. At the same time, the semi-supervised learning method uses a small amount of labeled data and a large amount of unlabeled simulated data, allowing the agent to continuously improve and optimize itself during the learning process. This method not only improves the environmental adaptability of the agent, but also reduces its dependence on a large amount of labeled data, reducing the cost of data collection and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of the steps of an intelligent agent optimization method based on a reinforcement learning strategy provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of the composition of an intelligent agent optimization system based on a reinforcement learning strategy provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] refer to Figure 1 to Figure 2 , an agent optimization method based on reinforcement learning strategy, specifically including:

[0064] Reinforcement learning algorithms that use proximal strategies to optimize agents;

[0065] Combine the optimized reinforcement learning algorithm with the deep neural network as the input processing of the intelligent agent;

[0066] The MADDPG algorithm is used to optimize the collaborative strategy of intelligent agents;

[0067] The environmental adaptability of the intelligent agent is optimized by using simulated environment and semi-supervised learning methods.

[0068] In this embodiment, by limiting the step size of the policy update through the proximal strategy, the stability of learning can be effectively improved, and the training instability or performance degradation caused by excessive policy updates can be reduced. This enables the intelligent agent to converge to a better strategy more smoothly during the training process, thereby improving the overall training efficiency and performance. The deep neural network can process high-dimensional input data. After combining with the reinforcement learning algorithm, the intelligent agent can handle more complex tasks and environments. This combination enables the intelligent agent to extract features from the raw data more effectively, and then make more accurate decisions, thereby improving the generalization ability of the intelligent agent and its ability to cope with complex environments. The MADDPG algorithm is used to consider the behavior of other intelligent agents to optimize Its own strategy, this collaborative mechanism enables the intelligent agent to better cooperate or compete with other intelligent agents in a shared environment to achieve global optimal or better collaborative effects. In a complex multi-agent system, this optimization can significantly improve the performance and efficiency of the overall system. The simulation environment provides a controllable and repeatable training environment for the intelligent agent, reducing its dependence on real-world interactions. At the same time, the semi-supervised learning method uses a small amount of labeled data and a large amount of unlabeled simulated data, so that the intelligent agent can continuously improve and optimize itself during the learning process. This method not only improves the environmental adaptability of the intelligent agent, but also reduces its dependence on a large amount of labeled data, reducing the cost of data collection and processing.

[0069] Preferably, the reinforcement learning algorithm for optimizing the agent using the proximal strategy specifically includes:

[0070] Setting an initial strategy parameter, wherein the initial strategy parameter is used to define the behavior strategy of the agent;

[0071] Run the current strategy in the environment and collect data through the interaction between the agent and the environment. The data includes the actions taken by the agent, the state changes of the environment, and the immediate rewards obtained.

[0072] Using the collected data, calculate the advantage function for each state-action pair;

[0073] Construct the objective function based on the advantage function;

[0074] Optimize the objective function based on the gradient ascent algorithm;

[0075] Update the agent’s strategy parameters based on the optimized objective function;

[0076] Put the updated strategy into the environment again for data collection, and repeat the above steps for iterative learning until the stopping condition is met.

[0077] In this embodiment, the strategy parameters are initialized: First, an initial strategy parameter needs to be set. These parameters define the behavior strategy of the agent, that is, the probability distribution of what action to take in a given state;

[0078] Data collection: Run the current strategy in the environment and collect data through the interaction between the agent and the environment. This data includes the actions taken by the agent, the state changes of the environment, and the immediate rewards obtained;

[0079] Calculate the advantage function: Use the collected data to calculate the advantage function of each state-action pair. The advantage function measures the performance of taking a certain action relative to the average performance of the current strategy and is an important indicator for evaluating the quality of actions in reinforcement learning.

[0080] Optimize the objective function: The proximal strategy updates the policy parameters by optimizing a surrogate loss function, which aims to maintain the similarity between the new and old policies (measured by KL divergence, etc.) while maximizing policy improvement. Specifically, the proximal strategy ensures the stability of learning by limiting the step size of the policy update (by clipping the upper and lower bounds of the ratio of the new and old policies);

[0081] Strategy update: Update the agent’s strategy parameters based on the optimized objective function. This step uses the gradient ascent method to gradually improve the strategy through iterative optimization.

[0082] Repeat the above steps: put the updated strategy into the environment again for data collection, and repeat the above steps for iterative learning until a certain performance standard is reached or the stopping condition is met.

[0083] It should be noted that the specific steps of constructing the objective function based on the advantage function include:

[0084] Calculate the advantage function: The advantage function is expressed as A(s,a)=Q(s,a)-V(s), where Q(s,a) is the state-action value function and V(s) is the state value function, which measures the advantage of taking an action relative to the average behavior in a given state.

[0085] Determine the objective function form: In the proximal policy optimization algorithm, the objective function is usually designed to maximize the expectation of a "surrogate advantage", which is defined based on the ratio between the current policy π and the old policy π_old. There are two common forms: one is a clipped objective based on the probability ratio (ClippedObjective), and the other is a target based on the squared error (Surrogate Loss);

[0086] Clipping objective function: If clipping objective is used, the objective function may be of the form L^CLIP(θ)=E_t[min(r_t(θ)A_t,clip(r_t(θ),1-e,1+e)A_t)], where r_t(θ) is the ratio of action probabilities, A_t is an estimate of the advantage function, and e is a small hyperparameter used to control the range of variation of the probability ratio;

[0087] Squared error objective function: If a squared error objective is used, the objective function may be constructed by minimizing some form of squared error, such as L2], where V_θ(s_t) is an estimate of the state value function and V^_t is the target value.

[0088] The optimization of the objective function according to the gradient ascent algorithm specifically includes:

[0089] Calculate the gradient: First, you need to calculate the gradient of the objective function with respect to the policy parameters. This usually involves differentiating the policy function and calculating the gradient based on the collected data (such as state, action, and reward). In algorithms such as proximal policy optimization (PPO), the gradient calculation involves differentiating the advantage function, probability ratio, etc.

[0090] Applying gradient ascent: Once we have the gradient, we can use the gradient ascent algorithm to update the policy parameters. The basic idea of ​​the gradient ascent algorithm is to update the parameters along the direction of the gradient (i.e., the direction of increasing the objective function value). The update formula is usually in the form of Where θ_new is the updated parameter, θ_old is the parameter before update, α is the learning rate, is the gradient of the objective function with respect to the parameters;

[0091] Adjust the learning rate: The learning rate α is an important hyperparameter that controls the step size of parameter updates. A learning rate that is too large may cause the parameters to update too quickly, thus missing the optimal solution; while a learning rate that is too small may cause the convergence speed to be too slow. In practical applications, it may be necessary to adjust the learning rate through experiments to find the optimal parameter update step size;

[0092] Iterative update: After updating the parameters, the new policy parameters need to be put into the environment again for data collection. Then, the gradient is recalculated based on the new data and the parameters are updated. This process needs to be iterated until the stopping condition is met (such as reaching the maximum number of iterations, the objective function value converges, etc.);

[0093] Strategy improvement and convergence: During the iteration process, the agent's strategy will gradually improve so that it can better adapt to the environment and maximize the expected return. Convergence is an important issue. Since reinforcement learning problems are usually highly nonlinear and random, it may be difficult for the algorithm to find the global optimal solution. However, through reasonable parameter settings and algorithm design, it can be ensured that the algorithm converges to a better solution to some extent.

[0094] Preferably, the optimized reinforcement learning algorithm combined with the deep neural network as the input processing of the intelligent agent specifically includes:

[0095] Preprocess the input data to adapt to the input requirements of the neural network;

[0096] Design the corresponding neural network architecture according to the task characteristics and data characteristics;

[0097] In the neural network architecture, embed the value function network of the reinforcement learning algorithm;

[0098] Define the loss function according to the goal of reinforcement learning;

[0099] Using the preprocessed data as input, a neural network embedded in a value function network is trained to obtain a deep neural network combined with an optimized reinforcement learning algorithm, and the deep neural network is processed as input to the intelligent agent.

[0100] In this embodiment, the deep neural network is good at processing high-dimensional input data, such as images, sounds, etc., which enables the intelligent agent to process information from complex environments, such as visual input, thereby expanding the application scope of the reinforcement learning algorithm; through the hierarchical structure of the neural network, the deep features of the input data are automatically learned, which are crucial for strategy formulation. These feature learning is automatic and does not require manual design, which improves the adaptability and generalization ability of the intelligent agent; the value function network of the reinforcement learning algorithm is embedded in the neural network, and the learning ability of the neural network can be directly used to optimize the policy parameters and accelerate the learning process. At the same time, the weights of the neural network can be adjusted through the back-propagation algorithm, so that the strategy can be adjusted more efficiently. , reduce the volatility of strategy updates and improve learning stability; due to the combination of the advantages of deep learning and reinforcement learning, the intelligent agent can more accurately evaluate the value of state-action pairs and make better decisions based on this, which improves the decision-making quality and success rate of the intelligent agent in complex environments; since deep neural networks have certain noise resistance and fault tolerance, after combining with reinforcement learning algorithms, the intelligent agent has stronger robustness to uncertainty and noise in the environment, and can maintain stable performance in complex and changing environments; by adjusting the architecture and parameters of the neural network, it can flexibly adapt to different tasks and scenarios. At the same time, combined with different reinforcement learning algorithms, the functions and application areas of the intelligent agent can be further expanded.

[0101] It should be noted that according to the specific task (such as classification, regression, generation, etc.) and the characteristics of the data, a suitable neural network architecture is designed, which includes selecting the number of neural network layers, the number of neurons in each layer, the activation function, the optimizer, etc. This step is the core of building a deep learning model; in the neural network architecture, a suitable location is selected to embed the value function network of the reinforcement learning algorithm. The value function network is usually used to estimate the expected return of taking an action in a given state. In deep reinforcement learning, this can be achieved by adding one or more additional output layers, which correspond to the estimate of the value function; according to the goal of reinforcement learning (such as maximizing the cumulative return), an appropriate loss function is defined. The loss function is an indicator that measures the difference between the model's predicted value and the true value. The parameters of the neural network are updated by optimizing the loss function. In reinforcement learning, the loss function is usually related to the error of the value function or the policy function.

[0102] Preferably, in the neural network architecture, a value function network of a reinforcement learning algorithm is embedded, specifically including:

[0103] Analyze the neural network architecture to determine the number and configuration of its input, output and hidden layers;

[0104] Determine the embedding point of the value function network based on the number and configuration of the input and output layers and hidden layers of the neural network architecture;

[0105] At the selected embedding point, a value function network layer is designed, and the value function network is embedded in the value function network layer.

[0106] In this embodiment, the embedding of the value function network enables the neural network to have stronger decision-making ability. Through the state value or action value output by the value function network, the neural network can more accurately evaluate the value of different strategies, thereby making better decisions; the combination of the value function network and the neural network can accelerate the learning process, and the instant feedback provided by the value function network can help the neural network adjust its internal parameters more quickly to adapt to environmental changes, thereby improving learning efficiency; since the value function network can evaluate the value of the strategy from a global perspective, the neural network embedded with the value function network can show stronger generalization ability when facing new environments or new tasks. It can quickly adapt to environmental changes and find strategies suitable for new environments; the embedding of the value function network enables the neural network to use resources more efficiently in the decision-making process. By accurately evaluating the value of different strategies, the neural network can avoid unnecessary calculations and attempts, thereby saving computing resources and time.

[0107] It should be noted that the analysis of neural network architecture is to carefully examine the existing neural network architecture, understand the structure of its input layer, hidden layer (multi-layer), output layer and the connection between them, and determine where in the neural network it is most appropriate to embed the value function network. This usually depends on the nature of the problem, the original design intention of the network and the role of the value function in the decision-making process; the embedding position selection is to choose to embed the value function network in the appropriate position of the neural network. For example, in the Actor-Critic architecture, the value function network (Critic) usually exists in parallel with the policy network (Actor), receiving the same input but outputting different results. The output of the value function network is used to evaluate the action quality of the policy network output. If it is embedded in a more general neural network, it may be necessary to select the output of one or more hidden layers as the input of the value function network, or to fuse the output of the value function network with other parts of the neural network in some form; the integration of the value function network is to modify the neural network architecture and embed the value function network in the selected position. This may involve adding new layers to the neural network, adjusting the connection between layers or modifying the parameters of the layers, ensuring that the input and output of the value function network are compatible with the rest of the neural network and that gradient information can be correctly transmitted for back-propagation training.

[0108] Preferably, the MADDPG algorithm is used to optimize the collaborative strategy of the intelligent agent, specifically including:

[0109] Clarify the roles and responsibilities of each agent in the environment;

[0110] Design communication mechanisms between agents;

[0111] The MADDPG algorithm is based on centralized training and decentralized execution, so that the strategy of each agent takes into account the strategies of other agents during training;

[0112] During the training process, the synergy between agents is regularly evaluated;

[0113] Based on the collaborative effect evaluation results, the agent's strategy is fine-tuned.

[0114] In this embodiment, the MADDPG algorithm uses centralized training to enable the intelligent agents to learn the behavior patterns and strategies of other intelligent agents during the training process, thereby avoiding conflicts and redundancies and improving collaborative efficiency. Since each intelligent agent takes the strategies of other intelligent agents into consideration during training, they can better adapt to complex and changing environments and improve the stability and robustness of the overall system. Through effective collaboration and division of labor, the intelligent agents can more reasonably utilize resources, reduce waste and duplication of work, and improve the efficiency of the overall system. The optimization of collaborative strategies not only improves the performance of a single intelligent agent, but also further improves the performance of the overall system through collaboration between intelligent agents, enabling it to better complete complex tasks. The design of the MADDPG algorithm enables the system to flexibly add or delete intelligent agents without the need for large-scale modifications and retraining of the entire system, thereby enhancing the scalability of the system.

[0115] It should be noted that the environment parameters are set, such as the number of agents, their initial positions, the range of executable actions, etc., to clarify the role and responsibilities of each agent in the environment, ensure that their respective tasks are complementary and interrelated, and implement information sharing and exchange strategies between agents, such as through communication networks or shared state spaces, to determine the format and frequency of communication content to ensure collaborative efficiency while avoiding redundancy. Using the characteristics of the MADDPG algorithm, agents are allowed to observe the states and behaviors of other agents during the training phase, so that they can learn how to work more effectively together in a team. In the execution phase, each agent makes decisions and performs actions independently to implement a decentralized execution strategy. During the training process, the collaborative effect between agents is regularly evaluated. The collaborative performance of agents can be tested by simulating tasks or specific scenarios. Evaluation indicators can include task completion, resource utilization efficiency, conflict frequency between agents, etc. According to the results of the collaborative effect evaluation, the strategy of the agent is fine-tuned, which can be achieved by modifying the parameters of the policy network or adjusting the parameters of the learning algorithm. The goal of fine-tuning is to improve the collaborative efficiency between agents and ensure that the entire multi-agent system can better cope with complex environments and tasks.

[0116] Preferably, the method of using a simulated environment and semi-supervised learning to optimize the environmental adaptability of the intelligent agent specifically includes:

[0117] Constructing a simulation environment, wherein the simulation environment is used to simulate the problems encountered by the intelligent agent in actual applications;

[0118] Run the agent in a simulation and record data as it performs tasks;

[0119] Use the collected simulation data and the labeled data to perform semi-supervised learning and obtain a semi-supervised learning model;

[0120] The semi-supervised learning model is deployed in the intelligent agent to optimize the environmental adaptability of the intelligent agent.

[0121] In this embodiment, by learning and training in a simulated environment, the intelligent agent can adapt to various complex and changeable environmental conditions in advance, thereby demonstrating stronger environmental adaptability in real applications; relying on a simulated environment for a large amount of learning can significantly reduce dependence on real-world interactions and reduce trial-and-error costs and risks in practical applications; the semi-supervised learning method utilizes a large amount of unlabeled simulated data, combined with a small amount of labeled data, which can quickly improve the learning efficiency and generalization ability of the model; after dual training in a simulated environment and semi-supervised learning, the intelligent agent can demonstrate greater robustness and stability when facing unknown or changing environments.

[0122] Preferably, the collected simulation data is combined with the labeled data to perform semi-supervised learning to obtain a semi-supervised learning model, which specifically includes:

[0123] Designing an initial semi-supervised learning model, wherein the initial semi-supervised learning model can use unlabeled simulated data and labeled data for learning;

[0124] Combine unlabeled simulated data and labeled data into a training set;

[0125] Use the training set to train the initial semi-supervised learning model to obtain the final semi-supervised learning model;

[0126] The method of using the training set to train the initial semi-supervised learning model to obtain the final semi-supervised learning model specifically includes:

[0127] Using the labeled data to train an initial semi-supervised learning model, and initializing parameters of the initial semi-supervised learning model;

[0128] Introduce unlabeled simulated data into the training process, use the initial semi-supervised learning model after initialization parameters to predict the unlabeled data and generate pseudo labels;

[0129] The pseudo labels are used as supervision information together with the labeled data for further training of the initial semi-supervised learning model after initializing the parameters. Through an iterative training process, the final semi-supervised learning model is obtained.

[0130] In this embodiment, by introducing a large amount of unlabeled simulated data, the model can learn on a wider data distribution, thereby enhancing the generalization ability of unseen data, which helps the model to better cope with various scenarios and changes in practical applications; labeled data usually requires high cost and time investment, while simulated data can be generated relatively easily. Through semi-supervised learning, the model can combine a large amount of unlabeled simulated data with a small amount of labeled data, effectively reducing the dependence on labeled data and reducing data labeling costs; in the initial stage, the labeled data is used to initialize the model parameters, so that the model has a good starting point in the early stage of learning, and then the unlabeled data is introduced to generate pseudo labels, and used as supervision information to further train the model. This iterative training This method can accelerate the convergence process of the model and improve the training efficiency. Since the simulated data can simulate various extreme or abnormal situations, by learning on these data, the model can learn to handle complex and abnormal inputs, thereby improving its stability in the face of abnormal or unpredictable situations in practical applications. By continuously generating and updating pseudo-labels during the iterative training process, the model can self-optimize and adjust to further improve its performance. This self-optimization ability enables the model to adapt to changing environments and needs. Due to the combination of the accuracy of labeled data and the richness of simulated data, the final semi-supervised learning model is often better than the model trained only with labeled data. Specifically, the model is expected to achieve high levels in key indicators such as accuracy, recall, and precision.

[0131] An intelligent agent optimization system based on reinforcement learning strategy, comprising:

[0132] The first optimization module is used to optimize the agent's reinforcement learning algorithm using a proximal strategy;

[0133] The second optimization module is used to combine the optimized reinforcement learning algorithm with the deep neural network as the input processing of the intelligent agent;

[0134] The third optimization module is used to optimize the collaborative strategy of the intelligent agent using the MADDPG algorithm;

[0135] The fourth optimization module is used to optimize the environmental adaptability of the intelligent agent by using a simulated environment and semi-supervised learning method.

[0136] A computing device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned intelligent agent optimization method based on reinforcement learning strategy when executing the computer program.

[0137] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned intelligent agent optimization method based on reinforcement learning strategy.

[0138] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An agent optimization method based on reinforcement learning strategy, characterized in that: Specifically include: Reinforcement learning algorithms that use proximal strategies to optimize agents; Combine the optimized reinforcement learning algorithm with the deep neural network as the input processing of the intelligent agent; The MADDPG algorithm is used to optimize the collaborative strategy of intelligent agents; The environmental adaptability of the intelligent agent is optimized by using simulated environment and semi-supervised learning methods.

2. The intelligent agent optimization method based on reinforcement learning strategy according to claim 1, characterized in that: The reinforcement learning algorithm using the proximal strategy to optimize the intelligent agent specifically includes: Setting an initial strategy parameter, wherein the initial strategy parameter is used to define the behavior strategy of the agent; Run the current strategy in the environment and collect data through the interaction between the agent and the environment. The data includes the actions taken by the agent, the state changes of the environment, and the immediate rewards obtained. Using the collected data, calculate the advantage function for each state-action pair; Construct the objective function based on the advantage function; Optimize the objective function based on the gradient ascent algorithm; Update the agent’s strategy parameters based on the optimized objective function; Put the updated strategy into the environment again for data collection, and repeat the above steps for iterative learning until the stopping condition is met.

3. The intelligent agent optimization method based on reinforcement learning strategy according to claim 2, characterized in that: The optimized reinforcement learning algorithm combined with the deep neural network as the input processing of the intelligent agent specifically includes: Preprocess the input data to adapt to the input requirements of the neural network; Design the corresponding neural network architecture according to the task characteristics and data characteristics; In the neural network architecture, embed the value function network of the reinforcement learning algorithm; Define the loss function according to the goal of reinforcement learning; Using the preprocessed data as input, a neural network embedded in a value function network is trained to obtain a deep neural network combined with an optimized reinforcement learning algorithm, and the deep neural network is processed as input to the intelligent agent.

4. The intelligent agent optimization method based on reinforcement learning strategy according to claim 3, characterized in that: In the neural network architecture, the value function network of the reinforcement learning algorithm is embedded, specifically including: Analyze the neural network architecture to determine the number and configuration of its input, output and hidden layers; Determine the embedding point of the value function network based on the number and configuration of the input and output layers and hidden layers of the neural network architecture; At the selected embedding point, a value function network layer is designed, and the value function network is embedded in the value function network layer.

5. The intelligent agent optimization method based on reinforcement learning strategy according to claim 4, characterized in that: The MADDPG algorithm is used to optimize the collaborative strategy of the intelligent agent, specifically including: Clarify the roles and responsibilities of each agent in the environment; Design communication mechanisms between agents; The MADDPG algorithm is based on centralized training and decentralized execution, so that the strategy of each agent takes into account the strategies of other agents during training; During the training process, the synergy between agents is regularly evaluated; Based on the collaborative effect evaluation results, the agent's strategy is fine-tuned.

6. The intelligent agent optimization method based on reinforcement learning strategy according to claim 5, characterized in that: The method of using a simulated environment and semi-supervised learning to optimize the environmental adaptability of the intelligent agent specifically includes: Constructing a simulation environment, wherein the simulation environment is used to simulate the problems encountered by the intelligent agent in actual applications; Run the agent in a simulation and record data as it performs tasks; Use the collected simulation data and the labeled data to perform semi-supervised learning and obtain a semi-supervised learning model; The semi-supervised learning model is deployed in the intelligent agent to optimize the environmental adaptability of the intelligent agent.

7. The intelligent agent optimization method based on reinforcement learning strategy according to claim 6, characterized in that: The collected simulation data is combined with the labeled data to perform semi-supervised learning to obtain a semi-supervised learning model, which specifically includes: Designing an initial semi-supervised learning model, wherein the initial semi-supervised learning model can use unlabeled simulated data and labeled data for learning; Combine unlabeled simulated data and labeled data into a training set; Use the training set to train the initial semi-supervised learning model to obtain the final semi-supervised learning model; The method of using the training set to train the initial semi-supervised learning model to obtain the final semi-supervised learning model specifically includes: Using the labeled data to train an initial semi-supervised learning model, and initializing parameters of the initial semi-supervised learning model; Introduce unlabeled simulated data into the training process, use the initial semi-supervised learning model after initialization parameters to predict the unlabeled data and generate pseudo labels; The pseudo labels are used as supervision information together with the labeled data for further training of the initial semi-supervised learning model after initializing the parameters. Through an iterative training process, the final semi-supervised learning model is obtained.

8. An intelligent agent optimization system based on reinforcement learning strategy, characterized in that: include: The first optimization module is used to optimize the agent's reinforcement learning algorithm using a proximal strategy; The second optimization module is used to combine the optimized reinforcement learning algorithm with the deep neural network as the input processing of the intelligent agent; The third optimization module is used to optimize the collaborative strategy of the intelligent agent using the MADDPG algorithm; The fourth optimization module is used to optimize the environmental adaptability of the intelligent agent by using a simulated environment and semi-supervised learning method.

9. A computing device, characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the intelligent agent optimization method based on reinforcement learning strategy as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the intelligent agent optimization method based on reinforcement learning strategy described in any one of claims 1 to 7.

Images