Multi-intelligence hybrid collaborative optimization method based on reinforcement learning

Abstract

The present invention discloses a multi-intelligent hybrid collaborative optimization method based on reinforcement learning. It includes the following steps: setting a local agent based on DQN, the local agent is an agent that makes decisions on node sub-problems independently in any application scenario; setting a central agent based on DQN, the central agent is an agent that adjusts the strategy of the local agent from a global level; the local agent realizes distributed control by optimizing local nodes; the central agent realizes centralized control by evaluating the quality of the decision-making behavior of the local agent; each local agent updates the node reinforcement learning model; the central agent updates the global reinforcement learning model. By implementing the present invention, the ability of local agents to make independent decisions can be retained while maximizing global returns, reducing the complexity of the algorithm.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a multi-intelligence hybrid collaborative optimization method based on reinforcement learning. Background Art

[0002] Reinforcement learning is a machine learning method that learns to make decisions through trial and error based on interaction with the environment. This method is universal and has applications in scenarios such as financial quantification, computer vision, and traffic control. Related technologies often involve multiple agents making decisions at the same time to jointly solve an overall problem. However, how to achieve collaborative decision-making optimization between agents remains a difficult problem.

[0003] For example, in the field of traffic control, the coordination control strategies of multiple intelligent agents at intersections are mainly divided into two categories: centralized and distributed. The centralized method is to jointly model the decision-making behaviors of all intelligent agents and optimize them uniformly. There are two dilemmas: first, the complexity of the algorithm explodes with the increase in the number of intelligent agents; second, the scalability is limited and the model cannot be transplanted. The distributed method is to integrate the traffic status and signal status of the intersection and adjacent intersections to update the state space of a single intelligent agent, and use independent optimization to achieve multi-agent collaboration. However, the intelligent agent interaction of this method is obviously insufficient and cannot guarantee global optimality. Summary of the invention

[0004] In view of this, the present invention provides a multi-intelligence hybrid collaborative optimization method based on reinforcement learning.

[0005] According to the first aspect, an embodiment of the present invention provides a multi-intelligent hybrid collaborative optimization method based on reinforcement learning, comprising the following steps: setting a local agent based on DQN, wherein the local agent is an agent that makes decisions on node sub-problems separately in any application scenario; setting a central agent based on DQN, wherein the central agent is an agent that adjusts the strategy of the local agent from a global level; the local agent realizes distributed control by optimizing local nodes, wherein the optimization of local nodes includes obtaining the current state data of the corresponding node and inputting it into the node reinforcement learning model for processing, outputting the Q value of all actions of the node, selecting an action based on the ε-greedy strategy, and applying the action to the corresponding node to obtain the next state data and reward value of the corresponding node; the central agent realizes centralized control by evaluating the quality of the decision-making behavior of the local agent, wherein the evaluation of the quality of the decision-making behavior of the local agent includes obtaining the global state data and inputting it into the global reinforcement learning model for processing, outputting the evaluation to each local agent, and obtaining the global next state data and reward value; each local agent updates the node reinforcement learning model; and the central agent updates the global reinforcement learning model.

[0006] According to the second aspect, an embodiment of the present invention provides a design method for a signal control model based on reinforcement learning, wherein the model includes: a feature extraction module, which uses a convolutional neural network to capture the key features of image information, and splices it with numerical characteristics into a state space and then inputs it into a mapping relationship mining module; a mapping relationship mining module, which is used to mine the mapping pattern of traffic status and potential rewards, input the state space from the feature extraction module into a fully connected layer for processing, and then input the obtained mapping pattern and the current phase information into a phase gating module; a phase gating module, which is used to mine the optimal output under different phase conditions during the model training stage, build an individual phase distributed training decision model, and use the decision model corresponding to the current phase to determine the decision behavior during macro decision-making.

[0007] According to the third aspect, an embodiment of the present invention provides a signal control model training method based on reinforcement learning, comprising the following steps: obtaining multiple historical samples; setting a separate storage area for each combination of phase and decision action in the state data; for the multiple historical samples, placing them in corresponding storage areas according to the phase-action combination in their data; randomly extracting an equal amount of samples from each storage area according to a preset sample size extracted from each area to form a training sample data set; updating the signal control model parameters as described in the second aspect or any embodiment of the second aspect according to the current state data and the corresponding output, next state data and reward value in each sample in the training sample data set.

[0008] Optionally, obtaining multiple historical samples includes: obtaining current state data; inputting the current state data into the signal control model as described in the second aspect or any embodiment of the second aspect to obtain corresponding decision data; applying an action to the environment based on the decision data to transfer the environment to the next state and obtain a reward from the environment; combining the current state data, the corresponding decision data, the next state data and the reward value into a sample; obtaining multiple historical samples through multiple interactions with the environment;

[0009] According to the fourth aspect, an embodiment of the present invention provides a signal optimization method for multi-intelligent hybrid collaborative decision-making on a trunk road based on reinforcement learning, comprising the following steps: setting a local intelligent agent based on DQN, wherein the local intelligent agent is a plurality of intelligent agents that control a single intersection on the trunk road; setting a central intelligent agent based on DQN, wherein the central intelligent agent is an intelligent agent that adjusts the strategy of the local intelligent agent from a global level; the local intelligent agent realizes distributed control by optimizing local nodes, wherein the optimization of local nodes includes obtaining the current state data of the corresponding intersection and inputting it into the signal control model as described in the second aspect or any embodiment of the second aspect for processing, and outputting the Q value of all actions at the single intersection, based on ε-gre The edy strategy selects an action and applies the action to the corresponding intersection to obtain the next state data and reward value of the corresponding intersection; the central agent realizes centralized control by evaluating the quality of the decision-making behavior of the local agents, and the evaluation of the quality of the decision-making behavior of the local agents includes obtaining the global state data and inputting it into the global signal control model for processing, and outputting it to each local agent for evaluation, to obtain the global next state data and reward value; each local agent updates the signal control model of a single intersection as described in the second aspect or any embodiment of the second aspect using the signal control model training method as described in the third aspect or any embodiment of the third aspect; the central agent updates the global signal control model using gradient descent.

[0010] Optionally, setting up a local intelligent agent includes: defining the local intelligent agent state as the number of vehicles carried on all entrance lanes of the corresponding intersection, the number of vehicles carried on the exit lanes, and the current phase data; defining the local intelligent agent output as whether the traffic signal at the corresponding intersection maintains the current phase or switches to the next phase; defining the local intelligent agent reward as the inverse of the sum of the queue lengths of the entrance lanes of the corresponding intersection, plus the evaluation from the central intelligent agent.

[0011] Optionally, a central intelligent agent is set, including: defining the central intelligent agent state as the number of vehicles carried by all lanes globally and the current phase data of all intersections; defining the central intelligent agent output as the evaluation fed back to each local intelligent agent; defining the central intelligent agent return as a linear weighted combination of the queue length at the entrance of the intersecting road, the queue length at the entrance of the main road, and the number of stops on the main road.

[0012] The technical solution of the present invention has the following advantages:

[0013] (1) The present invention combines the advantages of both distributed and centralized collaborative strategies and proposes a multi-intelligent hybrid collaborative optimization method: the local agent only makes decisions for the corresponding local nodes, the central agent evaluates the consistency of the local agent with the global goal based on the global state, and feeds back the evaluation to the local agent. This architecture maximizes the global return through interactive cooperation between agents while retaining the ability of local agents to make independent decisions, reducing the complexity of the algorithm.

[0014] (2) For specific signal optimization problems, phase information is more important than other state information. The phase gating module in the signal control model is trained through distributed training with different phases, ensuring the key role of phase in decision-making.

[0015] (3) During system operation, the interaction between the intelligent agent and the environment will produce a variety of phase-action combination samples, and the probability of occurrence of each combination varies greatly. The random sampling strategy has the risk of insufficient samples of low-frequency phase-action schemes, resulting in underfitting. For this reason, the signal control model training method stores different phase-action combination samples in independent memory banks. During training, samples are collected in equal amounts from multiple memory banks to ensure balanced training under different conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following is a brief introduction to the drawings required for the specific implementation or the prior art description:

[0017] Figure 1 Flow chart of a multi-intelligence collaborative optimization method based on reinforcement learning in an embodiment of the present invention;

[0018] Figure 2 It is a principle block diagram of a signal control model based on reinforcement learning in an embodiment of the present invention;

[0019] Figure 3 This is an example diagram of a feature extraction module in a signal control model based on reinforcement learning in an embodiment of the present invention;

[0020] Figure 4 This is an example diagram of a mapping relationship mining module in a signal control model based on reinforcement learning in an embodiment of the present invention;

[0021] Figure 5 This is an example diagram of a phase gating module in a signal control model based on reinforcement learning in an embodiment of the present invention;

[0022] Figure 6 Flow chart of a signal control model training method based on reinforcement learning in an embodiment of the present invention;

[0023] Figure 7 This is an example diagram of a signal control model training method based on reinforcement learning in an embodiment of the present invention;

[0024] Figure 8 This is an example diagram of obtaining multiple historical samples in the signal control model training method based on reinforcement learning in an embodiment of the present invention;

[0025] Fig. 9This is a specific example architecture diagram of the signal optimization method for trunk road multi-intelligent hybrid collaborative decision-making based on reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The embodiment of the present invention provides a multi-intelligence collaborative optimization method based on reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0028] A local agent based on DQN is set, where the local agent is an agent that makes decisions on node sub-problems independently in any application scenario.

[0029] Exemplarily, for different application scenarios, the definition of local agents is different. When the application scenario is a multiplayer game scenario, the local agent is a reinforcement learning agent that controls each player, and its agent design can be based on the game scenario as the state, the game operation as the action, and the single player score plus the evaluation from the central agent as the reward. When the application scenario is a multi-intersection traffic control scenario, the local agent is a reinforcement learning agent that controls the traffic signals of each intersection separately at each intersection, and its agent design can be based on the number of vehicles carried by all entrances of the intersection as the state, maintaining or switching the current phase as the action, and the opposite of the sum of the queue lengths of all entrances of the intersection plus the evaluation from the central agent as the reward. This embodiment does not limit the application scenario and agent design method, and those skilled in the art can determine it as needed.

[0030] A DQN-based central agent is set, where the central agent is an agent that adjusts the strategies of the local agents from a global level.

[0031] Exemplarily, for different application scenarios, the definition of the central agent is different. When the application scenario is a multiplayer game scenario, the central agent is a reinforcement learning agent that collaborates with all players, and its agent design can be based on the game scenario as the state, the additional score given to each player as the output, and whether all players pass the level as the reward. When the application scenario is a multi-intersection traffic control scenario, the central agent is a reinforcement learning agent that collaborates with all intersection traffic signals, and its agent design can be based on the number of vehicles carried by all lanes as the state, the evaluation of each local agent as the output, and the opposite of the sum of the queue lengths of all lanes as the reward. This embodiment does not limit the application scenario and agent design method, and those skilled in the art can determine it as needed.

[0032] The local intelligent agent realizes distributed control by optimizing local nodes. The optimization of local nodes includes obtaining the current state data of the corresponding node and inputting it into the node reinforcement learning model for processing, outputting the Q value of all actions at a single intersection, selecting actions based on the ε-greedy strategy, and applying the actions to the corresponding intersection to obtain the next state data and reward value of the corresponding node.

[0033] The central agent achieves centralized control by evaluating the quality of the decision-making behavior of local agents. The evaluation of the quality of the decision-making behavior of local agents includes obtaining global state data and inputting it into the global reinforcement learning model for processing, outputting the evaluation to each local agent, and obtaining the global next state data and reward value.

[0034] Exemplarily, the node reinforcement learning model and the global reinforcement learning model are DQN architectures, and the neural network consists of an input layer, three hidden layers of 128 neurons, and an output layer.

[0035] Each local agent updates the node reinforcement learning model.

[0036] For example, in each process of optimizing a local node by a local agent, the sum of the current state data, the corresponding decision data, the next state data, the reward from the environment and the evaluation from the central agent is used to form a sample, thereby obtaining multiple samples. The multiple samples are input into the reinforcement learning model to train the reinforcement learning model and update the neural network parameters.

[0037] The central agent updates the global reinforcement learning model.

[0038] For example, in each process in which the central agent evaluates the quality of the decision-making behavior of the local agent, the current state data, the corresponding output, the next state data and the global reward value are combined into a sample, thereby obtaining multiple samples. The multiple samples are input into the reinforcement learning model to train the reinforcement learning model and update the neural network parameters.

[0039] The multi-intelligence hybrid collaborative optimization method provided in this embodiment combines the advantages of both distributed and centralized collaborative strategies: local agents only make decisions for corresponding local nodes, and the central agent evaluates the consistency of local agents with global goals based on the global state, and feeds back the evaluation to the local agents. This architecture maximizes the global return through interactive cooperation between agents while also retaining the ability of local agents to make independent decisions, reducing the complexity of the algorithm.

[0040] The embodiment of the present invention provides a design method of a signal control model based on reinforcement learning, such as Figure 2 As shown, the model includes:

[0041] The feature extraction module uses convolutional neural networks to capture the key features of image information, and splices them with numerical characteristics into a state space and then inputs them into the mapping relationship mining module.

[0042] For example, Figure 3 As shown in the figure, the state data includes the corresponding intersection image, the waiting time of all vehicles on the entrance lanes, the queue length of all entrance lanes, and the current phase data. The image is processed by 8×8 convolutional neural network, 4×4 convolutional neural network and Faltten layer, and then spliced ​​with the numerical characteristics such as the waiting time of all vehicles on the entrance lanes, the queue length of all entrance lanes, and the current phase data as the state space.

[0043] The mapping relationship mining module is used to mine the mapping pattern between traffic status and potential rewards. The state space from the feature extraction module is input into the fully connected layer for processing, and then the obtained mapping pattern and current phase information are input into the phase gating module.

[0044] For example, Figure 4 As shown in the figure, a fully connected neural network with three hidden layers and 128 neurons in each hidden layer is built. The Adam optimization algorithm is used to input the state space from the feature extraction module into the neural network architecture for processing.

[0045] The phase gating module is used to explore the optimal output under different phase conditions during the model training stage, build an individual phase distributed training decision model, and use the decision model corresponding to the current phase to determine the Q value of all actions during macro decision-making.

[0046] For example, Figure 5 As shown, if the traffic signal contains four phases: 0 east-west straight, 1 east-west left turn, 2 north-south straight, and 3 north-south left turn, a separate decision model is set for each phase, and the data from the mapping relationship mining module and the current phase data from the sample will be input into the decision model corresponding to the phase for processing.

[0047] The design method of the signal control model based on reinforcement learning provided in this embodiment avoids the problem that phase information is ignored due to the huge state space. For specific signal optimization problems, phase information has a higher importance than other state information. The phase gating module in the signal control model ensures the key role of phase in decision-making through distributed training of different phases.

[0048] The embodiment of the present invention provides a signal control model training method based on reinforcement learning, such as Figure 6 As shown, the following steps are included:

[0049] Get multiple historical samples.

[0050] Exemplarily, traffic flow data is obtained through surveillance cameras and other means to extract state data; the state data is processed using a constructed signal control model to obtain decision data; the decision data is applied to traffic signals to transfer traffic flow to the next state and calculate the reward; such a set of traffic states, decision data, next state and corresponding rewards are combined into a sample, and the above process is repeated to obtain multiple historical samples.

[0051] For each combination of phase and decision action in the state data, a separate storage area is provided.

[0052] For example, Figure 7 As shown, if the traffic signal has four phases: 0 east-west straight, 1 east-west left turn, 2 north-south straight, and 3 north-south left turn, and the decision data has two actions: 0 to maintain the current phase or 1 to switch to the next phase, then there are eight combinations of phases and actions: 00, 01, 10, 11, 20, 21, 30, and 31, so eight storage areas are set.

[0053] The multiple historical samples are placed in corresponding storage areas according to the phase-action combinations in their data.

[0054] For example, if the phase in the traffic state of a certain sample is 1, going straight east-west, and the decision data is 0, maintaining the current phase, it will be placed in the storage area corresponding to 10, and the same operation will be performed on each sample in turn.

[0055] According to the preset sample quantity to be extracted from each area, an equal amount of samples are randomly extracted from each of the storage areas to form a training sample data set.

[0056] For example, Figure 7 As shown, if the preset sample size extracted from each area is 1, a sample will be randomly extracted from each of the eight storage areas to obtain eight samples to form a training sample data set. This embodiment does not limit the preset sample size extracted from each area, and those skilled in the art can determine it as needed.

[0057] The parameters of the signal control model in the above embodiment are updated according to the current state data and the corresponding output, next state data and reward value in each sample in the training sample data set.

[0058] The signal control model training method provided in this embodiment is conducive to uniform and sufficient training of the agent in each case. During the operation of the system, the interaction between the agent and the environment will produce a variety of phase-action combination samples, and the probability of occurrence of each combination varies greatly. The random sampling strategy has the risk of insufficient sample size of low-frequency phase-action schemes, resulting in underfitting. For this reason, the signal control model training method stores different phase-action combination samples in independent memory banks, and collects samples from multiple memory banks in equal amounts during training to ensure balanced training in different situations.

[0059] As an optional implementation of this embodiment, multiple historical samples are obtained, such as Figure 8 As shown, it includes: obtaining current state data; inputting the current state data into a signal control model such as in the above embodiment to obtain corresponding decision data; applying an action to the environment based on the decision data to transfer the environment to the next state and obtain a reward from the environment; combining the current state data, the corresponding decision data, the next state data and the reward value into a sample; and obtaining multiple historical samples through multiple interactions with the environment.

[0060] Exemplarily, traffic flow data is acquired through surveillance cameras or other means, and state data is extracted; a signal control model such as that in the above embodiment is constructed, and the state data is processed to obtain decision data to maintain the current phase; the current phase is maintained, the traffic flow is transferred to the next state and the reward obtained is calculated; such a set of traffic status, decision data, next state and corresponding reward are combined into a sample, and the above process is repeated to obtain multiple historical samples.

[0061] The embodiment of the present invention provides a signal optimization method for trunk road multi-intelligence hybrid collaborative decision-making based on reinforcement learning, comprising the following steps:

[0062] A local intelligent agent is set, wherein the local intelligent agent is an intelligent agent that controls a single intersection on a plurality of main roads.

[0063] Exemplarily, the state of the local agent is defined as the number of vehicles carried on all entrance lanes and exit lanes of the corresponding intersection, as well as the current phase data; the action of the local agent is defined as the traffic signal of the corresponding intersection maintaining the current phase or switching to the next phase; the reward of the local agent is defined as the inverse of the sum of the queue lengths of the entrance lanes of the corresponding intersection, plus the evaluation from the central agent.

[0064] A central agent is set, where the central agent is an agent that adjusts the strategies of the local agents from a global level.

[0065] Exemplarily, the state of the central agent is defined as the number of vehicles carried by all lanes globally and the current phase data of all intersections; the output of the central agent is defined as the evaluation fed back to each local agent; the reward of the central agent is defined as a linear weighted combination of the queue length at the entrance of the intersecting road, the queue length at the entrance of the main road, and the number of stops on the main road.

[0066] The local intelligent agent realizes distributed control by optimizing local nodes, and the optimization of local nodes includes obtaining the current state data of the corresponding intersection and inputting it into the signal control model in the above embodiment for processing, outputting the Q value of all actions of a single intersection, selecting actions based on the ε-greedy strategy, and applying the actions to the corresponding intersection, and applying the decision-making actions to the corresponding intersection to obtain the next state data and reward value of the corresponding intersection.

[0067] The central agent realizes centralized control by evaluating the quality of the decision-making behavior of the local agents. The evaluation of the quality of the decision-making behavior of the local agents includes obtaining the global state data and inputting it into the global signal control model for processing, outputting the evaluation to each local agent, and obtaining the global next state data and reward value.

[0068] For example, Fig. 9 As shown, for a main road with three intersections, a local agent is set for each intersection, namely local agent 1, local agent 2, and local agent 3. Each agent obtains the state input of the corresponding intersection and processes it as the signal control model in the above embodiment, outputs the Q value of each action, selects the action based on the ε-greedy strategy and acts on the corresponding intersection, and obtains the reward from the environment. Then, the central agent obtains the global state input to the global signal control model for processing, feeds back an evaluation to each local agent and receives the global reward from the environment. The global signal control model is a general DQN architecture, and the neural network consists of an input layer, three hidden layers of 128 neurons, and an output layer. The goal of the local agent is to maximize the sum of the reward from the environment and the evaluation from the central agent, while the goal of the central agent is to maximize the global reward.

[0069] Each local agent updates the signal control model of the single intersection in the above embodiment using the signal control model training method in the above embodiment. For details, please refer to the corresponding part of the above embodiment, which will not be repeated here.

[0070] The central agent updates the global signal control model using gradient descent.

[0071] Exemplarily, the central agent obtains the global state and inputs it into the global signal control model for processing, feeds back an evaluation to each local agent and receives a global reward R from the environment; each set of global state data, corresponding output, next state data and reward value are combined into a sample; multiple historical samples are obtained through multiple interactions with the environment; the value V of the current global state and the value V of the next state are calculated next , then TD_error is:

[0072] TD_error=R+γ*V next –V

[0073] The discount factor γ∈[0,1] controls the importance of immediate returns relative to future returns. TD_error backpropagation is used to update the parameters and weights of the global signal control model network.

[0074] The above embodiments are merely examples for clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A signal optimization method for multi-intelligent hybrid collaborative decision-making on arterial roads based on reinforcement learning, characterized in that: The following steps are involved: Setting a local agent based on DQN, wherein the local agent is an agent controlling a single intersection on a plurality of arterial roads; Setting a central agent based on DQN, wherein the central agent is an agent that adjusts the strategy of the local agents from a global level; The local agent realizes distributed control by optimizing local nodes, and the optimization of local nodes includes obtaining the current state data of the corresponding intersection and inputting it into the single intersection reinforcement learning model for processing, outputting the Q value of all actions of the single intersection, selecting actions based on the ε-greedy strategy, and applying the actions to the corresponding intersection to obtain the next state data and reward value of the corresponding intersection; The central agent realizes centralized control by evaluating the quality of the decision-making behavior of the local agents, wherein the evaluation of the quality of the decision-making behavior of the local agents includes obtaining global state data and inputting it into the global reinforcement learning model for processing, outputting the evaluation to each local agent, and obtaining the global next state data and reward value; Each local agent updates the single-intersection reinforcement learning model; The central agent updates the global reinforcement learning model; The single intersection reinforcement learning model includes: The feature extraction module uses a convolutional neural network to capture the key features of image information, and then splices them with numerical features into a state space and inputs them into the mapping relationship mining module; The mapping relationship mining module is used to mine the mapping pattern between traffic status and potential rewards. The state space from the feature extraction module is input into the fully connected layer for processing, and then the obtained mapping pattern and current phase information are input into the phase gating module. The phase gating module is used to explore the optimal output under different phase conditions during the model training stage, build an individual phase distributed training decision model, and use the decision model corresponding to the current phase to determine the decision behavior during macro decision-making.

2. The method according to claim 1, characterized in that Set up a DQN-based local agent, including: The local agent state is defined as the number of vehicles carried by all entrance lanes and exit lanes of the corresponding intersection, as well as the current phase data; Define the local agent action as the traffic signal at the corresponding intersection to maintain the current phase or switch to the next phase; The local agent reward is defined as the inverse of the sum of the queue lengths at the entrance of the corresponding intersection, plus the evaluation from the central agent.

3. The method according to claim 2, characterized in that The DQN-based central agent includes: The central agent state is defined as the number of vehicles carried by all lanes and the current phase data of all intersections; Define the central agent output as the evaluation fed back to each local agent; The central agent reward is defined as a linear weighted combination of the queue length at the entrance of the intersecting road, the queue length at the entrance of the main road, and the number of stops on the main road.

4. The method according to claim 1, characterized in that: The updating of the single intersection reinforcement learning model includes: Each set of current state data, corresponding output, next state data and reward value is combined into a sample; Obtain multiple historical samples through multiple interactions with the environment; The multiple historical samples are used to update the model parameters of the target reinforcement learning model.

5. The method according to claim 4, characterized in that The updating of the model parameters of the target reinforcement learning model by using the multiple historical samples includes: Use the partition memory method to obtain a uniformly distributed training sample data set; Update the model parameters of the target reinforcement learning model according to the current state data and the corresponding output, next state data and reward value in each sample in the training sample data set; The partition memory comprises the following steps: For each combination of phase and decision action in the state data, a separate storage area is provided; For the plurality of historical samples, the plurality of historical samples are placed into corresponding storage areas according to the phase-action combination in the data; According to the preset sample quantity to be extracted from each area, an equal amount of samples are randomly extracted from each of the storage areas to form a training sample data set.

Images