Strategy network training method and computing device
By dividing the parameters of the policy network into multiple segments in a distributed multi-agent reinforcement learning environment, the problem of differences in the performance of each agent's policy network is solved, and learning performance and decision-making efficiency are improved.
Patent Information
- Application Number
- CN202311051398.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-18
- Publication Date
- 2025-02-21
AI Technical Summary
In the distributed multiagent reinforcement learning environment, there are differences in the policy network performance of each agent, resulting in unstable learning performance and delayed decision making.
By dividing the parameters of the policy network into multiple segments and exchanging and mixing these segmented parameters between agents, the policy network collaborative training of each agent is achieved.
It effectively reduces the communication overhead caused by parameter exchange, improves the collaborative learning performance between agents, and enhances the stability of the policy network and the timeliness of decision-making.
Smart Images
Figure CN119494418A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a training method and computing device for a policy network. Background Art
[0002] In the field of AI, the multi-agent reinforcement learning (MARL) method adopts a centralized training method. Specifically, a centralized controller uses the state of the global environment, the observations and actions of all agents to train the policy network and the value network, transforming the learning of multi-agents into the learning of single agents. However, in some highly dynamic autonomous mobility scenarios, such as intelligent transportation systems, centralized methods pose a potential threat to the stability of the learning performance of the agents and the timeliness of decision-making. In addition, for multi-agent systems with large action spaces and state spaces, the introduction of centralized controllers will lead to dimensionality disasters, greatly increasing the computational complexity of the central controller, and even making it impossible to calculate in severe cases.
[0003] In order to adapt to the highly dynamic autonomous mobile scenarios, a distributed multi-agent independent reinforcement learning method has been proposed. In the distributed method, each agent has a certain computing power to process information and independent decision-making ability, which has great advantages in terms of computing power, operating cost, robustness, and scalability. In the distributed method, each agent usually adopts independent reinforcement learning (IRL), that is, each agent can access and respond to a limited part of the observation of the global environment, and take actions independently according to its own strategy based on this partial observation. However, in this distributed training method, the performance of the policy network of each agent has limitations. Each agent is independent and scattered, and each agent obtains different samples for different learning environments, resulting in huge differences in the performance of the strategies of the policy network of each agent. Summary of the invention
[0004] The present application provides a training method and computing device for a policy network. During a distributed training process, each intelligent agent achieves collaboration by exchanging parameters of each other's policy networks, thereby solving the problem that during a distributed training process, the policy networks of each intelligent agent have differences in performance due to independent learning of each other.
[0005] In the first aspect, the present application provides a method for training a policy network. The method includes: dividing multiple parameters of the policy network of a target agent into multiple segments according to a preset method; sending an exchange request to a reference agent corresponding to each segment, wherein multiple parameters of the policy network of the reference agent are divided into the same number of segments according to the preset method, and the exchange request corresponding to each segment is used to obtain the parameters of the same segment of the policy network of the reference agent corresponding to each segment; mixing the reference parameters and multiple parameters of the policy network of the target agent to obtain mixed parameters, wherein the reference parameters include the parameters of the same segment of the policy network of the reference agent corresponding to each segment; and updating multiple parameters of the policy network of the target agent according to the mixed parameters.
[0006] In the above scheme, the training method can be applied to the scenario of distributed training of the policy network of intelligent agents. After the policy network of an intelligent agent has trained multiple parameters of the policy network, the purpose of collaborative training of the policy networks of multiple intelligent agents is achieved by exchanging the policy network parameters of each intelligent agent. This solves the problem that the policy networks of each intelligent agent have different performances due to independent learning of each other during the distributed training process. In addition, in the process of exchanging the parameters of the policy network, for the parameters of each segment of the intelligent agent, the parameters of the same segment are requested from the reference intelligent agent corresponding to each segment, thereby realizing segmented coordination of the parameters of the policy network between intelligent agents, which can reduce the problem of high communication overhead in the training process caused by parameter exchange.
[0007] In a possible implementation of the first aspect, the mixing of the reference parameters and the multiple parameters of the strategy network of the target intelligent agent to obtain the mixed parameters includes: determining the mixing weights corresponding to the reference parameters; and mixing the reference parameters and the multiple parameters of the strategy network of the target intelligent agent according to the mixing weights.
[0008] In the above scheme, the parameters of the target agent and the reference parameters are mixed according to the weights corresponding to the reference parameters, thereby reducing the complexity of parameter mixing.
[0009] In a possible implementation of the first aspect, before mixing the reference parameters and multiple parameters of the strategy network of the target intelligent agent to obtain the mixed parameters, the method also includes: if the relative advantage value corresponding to the reference parameter is equal to or greater than the advantage threshold, determining to mix based on the reference parameter; if the relative advantage value is less than the advantage threshold, determining not to mix based on the reference parameter.
[0010] In the above scheme, judging whether the reference parameters can be mixed based on their relative advantage values can avoid mixing worthless parameters and improve the effectiveness of parameter interaction. Mixing the agent based on valuable reference parameters can enable the agent to obtain higher cumulative rewards during the training of the policy network and improve the performance of the policy network.
[0011] In a possible implementation of the first aspect, determining the hybrid weight corresponding to the reference parameter includes: determining an upper bound of the hybrid weight based on the relative advantage value corresponding to the reference parameter and the Fisher information matrix corresponding to the target agent; determining the hybrid weight based on the upper bound of the hybrid weight.
[0012] In a possible implementation manner of the first aspect, the method further includes: determining the Fisher information matrix based on the multiple parameters corresponding to the target intelligent agent and the multiple historical parameters obtained from the previous training corresponding to the target intelligent agent.
[0013] In a possible implementation manner of the first aspect, the method further includes: determining the relative advantage value according to the reference parameter and the multiple parameters corresponding to the target agent.
[0014] In a possible implementation of the first aspect, the policy network includes a behavior policy network and a target policy network. Before dividing the multiple parameters of the policy network of the target intelligent agent into multiple segments, the method also includes: the behavior policy network determines the training samples according to the initial parameters and state space corresponding to the target intelligent agent; the value network determines the advantage values corresponding to the initial parameters according to the training samples; and the target policy network determines the multiple parameters of the policy network of the target intelligent agent according to the training samples and the advantage values.
[0015] In a possible implementation of the first aspect, the target agent runs on a target vehicle, the reference agent runs on a reference vehicle of the target vehicle, the policy network of the target vehicle is used to determine the target vehicle's next speed based on the target vehicle's current speed, the current speeds of surrounding vehicles corresponding to the target vehicle, and the relative distance between the target vehicle and the surrounding vehicles, and the reference vehicle includes the surrounding vehicles. The surrounding vehicles may include vehicles located in front of, behind, on the left, and on the right of the target vehicle.
[0016] In the above scheme, the vehicle's decision-making performance on vehicle speed during autonomous driving can be improved and the decision-making error can be reduced.
[0017] In a second aspect, the present application provides an automatic driving method. The method includes: determining the state of the target vehicle at the current time, the state including the driving speed of the target vehicle at the current time, the driving speed of the surrounding vehicles corresponding to the target vehicle at the current time, and the relative distance between the target vehicle and the surrounding vehicles; using a policy network to determine the change in the driving speed of the target vehicle at the next time according to the state of the target vehicle at the current time; determining the driving speed of the target vehicle at the next time according to the driving speed of the target vehicle at the current time and the change in the driving speed of the target vehicle at the next time. Wherein, the policy network is obtained according to the training method of the policy network provided by the first aspect or any possible implementation of the first aspect. Wherein, the change in driving speed can indicate the driving operation of the vehicle. When the change is positive, the vehicle can accelerate at the next time; when the change is negative, the vehicle can decelerate at the next time.
[0018] In a third aspect, the present application provides a training device for a policy network, which includes: a switching module, a mixing module and a training module.
[0019] Among them, the exchange module is used to divide multiple parameters of the strategy network of the target intelligent agent into multiple segments according to a preset method, and send exchange requests to the reference intelligent agent corresponding to each segment. The multiple parameters of the strategy network of the reference intelligent agent are divided into the same number of segments according to the preset method, and the exchange requests corresponding to each segment are used to obtain the parameters of the same segment of the strategy network of the reference intelligent agent corresponding to each segment.
[0020] Among them, the mixing module is used to mix the reference parameters and multiple parameters of the strategy network of the target intelligent agent to obtain mixed parameters, and the reference parameters include the parameters of the same segment of the strategy network of the reference intelligent agent corresponding to each segment.
[0021] The training module is used to update multiple parameters of the strategy network of the target agent according to the mixed parameters.
[0022] In a possible implementation manner of the third aspect, the mixing module is specifically used to: determine the mixing weight corresponding to the reference parameter; and mix the reference parameter and multiple parameters of the strategy network of the target intelligent agent according to the mixing weight.
[0023] In a possible implementation of the third aspect, before mixing the reference parameter and the multiple parameters of the policy network of the target agent, the mixing module is further used to: determine whether to mix based on the reference parameter according to the relative advantage value and the advantage threshold. If the relative advantage value corresponding to the reference parameter is equal to or greater than the advantage threshold, it is determined to mix based on the reference parameter; if the relative advantage value is less than the advantage threshold, it is determined not to mix based on the reference parameter.
[0024] In a possible implementation of the third aspect, the mixing module is specifically used to: determine the upper limit of the mixing weight based on the relative advantage value corresponding to the reference parameter and the Fisher information matrix corresponding to the target intelligent agent; determine the mixing weight based on the upper limit of the mixing weight.
[0025] In a possible implementation manner of the third aspect, the mixing module is specifically used to determine the relative advantage value according to the reference parameter and the multiple parameters corresponding to the target intelligent agent.
[0026] In a possible implementation of the third aspect, the hybrid module is specifically used to determine the Fisher information matrix based on the multiple parameters corresponding to the target intelligent agent and the multiple historical parameters obtained from the previous training corresponding to the target intelligent agent.
[0027] In a possible implementation of the third aspect, the policy network includes a behavior policy network and a target policy network. Before dividing the multiple parameters of the policy network of the target intelligent agent into multiple segments, the training module is also used to: use the behavior policy network to determine the training samples according to the initial parameters and state space corresponding to the target intelligent agent; use the value network to determine the advantage values corresponding to the initial parameters according to the training samples; use the target policy network to determine the multiple parameters of the policy network of the target intelligent agent according to the training samples and the advantage values.
[0028] In a possible implementation of the third aspect, the target agent runs in a target vehicle, the reference agent runs in a reference vehicle around the target vehicle, and the strategy network of the target vehicle is used to determine the driving speed of the target vehicle at the next time based on the current driving speed of the target vehicle, the current driving speed of the reference vehicle, and the relative distance between the target vehicle and the reference vehicle.
[0029] In a fourth aspect, the present application provides a computing device. The computing device includes a processor and a memory. The processor is used to execute a computer program stored in the memory to implement the training method of the policy network provided by the first aspect or any possible implementation of the first aspect.
[0030] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, the computer executes the training method of the policy network provided by the first aspect or any possible implementation of the first aspect.
[0031] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the policy network training method provided by the first aspect or any possible implementation of the first aspect.
[0032] Any of the above-mentioned devices or computing equipment or computer storage media or computer program products are used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding schemes in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 is a flow chart of a training method for a policy network provided in an embodiment of the present application;
[0034] Figure 2 This is a training diagram of a strategy network provided in an embodiment of the present application:
[0035] Figure 3 It is a schematic diagram of collaborative training by exchanging parameters during the training process of a strategy network provided by an embodiment of the present application;
[0036] Figure 4 It is a schematic diagram of determining the mixed weights of reference parameters in a training process of a strategy network provided in an embodiment of the present application;
[0037] Figure 5 This embodiment of the present application provides a method based on Figure 1 A flowchart of a method for performing autonomous driving using a policy network obtained by the method shown;
[0038] Figure 6 It is a schematic diagram of comparing average rewards of different reinforcement learning methods applied to an autonomous driving method provided in an embodiment of the present application;
[0039] Figure 7 This is a method provided by the embodiment of the present application. Figure 2 A schematic diagram comparing the average driving speeds of different numbers of interactions in the method shown and applied to the autonomous driving method;
[0040] Figure 8 This is a method provided by the embodiment of the present application. Figure 2 Schematic diagram comparing the average rewards of different numbers of interactions in the methods shown and applied to autonomous driving methods;
[0041] Fig. 9 This is a method provided by the embodiment of the present application. Figure 2 A schematic diagram showing a comparison of average driving speeds divided into different segments and applied to the autonomous driving method in the method shown;
[0042] Fig.10 It is a structural diagram of a training device for a strategy network provided in an embodiment of the present application;
[0043] Fig.11 It is a structural diagram of a computing device provided in an embodiment of the present application.
[0044] Specific implementation method
[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0046] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.
[0047] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple screen terminals refers to two or more screen terminals.
[0048] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.
[0049] In the field of AI, multi-agent reinforcement learning (MARL) methods are divided into centralized learning systems and distributed learning systems.
[0050] In a centralized learning system, each agent uses a policy network trained by a centralized controller to make decisions based on its observed state, and the centralized controller collects information from each agent as samples and updates the policy network. However, in a centralized learning system, the controller must collect information from all agents before training, and the overall training process is slow. In addition, the size of the agent's state space and action space increases exponentially with the increase in the number of agents, which makes the controller's computational workload large. In addition, in highly dynamic scenarios, the number of agents will change, and the algorithm has poor scalability. In order to solve the problems existing in centralized learning, a distributed learning system was proposed.
[0051] In a distributed learning system, each agent interacts with the environment and updates its own policy network with its own observation information. In such a distributed learning system, such as an IRL system, each agent has a certain computing power to process information and independent decision-making ability, which has great advantages in terms of computing power, operating cost, robustness, scalability, etc., and can solve the problems existing in centralized solutions.
[0052] However, in the IRL system, there is no communication between agents, and the overall convergence of the training process cannot be guaranteed due to the non-stationarity of the learning environment. In addition, in the policy gradient-based reinforcement learning algorithm, the training effect of the policy is usually related to the number and diversity of training samples, that is, the more thorough the exploration of the state space, the more accurate the estimation of the cumulative reward signal. However, in this scheme, multiple agents are independent and dispersed, and different learning environments and deployment methods have different effects on the acquisition of samples. Even if these agents are deployed in the same global environment and have exactly the same configuration, their policy networks have huge differences in performance.
[0053] In order to solve the problem of difficult convergence in the distributed learning system IRL in which each intelligent agent is completely independent, two solutions combining distributed learning systems with communication mechanisms are introduced below.
[0054] In one embodiment, in the distributed learning system, each agent can determine whether the agents need to communicate with each other according to the states observed by each agent during the training of the strategy network, and when communication is determined to be necessary, determine the communication object and communication information. In this solution, the central controller usually needs to pre-train an A1 module for communication decision-making, which is relatively complex.
[0055] In one embodiment, in the distributed learning system, each agent can have a central controller collect the gradient information obtained by each agent during the training of the policy network, and then send it to each agent to accelerate the training after the gradient aggregation, so that the policy network of each agent converges as soon as possible. However, this solution also requires a central controller, and the central controller usually uses the parameter averaging method to aggregate the gradient, resulting in poor performance of the policy network.
[0056] To this end, an embodiment of the present application provides a training method for a policy network, which can solve the problems existing in the above-mentioned solution.
[0057] In the training method of the policy network provided in the embodiment of the present application, for each agent in the distributed learning system, after the policy network is independently trained to obtain multiple parameters of the policy network, the multiple parameters are divided into multiple segments, and each segment of each agent corresponds to at least one reference agent. Taking one agent as an example, the agent sends an exchange request to the reference agent corresponding to each segment to obtain the parameters of the same segment of the reference agent as reference parameters. The reference parameters are then mixed with the multiple parameters obtained from the independent training, and the mixed parameters are used to train its own policy network. By exchanging the parameters of the policy network between the agents, the problem of differences in performance of the policy network caused by independent learning of each agent can be solved, and the network convergence can be accelerated. Exchanging the parameters of the policy network by means of segmented requests can reduce the problem of high communication overhead in the training process caused by parameter exchange. The following is combined with Figure 1 and Figure 2 The training method of the policy network provided in the embodiment of the present application is specifically introduced.
[0058] Figure 1 is a flow chart of a training method for a policy network provided in an embodiment of the present application. The method can be applied to computing devices in highly dynamic application scenarios. An agent is run in each computing device, and each agent corresponds to a policy network. Figure 1 The training method shown in the figure trains its own policy network. Figure 1 As shown, the method may include S101 to S104.
[0059] In S101, the target agent determines multiple parameters of the policy network of the target agent, and divides the multiple parameters into P segments according to a preset method.
[0060] In this step, if Figure 2 As shown, the policy network can include a behavior policy network and the target policy network π θ . Behavioral Policy Network and the target policy network π θThe two networks may include inputs and outputs of the same type. The structures of the two networks may be the same or different, and the embodiments of the present application do not impose any specific restrictions on the types of the two networks. For target-based policy network π θ The training parameters θ old Interact with the environment to obtain training samples. Target policy network π θ Used to update the network parameters θ based on the training samples old , we get the parameter θ. It should be noted that θ old Both θ and θ represent the set of policy network parameters, that is, multiple parameters, not just one parameter. Using two policy networks to update parameters can avoid frequent updates of policy network parameters and improve training efficiency.
[0061] First, at each time step, the target agent observes the environment locally and obtains the state s at the current time step t t , using the current policy network For the current state t Make a decision and get the current state s t The corresponding action a t The target agent will take action a t After acting on the environment, receive the reward r from the environment t and the next state s r+1 . Set the current state s r 、Action a r , Reward t and the next state s t+1 It is stored in the experience replay pool as a training sample.
[0062] Then, at regular intervals, such as time step M, the target agent uses the M training samples in the experience replay pool within that time period to replay the target policy network π θ Perform multiple stochastic gradient descent processes to update the parameters. Specifically, Figure 2 As shown, the optimizer 1 of the target agent replays the state s of the samples in the experience replay pool t Input target policy network π θ , get the state s t The decision action a t and its corresponding probability π θ (a t |s t ), and calculate the loss function L(θ) of the policy network according to formula (1), then calculate the gradient of L(θ) with respect to θ, and update the parameter θ according to the stochastic gradient descent formula (2). It should be noted that when the target agent performs K times of stochastic gradient descent, the last updated parameter θ is KThe target agent is the object of exchange, that is, the parameter θ K Divided into P segments.
[0063]
[0064] In formula (1), λ t Represents π θ (a t |s t )and The ratio of Representing behavioral policy networks The output status s t The corresponding action a t The probability of Behavioral Strategy Network Action t The advantage value of The calculation of clip(λ t , 1-ε, 1+ε) is used to convert λ beyond the bounds (1-ε, 1+ε) t Restricted within the boundary (1-ε, 1+ε), ε is a constant; π θ (s t ) represents the target policy network π θ The output status s t The corresponding probability distribution of the action space, H[π θ (s t )] represents π θ (s t ), β represents the entropy of H[π θ (s t )]’s weight.
[0065]
[0066] In formula (2), θ k+1 represents the parameters of the policy network obtained after the kth stochastic gradient descent, η a represents the weight of the gradient, is the loss function L(θ corresponding to the first updated parameter k )’s gradient value.
[0067] The target agent uses the target policy network π θ While updating the parameters, use the value network V ω Get the state s of each sample in the experience replay pool t The corresponding value V ω (s t ). And, if Figure 2 As shown, the current V is calculated by the optimizer 2 according to formula (3).ω (s t ) and determine the parameter ω of the updated value network according to L(ω) according to formula (4).
[0068] After the parameter ω of the value network is iteratively updated, as Figure 2 As shown, the advantage estimator is used according to formula (5) based on the value V obtained in the last iteration ω (s t ) Calculate the one-step advantage value δ of each sample of the target agent t , and according to formula (6) based on the final obtained δ t Get the above
[0069]
[0070]
[0071] δ t =r t +γV ω (s t+1 )-V ω (s t ) (5)
[0072]
[0073] In the above formulas (3) to (6), Indicates the corresponding M samples Calculate the expected value, γ represents the discount factor, M represents the total number of samples, k∈[1,K], η b express The weight of .
[0074] In this step, the target agent sets the parameter θ K The preset method of dividing into P segments may include dividing the parameter θ according to the number of layers of the target strategy network K Divide into P segments, or divide the parameters of the target policy network into P segments according to P. The present application embodiment does not impose any specific restrictions, as long as the method of dividing the P segments by each agent is the same. In this way, it can be ensured that the meanings of the parameters of the segments exchanged by the agents are consistent.
[0075] Here, P may be a preset value, P is less than or equal to the reference agent with which the target agent can communicate, or P is less than or equal to the total number of agents in the distributed learning system.
[0076] In S102, the target agent sends an exchange request to the reference agent corresponding to each segment, obtains the parameters of the same segment of the policy network of the reference agent corresponding to each segment, and concatenates the parameters of the same segment of the policy network of the reference agent corresponding to each segment to obtain reference parameters.
[0077] In this step, each segment corresponds to at least one reference agent, and the reference agents corresponding to each segment may be the same or different.
[0078] In this step, the target agent determines the exchange request corresponding to each segment. The exchange request corresponding to each segment is used to obtain the parameters of the same segment of the policy network of the reference agent corresponding to the segment.
[0079] The exchange request may include the number of segments P and the segment identifier requested for exchange. After receiving the exchange request, the reference agent divides its multiple parameters into P segments according to a preset method, and sends the parameters of the segment corresponding to the segment identifier to the target agent. The two agents can communicate with each other using D2D (device to device) communication technology. Figure 3 As shown, the target agent can use D2D communication technology to send an exchange request corresponding to segment 2 to the reference agent corresponding to segment 2. The exchange request corresponding to segment 2 may include the number of segments P and segment identifier 2. After receiving the exchange request, the reference agent corresponding to segment 2 divides multiple parameters of its policy network into P segments, and uses D2D communication technology to send the parameters corresponding to segment 2 to the target agent.
[0080] In S103, the target agent mixes the reference parameters with multiple parameters of its own strategy network to obtain mixed parameters.
[0081] Before this step, the target agent can repeatedly execute S102 for multiple rounds of exchanges, and after each round of exchanges, after receiving the parameters returned by the reference agent corresponding to each segment, the parameters of the same segment of the strategy network of the reference agent corresponding to each segment are spliced and reconstructed into reference parameters. So as to obtain multiple reference parameters Among them, the target agent can update the reference agent corresponding to each segment during each round of exchange to obtain more information about the reference agent and improve accuracy.
[0082] In this step, the target agent obtains multiple reference parameters After that, the optimizer 1 can be used according to Figure 4 S401-S403 shown in FIG. 4 obtains the mixing parameter θ mix The specific mixing process will be discussed later. Figure 4I will not go into details here.
[0083] In S104, the target agent updates the policy network of the target agent according to the hybrid parameter.
[0084] In this step, the target agent obtains the mixed parameter θ mix Afterwards, Figure 2 As shown, optimizer 1 sets the parameters θ in the behavior strategy network old and the parameters θ in the target policy network are updated to the mixing parameters θ mix .
[0085] In the above method embodiment, Figure 1 The method shown can be applied to the scenario of distributed training of the policy network of intelligent agents. After the policy network of each intelligent agent has trained multiple parameters of the policy network, the policy network parameters of each intelligent agent are exchanged to achieve the purpose of collaborative training of the policy networks of multiple intelligent agents, thereby solving the problem that the policy networks of each intelligent agent have different performances due to independent learning of each other during the distributed training process.
[0086] In addition, in the process of exchanging the parameters of the policy network, for each segment of the agent, such as Figure 2 As shown, the parameters of the same segment are requested from the reference agents corresponding to each segment respectively, so as to achieve segmented parameter coordination of the policy network between agents, which can reduce the problem of high communication overhead in the training process caused by parameter exchange.
[0087] Figure 4 1 is a flow chart of a method for mixing reference parameters and parameters corresponding to a target agent provided by an embodiment of the present application. Figure 4 As shown, the method may include S401-S403.
[0088] In S401, the target agent obtains multiple reference parameters Finally, effective reference parameters can be screened out from multiple reference parameters for mixing according to the relative advantage values corresponding to each reference parameter.
[0089] Specifically, the target agent can compare the relative advantage value corresponding to the reference parameter with the advantage threshold to determine whether to mix based on the reference parameter. Taking the advantage threshold as 0 as an example, if the reference parameter The corresponding relative advantage value is greater than 0, and the mixture is determined based on the reference parameter; if the reference parameter If the corresponding relative advantage value is equal to or less than 0, it is determined that the reference parameters are not mixed. In this way, effective parameters can be selected from multiple reference parameters for mixing, avoiding negative optimization of the target agent's policy network, so that the target agent can obtain higher cumulative rewards after parameter mixing, thereby improving the accuracy of the network.
[0090] Among them, the target agent can calculate the relative strategy advantage value corresponding to each reference parameter according to formula (7):
[0091]
[0092] In formula (7), Indicates the corresponding M samples Calculate the expected value, Represents the target policy network based on the reference parameters The output status s t The corresponding action a t The probability of t Represents the updated value network V ω The one-step advantage value of the t-th sample calculated before ω. The specific calculation process can refer to formula (5).
[0093] In S402, the target intelligent agent can determine the upper bound of the mixed weight corresponding to each reference parameter according to the relative advantage value corresponding to each reference parameter and the Fisher information matrix corresponding to the target intelligent agent, and then determine the mixed weight corresponding to each reference parameter based on the upper bound of the mixed weight corresponding to each reference parameter.
[0094] Specifically, the target agent can calculate the Fisher information matrix G(self) corresponding to the target agent according to formula (8).
[0095]
[0096] Formula (8), Indicates the corresponding M samples Calculate the expected value.
[0097] Then, the target agent calculates the upper bound of the mixed weight according to formula (9): and choose no greater than the mixing weight The upper bound value of is used as the mixing weight α.
[0098]
[0099] In formula (9), T represents the transpose of the matrix.
[0100] In S403, the target agent sequentially mixes the mixed weights corresponding to the reference parameters and the multiple parameters corresponding to the target agent to obtain mixed parameters.
[0101] Specifically, the target agent can obtain the mixed parameter θ by sequentially mixing each reference parameter and multiple parameters corresponding to the target agent according to formula (10): mix .
[0102]
[0103] In formula (10), represents the reference parameter obtained by the d-th exchange, α d represents the mixing weight corresponding to the reference parameter obtained by the d-th exchange, d∈[1,D].
[0104] Above Figure 4 In the method embodiment shown, valid reference parameters can be screened out by calculating whether the relative strategy advantage value meets a preset advantage threshold. For example, when the advantage threshold is 0, only reference parameters with positive relative strategy advantage values can be selected for mixing. In addition, for each reference parameter, the reference parameter is mixed with its own parameter using a mixing weight based on theoretical derivation, which can minimize the loss of learning performance compared to the method of mixing by averaging.
[0105] based on Figure 2 and Figure 4 The method embodiment shown, the embodiment of the present application provides an automatic driving method.
[0106] Figure 5 This is a flow chart of an autonomous driving method provided by an embodiment of the present application. The method can be applied to various vehicles in an autonomous driving scenario, and an intelligent agent runs in each vehicle to determine the driving speed of the vehicle according to a pre-trained strategy model.
[0107] like Figure 5 As shown, the method includes S501-S503. Figure 5 The steps shown are described in detail.
[0108] In S501, the target agent running in the target vehicle determines the state of the target vehicle at the current time. The state of the target vehicle at the current time includes the driving speed of the target vehicle at the current time, the driving speed of the surrounding vehicles corresponding to the target vehicle, and the relative distance between the target vehicle and the surrounding vehicles. The surrounding vehicles may include, for example, vehicles located in front of, behind, on the left side, and on the right side of the target vehicle.
[0109] In S502, the target agent uses a pre-trained policy network to determine the change in the target vehicle's driving speed at the next time according to the state of the target vehicle at the current time. The vehicle's driving speed at the next time is the sum of the current driving speed and the change. The change in driving speed can indicate the driving operation of the vehicle. When the change is positive, the vehicle can accelerate at the next time; when the change is negative, the vehicle can decelerate at the next time.
[0110] In this step, the target agent can Figure 2 and Figure 4 The policy model trained by the method steps shown. During the training process, the current state of the target vehicle includes the current speed of the target vehicle, the speed of the surrounding vehicles corresponding to the target vehicle, and the relative distance between the target vehicle and the reference vehicle as the current state of the target vehicle, the change in the target vehicle's speed at the next time is taken as the action of the target vehicle at the next time, and the normalized average speed (NAS) of all vehicles is taken as the reward. The specific training process of the target vehicle can be found in the above Figure 2 and Figure 4 The introduction of the target agent in the policy network training will not be repeated here. It should be noted that in the autonomous driving scenario, all reference agents corresponding to the target agent in the policy network training include agents running in all reference vehicles corresponding to the target vehicle. The reference vehicles can be surrounding vehicles and all vehicles within their communication range.
[0111] In S503, the target agent determines the driving speed of the target vehicle at the next time according to the driving speed of the target vehicle at the current time and the change in the driving speed of the target vehicle at the next time.
[0112] Figure 6 The distributed independent reinforcement learning method (IRL without communication), the distributed reinforcement learning method using average mixture, the distributed reinforcement learning method combining centralized federated learning (FL) and IRL, and Figure 2 Schematic diagram of comparison of effects of the method of the illustrated embodiment (regulated segment mixture multi agent proximal policy optimization, RSM-MAPPO) in an autonomous driving scenario.
[0113] in, Figure 6The figure specifically shows the results of the distributed independent reinforcement learning method using average weighted mixing and the method (RSM-MAPPO) based on division into 3 segments and division into 4 degrees.
[0114] according to Figure 6 It can be seen that compared with IRL without communication, the introduction of the communication process in this method (RSM-MAPPO) can strengthen the cooperation between agents, accelerate the convergence speed and learn better policy network parameters. In addition, for the MARL scenario with N agents in the system, assuming that the parameter size of the transmitted policy network is υ, the total amount of communication required for each round of communication in the fully connected communication network is N×(N-1)×υ. If the communication overhead is reduced only by reducing the number of agent communication pairs, such as Figure 7 As shown, the policy network parameters of m agents among N agents are randomly selected and mixed evenly, and the communication volume is N×m×υ. Although it is reduced by (N-1) / m times, the learning performance is significantly reduced compared with RSM-MAPPO. In addition, this method adopts the idea of segmented transmission of model parameters. For each segment request, the D2D communication transmission overhead is υ / P, then the total communication overhead of each round is N×(υ / P)×P=N×υ, which is reduced by N-1 times. On the other hand, for the case with D model copies, the communication overhead of each round becomes N×υ×D, which is still reduced by (N-1) / D times compared with the original. And at the same time, thanks to the screening of the effectiveness of the reference strategy and the determination of the mixing weight, it is possible to minimize the loss of learning performance on the basis of reducing the communication volume, and it is close to Figure 6 The centralized reinforcement learning method (Centralized FL&IRL) finally converges to the best performance.
[0115] Figure 8 The figure shows the effect of applying different numbers of model replicas to NAS using this method (RSM-MAPPO). It can be seen that the increase in D can accelerate the convergence of the training process to a certain extent, but this slight improvement comes at the cost of significantly increasing communication overhead. Therefore, in practical applications, an appropriate value of D (such as 2, 4) can be selected to balance the improvement in convergence speed and communication overhead. In addition, Figure 7 and Fig. 9 It shows that RSM-MAPPO's determination of the mixed weight of each reference strategy has a better learning effect than the average mixture that directly takes the average number of all model copies as the mixed metric. Fig. 9 As shown, dividing the model parameters into different numbers of segments (i.e. different P) leads to similar trends.
[0116] Figure 2 and Figure 4 The method embodiment shown is applied not only to the above-mentioned autonomous driving scenario, but also to the mobile communication scenario. In the mobile communication scenario, the terminal device, the base station, and the network element device of the core network can run an intelligent agent to pre-train their respective policy networks and decide the actions that they should take in the current state according to the policy network.
[0117] Taking the terminal device as an example, when the intelligent agent running the terminal device makes a base station access decision, it can use the base station to which the terminal device is currently connected as the current state of the terminal device, and use the load of the base station and the resources that the base station can allocate as rewards to decide the next base station access action.
[0118] Taking the base station as an example, when the intelligent agent running the base station makes interference management decisions, it can use the number of users to be scheduled at the base station and the channel conditions of each user as the state, and the throughput of the entire cell as the reward to decide the power allocation of the base station at the next time.
[0119] Taking network element devices as an example, when the intelligent agent running the user plane function (UPF) of the network element device performs load balancing, the traffic, number of sessions and coverage area on the UPF can be used as the state, and the congestion rate of the UPF can be used as a reward to decide the load that the UPF can currently accommodate.
[0120] based on Figure 2 and Figure 4 The method embodiment shown, the embodiment of the present application also provides a training device for a policy network.
[0121] Fig.10 1 is a schematic diagram of a training device 1000 for a policy network provided in an embodiment of the present application. The training device 1000 can be applied to a computing device to execute Figure 2 and Figure 4 The method steps shown. In different application scenarios, the computing device may include different devices. For example, in an autonomous driving scenario, the computing device may include a vehicle-mounted computer installed in a vehicle; in a mobile communication scenario, the computing device may include the above-mentioned terminal device, base station or network element device, etc.
[0122] like Fig.10 As shown, the training device 1000 may include a switching module 1001 , a mixing module 1002 and a training module 1003 .
[0123] Among them, the exchange module 1001 is used to divide multiple parameters of the strategy network of the target intelligent agent into P segments according to a preset method, and send exchange requests to the reference intelligent agent corresponding to each segment. The multiple parameters of the strategy network of the reference intelligent agent are divided into the same number of segments according to the preset method, and the exchange requests corresponding to each segment are used to obtain the parameters of the same segment of the strategy network of the reference intelligent agent corresponding to each segment.
[0124] Among them, the mixing module 1002 is used to mix the reference parameters and multiple parameters of the strategy network of the target agent to obtain mixed parameters, and the reference parameters include the parameters of the same segment of the strategy network of the reference agent corresponding to each segment.
[0125] Among them, the training module 1003 is used to update multiple parameters of the strategy network of the target intelligent agent according to the mixed parameters.
[0126] It should be noted that Fig.10 The training device 1000 provided in the illustrated embodiment only uses the division of the above-mentioned functional modules as an example when executing the training method of the policy network. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Figure 2 The training method embodiment of the strategy network shown belongs to the same concept, and its specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0127] Fig.11 1 is a schematic diagram of the hardware structure of a computing device 1100 provided in an embodiment of the present application. The computing device 1100 may be the above-mentioned computing device. In different application scenarios, the computing device may include different devices. For example, in an autonomous driving scenario, the computing device may include a vehicle-mounted computer installed in a vehicle; in a mobile communication scenario, the computing device may include the above-mentioned terminal device, base station or network element device, etc.
[0128] See also Fig.11 The computing device 1100 includes a processor 1101, a memory 1102, a communication interface 1103, and a bus 1104. The processor 1101, the memory 1102, and the communication interface 1103 are connected to each other via the bus 1104. The processor 1101, the memory 1102, and the communication interface 1103 may also be connected in other connection modes besides the bus 1104.
[0129] Among them, the memory 1102 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, hard disk, etc.
[0130] The processor 1101 may be a general-purpose processor, which may be a processor that performs specific steps and / or operations by reading and executing the contents stored in a memory (e.g., memory 1102). For example, the general-purpose processor may be a central processing unit (CPU). The processor 1101 may include at least one circuit to execute Figure 2 or Figure 4 The illustrated embodiment provides all or part of the steps of the policy network training method.
[0131] The communication interface 1103 includes an input / output (I / O) interface, a physical interface, and a logical interface, etc., which are used to interconnect devices within the computing device 1100, and an interface for interconnecting the computing device 1100 with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.
[0132] The bus 1104 may be any type of communication bus for interconnecting the processor 1101 , the memory 1102 , and the communication interface 1103 , such as a system bus.
[0133] The above devices may be arranged on independent chips, or at least partially or completely on the same chip. Whether to arrange each device independently on different chips or to integrate them on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above devices.
[0134] Fig.11 The computing device 1100 shown is merely exemplary. During implementation, the computing device 1100 may also include other components, which are not listed one by one herein.
[0135] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0136] It is understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. It should be understood that in the embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0137] The specific implementation method described above further explains the purpose, technical solution and beneficial effects of the present application in detail. It should be understood that the above is only a specific implementation method of the present invention and is not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present application should be included in the protection scope of the present application.
Claims
1. A training method for a policy network, characterized in that: The method comprises: Divide multiple parameters of the policy network of the target intelligent agent into multiple segments according to a preset method; Sending an exchange request to a reference agent corresponding to each segment, wherein multiple parameters of the strategy network of the reference agent are divided into the same number of segments according to the preset method, and the exchange request corresponding to each segment is used to obtain the parameters of the same segment of the strategy network of the reference agent corresponding to each segment; Mixing reference parameters and multiple parameters of the policy network of the target agent to obtain mixed parameters, wherein the reference parameters include parameters of the same segment of the policy network of the reference agent corresponding to each segment; Multiple parameters of the policy network of the target agent are updated according to the mixed parameters.
2. The method according to claim 1, characterized in that: The mixing of the reference parameters and the multiple parameters of the strategy network of the target agent to obtain the mixed parameters includes: Determine the mixing weight corresponding to the reference parameter: The reference parameters and multiple parameters of the policy network of the target agent are mixed according to the mixing weights.
3. The method according to claim 2, characterized in that Before mixing the reference parameters with the multiple parameters of the policy network of the target agent to obtain the mixed parameters, the method further includes: If the relative advantage value corresponding to the reference parameter is greater than the advantage threshold, determining to mix based on the reference parameter; If the relative advantage value is less than or equal to the advantage threshold, it is determined not to mix based on the reference parameter.
4. The method according to claim 3, characterized in that Determining the mixing weight corresponding to the reference parameter includes: Determining an upper bound of the hybrid weight according to the relative advantage value corresponding to the reference parameter and the Fisher information matrix corresponding to the target agent; The mixing weight is determined according to an upper bound of the mixing weight.
5. The method according to claim 4, characterized in that The method further comprises: The Fisher information matrix is determined according to the multiple parameters corresponding to the target intelligent agent and the multiple historical parameters obtained from the previous training corresponding to the target intelligent agent.
6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: The relative advantage value is determined according to the reference parameter and the multiple parameters corresponding to the target agent.
7. The method according to any one of claims 1 to 6, characterized in that: The policy network includes a behavior policy network and a target policy network. Before dividing the multiple parameters of the policy network of the target agent into multiple segments, the method further includes: The behavior strategy network determines training samples according to the initial parameters and state space corresponding to the target intelligent agent; The value network determines the advantage value corresponding to the initial parameter according to the training sample; The target policy network determines multiple parameters of the policy network of the target agent according to the training samples and the advantage value.
8. The method according to any one of claims 1 to 5, characterized in that: The target agent runs on the target vehicle, the reference agent runs on the reference vehicle of the target vehicle, the strategy network of the target vehicle is used to determine the driving speed of the target vehicle at the next time according to the current driving speed of the target vehicle, the current driving speed of the surrounding vehicles corresponding to the target vehicle, and the relative distance between the target vehicle and the surrounding vehicles, and the reference vehicle includes the surrounding vehicles.
9. A training device for a policy network, characterized in that: The device comprises: An exchange module, used to divide multiple parameters of the policy network of the target agent into multiple segments according to a preset method, and send exchange requests to the reference agent corresponding to each segment, wherein the multiple parameters of the policy network of the reference agent are divided into the same number of segments according to the preset method, and the exchange requests corresponding to each segment are used to obtain the parameters of the same segment of the policy network of the reference agent corresponding to each segment; A mixing module, used for mixing reference parameters and multiple parameters of the policy network of the target agent to obtain mixed parameters, wherein the reference parameters include parameters of the same segment of the policy network of the reference agent corresponding to each segment; A training module is used to update multiple parameters of the policy network of the target agent according to the mixed parameters.
10. A computing device, characterized in that The computing device comprises: a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 8.