Model determination method and device based on reinforcement learning, equipment, medium and product

Abstract

The invention relates to the technical field of artificial intelligence and intelligent traffic, and discloses a model determination method and device based on reinforcement learning, equipment, a medium and a product, and the method comprises the steps: obtaining training data; inputting the training data into a first network of the initial model to obtain a first action; interacting the first action with the current environment to obtain a next state; inputting the current state, the first action and the next state into a second network of the initial model to obtain a first value; and according to the first value and a target loss function, adjusting model parameters of the initial model to obtain a target model, the target loss function being jointly determined based on a strategy gradient loss function and a value constraint loss function. According to the invention, the training cost and computing power demand of the traffic light control strategy network can be reduced based on the teaching data dynamic constraint reinforcement learning algorithm, the convergence and stability of the traffic light control strategy network are improved, and the cross-scene generalization ability of the traffic light control method is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence and intelligent transportation technology, and in particular to a model determination method, device, equipment, medium and product based on reinforcement learning. Background Art

[0002] With the rapid development of the economy, people's living standards are gradually improving, and residents' willingness to travel has increased significantly. The number of motor vehicles has shown explosive growth as the demand for travel increases, making travel more convenient and flexible. The rapid development of transportation has not only promoted social progress and improved people's lives, but also promoted scientific and technological progress. However, the existing urban transportation resources cannot meet the transportation needs well. The explosive growth of motor vehicles has aggravated the imbalance between supply and demand of urban transportation, leading to increasingly serious traffic congestion problems. At the same time, the exhaust gas generated by traffic congestion has also aggravated environmental pollution.

[0003] To address the above issues, related technologies usually adopt the following three methods: (1) timed traffic light control method; (2) traffic light control method based on non-learning control algorithm; (3) traffic light control method based on reinforcement learning. At present, the industry generally believes that the traffic light timing method based on reinforcement learning is superior to other traditional methods in reducing vehicle delay time and improving traffic efficiency. However, it still has some problems, mainly reflected in: (1) reinforcement learning requires a large amount of training data and training time to learn to adapt to complex environments; (2) reinforcement learning requires a large amount of computing resources to process complex states and action spaces, which places high demands on hardware and computing power; (3) during the training process, exploration and trial and error may cause the behavior of the intelligent agent to be unstable, and convergence problems may occur, which increases the uncertainty of the algorithm.

[0004] Therefore, the traffic light control method based on reinforcement learning in the related art has the problems of high training cost, high computing power requirement, and poor training convergence and stability. Summary of the invention

[0005] In view of this, the present disclosure provides a model determination method, device, equipment, medium and product based on reinforcement learning to solve the problems of high training cost, poor training convergence and stability, and high computing power requirement in the traffic light control method based on reinforcement learning in the related art.

[0006] In a first aspect, the present disclosure provides a model determination method based on reinforcement learning, the method comprising:

[0007] Get training data;

[0008] Input the training data into the first network of the initial model to obtain the first action;

[0009] Interact the first action with the current environment to obtain the next state;

[0010] Input the current state, the first action, and the next state into the second network of the initial model to obtain the first value;

[0011] The model parameters of the initial model are adjusted according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function.

[0012] In the disclosed embodiment, training data is obtained; the training data is input into the first network of the initial model to obtain the first action; the first action is interacted with the current environment to obtain the next state; the current state, the first action, and the next state are input into the second network of the initial model to obtain the first value; the model parameters of the initial model are adjusted according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function. Since the disclosed embodiment obtains the first action based on the first network, obtains the first value based on the current state, the first action, the next state, and the second network, and then jointly determines the target loss function based on the first value, the policy gradient loss function, and the value constraint loss function, adjusts the model parameters, and realizes dynamic constraint of the first network based on the training data, reduces the training cost and computing power requirements of the first network, and improves the convergence, stability, and cross-scenario generalization ability of the first network.

[0013] In an optional implementation, obtaining training data includes:

[0014] Acquire teaching data, wherein the teaching data is reference data when the initial model stops adjusting the model parameters during the training process, and the teaching data is included in the training data;

[0015] Loading teaching data into the first buffer;

[0016] Get the real-time data after the interaction with the current environment. The real-time data is included in the training data.

[0017] Load real-time data into the second buffer.

[0018] In the disclosed embodiment, teaching data is introduced during the training process of the first network to dynamically constrain the training process, thereby improving the convergence and stability of the first network. At the same time, since teaching data is added during the training process of the first network, the first network does not need a large amount of training data and training time to adapt to the complex environment, thereby reducing the training cost and computing power requirements of the first network.

[0019] In an optional implementation, inputting the training data into the first network of the initial model to obtain the first action includes:

[0020] Acquire a first preset number of first sampling data from the first buffer, and acquire a second preset number of second sampling data from the second buffer;

[0021] The first sampling data and the second sampling data are input into the first network to obtain a first action.

[0022] In the disclosed embodiment, the first sampling data is selected from the first buffer mainly for constraining the first network training process so that it maintains good stability; the second sampling data is selected from the second buffer mainly for improving the generalization ability of the first network in unseen scenes, solving the problem that the teaching data cannot cover all scenes, and making the first network applicable to general scenes.

[0023] In an optional implementation, before adjusting the model parameters of the initial model according to the first value and the target loss function to obtain the target model, the method further includes:

[0024] A second value is obtained according to the teaching data and the second network;

[0025] Compare the first value and the second value to determine a value constraint loss function;

[0026] The target loss function is determined based on the policy gradient loss function and the value constraint loss function.

[0027] In the disclosed embodiment, by comparing the first value and the second value and then determining the value constraint loss function, stable strategy improvement can be achieved while preventing catastrophic forgetting of the pre-training strategy.

[0028] In an optional implementation, comparing the first value and the second value to determine the value constraint loss function includes:

[0029] When the first value is greater than the second value, determining a first value constraint loss function, and using the first value constraint loss function as the value constraint loss function;

[0030] When the first value is less than or equal to the second value, a second value constraint loss function is determined and used as the value constraint loss function.

[0031] In the disclosed embodiment, by judging the comparison result between the first value and the second value, it is possible to judge the pros and cons of the first action compared with the teaching data, set the corresponding value constraint loss function, and guide the first network training process, thereby preventing catastrophic forgetting of the pre-training strategy and achieving stable strategy improvement.

[0032] In an optional implementation, determining the target loss function according to the policy gradient loss function and the value constraint loss function includes:

[0033] Get the sum of the policy gradient loss function and the value constraint loss function to determine the target loss function.

[0034] In the disclosed embodiment, by adding a value constraint loss function to the objective loss function, the convergence and stability of the first network training are improved, and the training convergence and training efficiency of the first network can be balanced.

[0035] In a second aspect, the present disclosure provides a model determination device based on reinforcement learning, the device comprising:

[0036] A first acquisition module, used to acquire training data;

[0037] A first obtaining module, used for inputting the training data into the first network of the initial model to obtain a first action;

[0038] The second obtaining module is used to interact the first action with the current environment to obtain the next state;

[0039] A third obtaining module is used to input the current state, the first action, and the next state into the second network of the initial model to obtain the first value;

[0040] The fourth obtaining module is used to adjust the model parameters of the initial model according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function.

[0041] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the model determination method based on reinforcement learning of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0042] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the reinforcement learning-based model determination method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0043] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the reinforcement learning-based model determination method of the above-mentioned first aspect or any corresponding embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flow chart of a model determination method based on reinforcement learning according to an embodiment of the present disclosure;

[0046] Figure 2 is a flow chart of another reinforcement learning-based model determination method according to an embodiment of the present disclosure;

[0047] Figure 3 is a network framework diagram of a model determination method based on reinforcement learning according to an embodiment of the present disclosure;

[0048] Figure 4 is a structural block diagram of a model determination device based on reinforcement learning according to an embodiment of the present disclosure;

[0049] Figure 5 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0051] With the rapid development of the economy, people's living standards are gradually improving, and residents' willingness to travel has increased significantly. The number of motor vehicles has shown explosive growth as the demand for travel increases, making travel more convenient and flexible. The rapid development of transportation has not only promoted social progress and improved people's lives, but also promoted scientific and technological progress. However, the existing urban transportation resources cannot meet the transportation needs well. The explosive growth of motor vehicles has aggravated the imbalance between supply and demand of urban transportation, leading to increasingly serious traffic congestion problems. At the same time, the exhaust gas generated by traffic congestion has also aggravated environmental pollution.

[0052] However, the traffic light control method based on reinforcement learning in related technologies has the problems of high training cost, high computing power requirement, and poor training convergence and stability. Specifically, (1) reinforcement learning requires a large amount of training data and training time to learn and adapt to complex environments; (2) reinforcement learning requires a large amount of computing resources to process complex states and action spaces, which places high demands on hardware and computing power; (3) during the training process, exploration and trial and error may cause the behavior of the intelligent agent to be unstable, and convergence problems may occur, which increases the uncertainty of the algorithm.

[0053] In order to solve the above problems, according to an embodiment of the present disclosure, a model determination method embodiment based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0054] In this embodiment, a model determination method based on reinforcement learning is provided, such as Figure 1 As shown, Figure 1 : is a flow chart of a model determination method based on reinforcement learning according to an embodiment of the present disclosure. The flow chart can be applied to a server and includes the following steps:

[0055] Step S101, obtaining training data.

[0056] Step S102: input the training data into the first network of the initial model to obtain the first action.

[0057] Alternatively, if Figure 3 As shown, in the embodiment of the present disclosure, the initial model includes a first network. The first network is a strategy network π θ The first action is when the server inputs the training data into the policy network π of the initial model θ After that, the action π is calculated θ (s t ).

[0058] Specifically, the server first uses imitation learning to train the traffic light control strategy, and then transfers the trained strategy network to obtain the initial model of reinforcement learning.

[0059] It should be noted that the set of all possible actions in the environment is the action space. In this embodiment, the action space includes all possible forms of traffic lights. There are usually four types of actions: setting the duration of the current phase, setting the phase ratio, continuing or switching the phase, and selecting any phase as the next phase. Since continuing or switching the phase has better versatility, this embodiment selects this action as the action representation.

[0060] Step S103, the first action interacts with the current environment to obtain a next state.

[0061] Alternatively, if Figure 3 As shown, in the embodiment of the present disclosure, the next state means that the server will take action π θ (s t ) After interacting with the environment, the next intersection state s is obtained t+1 .

[0062] Specifically, the set of all possible states in the environment is the state space. In this embodiment, the state space is an abstract description of the current real-scene traffic conditions. The state is represented by a combination of the following information: vehicle queue length, vehicle waiting time, vehicle flow, vehicle speed, current phase duration, and congestion level.

[0063] Step S104, input the current state, the first action, and the next state into the second network of the initial model to obtain the first value.

[0064] Alternatively, reinforcement learning (RL) is a machine learning method that focuses on how agents should act in the environment to maximize the cumulative rewards obtained. This learning method is similar to the behaviorism theory in psychology, that is, humans will adjust their behavior according to the effect of the behavior (the feedback of the environment on the behavior - reward or punishment) to adapt to the environment and obtain the maximum value.

[0065] Specifically, Figure 3 As shown in the figure, reinforcement learning adopts the Actor-Critic algorithm framework. The Actor-Critic algorithm is a reinforcement learning method that combines policy gradient and temporal difference learning. It consists of two parts: Actor and Critic. In the traffic light control system, the Actor corresponds to the policy network, which is used to generate actions and interact with the environment. The policy network is trained to get the highest possible reward. The Critic corresponds to the evaluation function, which can evaluate the performance of the Actor and guide the Actor to perform the next stage of action. Figure 3 As shown, in the embodiment of the present disclosure, the current state is state s tThe initial model includes not only the first network mentioned in the above embodiment, but also a second network. The second network is the evaluation network Q φ The first value is the server status s t 、Action π θ (s t ) and state s t+1 Input evaluation network Q φ After that, the value Q is calculated.

[0066] More specifically, value is the expectation of return. Return is the accumulation of reward value starting from a certain state at a certain moment to the final state at the final moment. The reward value is a specific value corresponding to the change in the environment, which is used to feedback the quality of the action taken by the traffic light and can be calculated by the reward function. The reward function is a function of the state space and the action space. The reward function is defined as follows:

[0067] r=w1·waiting+w2·time+w3·capacity+w4·speed+w5·pressure

[0068] Among them, waiting is the number of vehicles in the waiting queue at the intersection, time is the average waiting time of vehicles in the waiting queue, capacity is the vehicle throughput of the intersection per unit time, speed is the average speed of vehicles at the intersection, and pressure is the number of vehicles entering the intersection area minus the number of vehicles leaving the intersection area. In this embodiment, waiting, time, capacity, speed, and pressure are normalized, that is, they are divided by the maximum value that has appeared in the intersection area. w1, w2, w3, w4, and w5 are weights, and the weights can be adjusted according to actual needs. In general, w1 and w2 are set to 5, and w3, w4, and w5 are set to 1.

[0069] It should be noted that the task of determining the reward function is policy optimization, and the goal is to maximize the expected discounted return. The policy refers to the rule for selecting actions in a given state, which determines how the traffic light control system acts in order to obtain the maximum reward value in the interaction with the environment. Policy optimization is defined as follows:

[0070]

[0071] Among them, s t is the state at time t, r k is the reward at time t, γ is the discount factor, T is the maximum time step, that is, the number of steps at the end of the round, π is the strategy, and π * The optimal strategy is obtained by solving the above optimization equation.

[0072] It needs to be explained that the evaluation network Q φ The parameter φ of is updated using the time difference error.

[0073] Step S105, adjusting the model parameters of the initial model according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function.

[0074] Optionally, the objective loss function J A (θ) is the policy network π θ The target loss function J A (θ) includes the policy gradient loss function J PG (θ) and the value constraint loss function J D (θ). Policy network π θ The parameters θ are used with the policy gradient loss function J PG (θ) and the value constraint loss function J D (θ) is updated.

[0075] In the disclosed embodiment, training data is obtained; the training data is input into the first network of the initial model to obtain the first action; the first action is interacted with the current environment to obtain the next state; the current state, the first action, and the next state are input into the second network of the initial model to obtain the first value; the model parameters of the initial model are adjusted according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function. Since the disclosed embodiment obtains the first action based on the first network, obtains the first value based on the current state, the first action, the next state, and the second network, and then jointly determines the target loss function based on the first value, the policy gradient loss function, and the value constraint loss function, adjusts the model parameters, and realizes dynamic constraint of the first network based on the training data, reduces the training cost and computing power requirements of the first network, and improves the convergence, stability, and cross-scenario generalization ability of the first network.

[0076] In some optional implementations, obtaining training data includes:

[0077] Acquire teaching data, wherein the teaching data is reference data when the initial model stops adjusting the model parameters during the training process, and the teaching data is included in the training data;

[0078] Loading teaching data into the first buffer;

[0079] Get the real-time data after the interaction with the current environment. The real-time data is included in the training data.

[0080] Load real-time data into the second buffer.

[0081] Optionally, in this embodiment, the training data includes real-time data obtained after the teaching data and the current environment interact. The teaching data refers to a series of state-action pairs, including representative cases at multiple intersections and different traffic conditions. In this embodiment, the server collected teaching data of traffic police manually controlling traffic lights at 20 intersections with a total duration of about 30 hours, and stored these data in the form of state-action pairs. Real-time data is the state data obtained after the server interacts with the current environment.

[0082] In addition, the buffer includes a first buffer and a second buffer. The first buffer is D d , the second buffer is D a .

[0083] Specifically, the server preloads the teaching data into the first buffer D d In the process, the real-time data obtained after the interaction with the current environment is saved in the second buffer D a In the first buffer D d It remains unchanged during reinforcement learning training.

[0084] It should be noted that before obtaining the initial model, the server uses behavioral cloning to train the traffic light control policy network. Behavioral cloning is a method of imitation learning, which is mainly used to learn policies from the teaching data of human experts. Its core idea is to imitate human actions through supervised learning, so as to train a policy network so that its decision in a given state is as close as possible to that of human experts. In the imitation learning stage, the server uses the teaching data as training data and uses supervised learning methods to train the policy network. For each state, the policy network outputs an action. The difference between the action output by the policy network and the expert action is measured by defining a loss function. The loss function is minimized using optimization algorithms such as gradient descent, so as to continuously adjust the parameters of the policy network so that the action it outputs is closer and closer to the expert action.

[0085] More specifically, the server utilizes N s teaching data, each data is represented as a sequence of state-action pairs, denoted as Where T i is the length of the i-th sequence. The mapping from state data to teaching actions is learned in a supervised learning manner, and the loss function L BC The definition is as follows:

[0086]

[0087] In this embodiment, teaching data is introduced during the training process of the first network to dynamically constrain the training process, thereby improving the convergence and stability of the first network. At the same time, since teaching data is added during the training process of the first network, the first network does not need a large amount of training data and training time to adapt to the complex environment, thereby reducing the training cost and computing power requirements of the first network.

[0088] In some optional implementations, inputting the training data into the first network of the initial model to obtain the first action includes:

[0089] Acquire a first preset number of first sampling data from the first buffer, and acquire a second preset number of second sampling data from the second buffer;

[0090] The first sampling data and the second sampling data are input into the first network to obtain a first action.

[0091] Optionally, in this embodiment, the sampled data includes first sampled data and second sampled data. d Get the first sample data from the second buffer D a The second sampling data is obtained from the teaching data, that is, the first sampling data comes from the teaching data, and the second sampling data comes from the real-time data. The preset quantity includes a first preset quantity and a second preset quantity. Among them, the preset quantity is B, and the first preset quantity is B d .

[0092] Specifically, for each optimization update of the network, the server randomly obtains B sample data from the buffer. d The sample data comes from the first buffer D d The remaining sampled data comes from the second buffer D a , where B d =ρB. The hyperparameter ρ is used to control the amount of teaching data in the sampled data, and the preferred value selected in this embodiment is ρ=0.5.

[0093] In this embodiment, the first sampling data is selected from the first buffer mainly to constrain the first network training process so that it maintains good stability; the second sampling data is selected from the second buffer mainly to improve the generalization ability of the first network in unseen scenes, solve the problem that the teaching data cannot cover all scenes, and enable the first network to be applicable to general scenes.

[0094] In some optional implementations, before adjusting the model parameters of the initial model according to the first value and the target loss function to obtain the target model, the method further includes:

[0095] A second value is obtained according to the teaching data and the second network;

[0096] Compare the first value and the second value to determine a value constraint loss function;

[0097] The target loss function is determined based on the policy gradient loss function and the value constraint loss function.

[0098] Optionally, in this embodiment, the action includes not only the first action mentioned in the above embodiment, but also a second action. The second action refers to the action in the teaching data. The value includes not only the first value mentioned in the above embodiment, but also a second value. The second value refers to the value of the server inputting the teaching data into the evaluation network Q φ Then, calculate the value.

[0099] Specifically, the server can dynamically determine the relationship between the teaching data action and the action π generated by the current strategy network by comparing the size relationship between the first value Q and the second value θ (s t ) and select the one with higher value for network training, and then determine the value constraint loss function J D (θ).

[0100] In this embodiment, by comparing the first value and the second value and then determining the value constraint loss function, stable strategy improvement can be achieved while preventing catastrophic forgetting of the pre-training strategy.

[0101] In some optional implementations, comparing the first value and the second value to determine the value constraint loss function includes:

[0102] When the first value is greater than the second value, determining a first value constraint loss function, and using the first value constraint loss function as the value constraint loss function;

[0103] When the first value is less than or equal to the second value, a second value constraint loss function is determined and used as the value constraint loss function.

[0104] Optionally, in this embodiment, the value constraint loss function J D (θ) includes the first value constraint loss function and the second value constraint loss function. When the first value Q is greater than the second value, the server uses the first value constraint loss function as the value constraint loss function J D (θ); When the first value Q is less than or equal to the second value, the server uses the second value constraint loss function as the value constraint loss function J D (θ).

[0105] Specifically, the value constraint loss function J D (θ) is defined as follows:

[0106]

[0107] in, is an indicative function. When the first value Q is greater than the second value, it indicates that the evaluation network Q φ Evaluate the action π generated by the current policy network θ (s t ) is better than the teaching data action, and the value of the characteristic function is set to 0; when the first value Q is less than or equal to the second value, it means that the evaluation network Q φ Evaluate the action π generated by the current policy network θ (s t ) Compared with the teaching data action, there is no better result, and the value of the characteristic function is set to 1.

[0108] In this embodiment, by judging the comparison result between the first value and the second value, it is possible to judge the pros and cons of the first action compared with the teaching data, set the corresponding value constraint loss function, and guide the first network training process, thereby preventing catastrophic forgetting of the pre-training strategy and achieving stable strategy improvement.

[0109] In some optional implementations, determining the target loss function according to the policy gradient loss function and the value constraint loss function includes:

[0110] Get the sum of the policy gradient loss function and the value constraint loss function to determine the target loss function.

[0111] Optionally, in this embodiment, the server obtains the policy gradient loss function J PG (θ) and the value constraint loss function J D The sum of (θ) can determine the target loss function J A (θ).

[0112] Specifically, the objective loss function J A (θ) is the policy gradient loss function J PG (θ) and the value constraint loss function J D The objective loss function J is a linear combination of (θ). A (θ) is defined as follows:

[0113] J A (θ)=λJ PG (θ)+J D (θ)

[0114] Among them, λ is a hyperparameter used to balance the policy gradient loss function J PG (θ) and the value constraint loss function J D (θ). In order to overcome the random initialization of the evaluation network Q φIn the early learning process, the server uses a warm-up phase to evaluate the network Q φ Perform several rounds of pre-training. In the warm-up phase, set λ = 0, that is, the target loss function J A (θ) only uses the value constraint loss function J D (θ) is preheated; when the evaluation network Q φ After warming up, set λ>0, that is, the policy gradient loss function J PG (θ) is added to the objective loss function J A (θ), used to optimize the policy network π θ Experiments show that a larger λ will make the training unstable, and a smaller λ will reduce the optimization efficiency. Therefore, in order to balance the convergence and efficiency of training, the preferred value selected in this embodiment is λ=0.1.

[0115] In this embodiment, by adding the value constraint loss function to the objective loss function, the convergence and stability of the first network training are improved, and the training convergence and training efficiency of the first network can be balanced.

[0116] In some optional embodiments, such as Figure 2 As shown, Figure 2 It is a flow chart of another reinforcement learning-based model determination method according to an embodiment of the present disclosure. First, the server obtains the teaching data of human experts through data collection. Next, the server uses imitation learning to learn the traffic light control strategy network (i.e., the strategy network), wherein the teaching data provides supervision information for the training process. Then, the server performs strategy migration on the trained strategy network to obtain an initial model with reinforcement learning added. Finally, the server uses reinforcement learning to optimize the traffic light control strategy network, wherein the teaching data provides knowledge guidance for the training process.

[0117] In this embodiment, a model determination device based on reinforcement learning is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0118] This embodiment provides a model determination device based on reinforcement learning, such as Figure 4 As shown, including:

[0119] A first acquisition module 401 is used to acquire training data;

[0120] A first obtaining module 402 is used to input the training data into the first network of the initial model to obtain a first action;

[0121] The second obtaining module 403 is used to interact the first action with the current environment to obtain the next state;

[0122] A third obtaining module 404 is used to input the current state, the first action, and the next state into the second network of the initial model to obtain a first value;

[0123] The fourth obtaining module 405 is used to adjust the model parameters of the initial model according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function.

[0124] In the disclosed embodiment, training data is obtained; the training data is input into the first network of the initial model to obtain the first action; the first action is interacted with the current environment to obtain the next state; the current state, the first action, and the next state are input into the second network of the initial model to obtain the first value; the model parameters of the initial model are adjusted according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function. Since the disclosed embodiment obtains the first action based on the first network, obtains the first value based on the current state, the first action, the next state, and the second network, and then jointly determines the target loss function based on the first value, the policy gradient loss function, and the value constraint loss function, adjusts the model parameters, and realizes dynamic constraint of the first network based on the training data, reduces the training cost and computing power requirements of the first network, and improves the convergence, stability, and cross-scenario generalization ability of the first network.

[0125] In some optional implementations, the first acquisition module 401 includes:

[0126] A first acquisition unit is used to acquire teaching data, wherein the teaching data is reference data when the initial model stops adjusting the model parameters during the training process, and the teaching data is included in the training data;

[0127] A first loading unit, used for loading teaching data into a first buffer;

[0128] A second acquisition unit is used to acquire real-time data obtained after the interaction with the current environment, where the real-time data is included in the training data;

[0129] The second loading unit is used to load the real-time data into the second buffer.

[0130] In some optional embodiments, the first loading unit and the second loading unit include:

[0131] A first acquisition submodule, configured to acquire a first preset number of first sampling data from the first buffer, and acquire a second preset number of second sampling data from the second buffer;

[0132] The first obtaining submodule is used to input the first sampling data and the second sampling data into the first network to obtain the first action.

[0133] In some optional embodiments, the device further comprises:

[0134] A fifth obtaining module is used to obtain a second value according to the teaching data and the second network before adjusting the model parameters of the initial model according to the first value and the target loss function to obtain the target model;

[0135] A first determination module, used for comparing the first value and the second value to determine a value constraint loss function;

[0136] The second determination module is used to determine the target loss function according to the policy gradient loss function and the value constraint loss function.

[0137] In some optional implementations, the first determining module includes:

[0138] A first determining unit, configured to determine a first value constraint loss function when the first value is greater than the second value, and use the first value constraint loss function as the value constraint loss function;

[0139] The second determining unit is used to determine a second value constraint loss function when the first value is less than or equal to the second value, and use the second value constraint loss function as the value constraint loss function.

[0140] In some optional implementations, the second determining module includes:

[0141] The third determination unit is used to obtain the sum of the policy gradient loss function and the value constraint loss function to determine the target loss function.

[0142] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0143] The reinforcement learning-based model determination device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0144] The present disclosure also provides a computer device having the above Figure 4The model determination device based on reinforcement learning is shown.

[0145] See also Figure 5 , Figure 5 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0146] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0147] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0148] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0150] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0151] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0152] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0153] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A model determination method based on reinforcement learning, characterized in that: The method comprises: Get training data; Inputting the training data into a first network of an initial model to obtain a first action; The first action interacts with the current environment to obtain the next state; Inputting the current state, the first action, and the next state into the second network of the initial model to obtain a first value; The model parameters of the initial model are adjusted according to the first value and the target loss function to obtain a target model, wherein the target loss function is determined based on a policy gradient loss function and a value constraint loss function.

2. The method according to claim 1, characterized in that The obtaining of training data comprises: Acquire teaching data, wherein the teaching data is reference data when the initial model stops adjusting the model parameters during the training process, and the teaching data is included in the training data; Loading the teaching data into a first buffer; Acquire real-time data obtained after interaction with the current environment, wherein the real-time data is included in the training data; The real-time data is loaded into a second buffer.

3. The method according to claim 2, characterized in that The step of inputting the training data into a first network of an initial model to obtain a first action comprises: Acquire a first preset number of first sampling data from the first buffer, and acquire a second preset number of second sampling data from the second buffer; The first sampling data and the second sampling data are input into the first network to obtain the first action.

4. The method according to claim 2, characterized in that: Before adjusting the model parameters of the initial model according to the first value and the target loss function to obtain the target model, the method further includes: Obtaining a second value according to the teaching data and the second network; comparing the first value and the second value to determine the value constraint loss function; The target loss function is determined according to the policy gradient loss function and the value constraint loss function.

5. The method according to claim 4, characterized in that The comparing the first value and the second value to determine the value constraint loss function includes: In a case where the first value is greater than the second value, determining a first value constraint loss function, and using the first value constraint loss function as the value constraint loss function; When the first value is less than or equal to the second value, a second value constraint loss function is determined, and the second value constraint loss function is used as the value constraint loss function.

6. The method according to claim 4, characterized in that The determining the target loss function according to the policy gradient loss function and the value constraint loss function includes: The sum of the policy gradient loss function and the value constraint loss function is obtained to determine the target loss function.

7. A model determination device based on reinforcement learning, characterized in that: The device comprises: A first acquisition module, used to acquire training data; A first obtaining module, used for inputting the training data into a first network of an initial model to obtain a first action; A second obtaining module is used to interact the first action with the current environment to obtain a next state; A third obtaining module is used to input the current state, the first action, and the next state into the second network of the initial model to obtain a first value; The fourth obtaining module is used to adjust the model parameters of the initial model according to the first value and the target loss function to obtain the target model, wherein the target loss function is determined based on the policy gradient loss function and the value constraint loss function.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the reinforcement learning-based model determination method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the reinforcement learning-based model determination method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the reinforcement learning-based model determination method according to any one of claims 1 to 6.

Images