Intelligent Vehicle Intersection Decision-making Method, Device and Medium Based on Meta-Reinforcement Learning

Abstract

The present application discloses an intelligent vehicle intersection decision-making method, device and medium based on meta-reinforcement learning, aiming to solve the technical problems that the existing vehicle behavior decision-making methods do not consider the complexity and uncertainty of intersection scenarios and have poor adaptability. The method first determines the first observation states of each environmental vehicle corresponding to the target vehicle in each first intersection scenario; then obtains the second observation states, random actions and reward values corresponding to each first observation state; retrains the pre-trained decision model to obtain a time-series-based decision model; determines the third observation states of each environmental vehicle of the target vehicle in each second intersection scenario; respectively determines the descending gradients corresponding to each second intersection scenario to determine the corresponding meta-goals; determines the sensitive parameters of the time-series-based decision model according to the meta-goals; and updates the time-series-based decision model.

Description

Technical Field

[0001] This application relates to the technical field of intelligent vehicles, and in particular, to an intelligent vehicle intersection decision-making method, device, and medium based on meta-reinforcement learning. Background Art

[0002] Intelligent vehicles have great potential in terms of safety, efficiency, comfort, etc., and have gradually become the core of future transportation. However, to achieve autonomous driving in a high-density and mixed traffic flow environment, the behavior decision-making ability of intelligent vehicles still faces severe challenges.

[0003] There are mainly three existing decision-making methods. One is rule-based behavior decision-making, the second is behavior decision-making based on probability models, and the third is a decision-making model based on learning. Although these decision-making systems have the advantages of strong execution and simplicity, they ignore the complexity and uncertainty of dynamic traffic factors in the environment. In the case of turning left at an intersection, the research on vehicle behavior decision-making ability depends on a fixed turning path. However, in an actual intersection scenario, the turning path of a vehicle will change according to the surrounding traffic flow speed, traffic volume, and driver style, with a high degree of uncertainty. Moreover, although the existing learning-based decision-making models consider the complexity and uncertainty of dynamic traffic factors in the environment, they cannot meet the real-time and self-adaptive requirements in complex environments. For example, an algorithm that converges well during training at a certain intersection performs poorly at other intersections, or when the number and motion state of vehicles within the same intersection change, the decision-making model performs poorly and cannot give a relatively reasonable decision. Summary of the Invention

[0004] Embodiments of this application provide an intelligent vehicle intersection decision-making method, device, and medium based on meta-reinforcement learning, to solve the technical problem that the existing vehicle behavior decision-making methods do not consider the complexity and uncertainty of intersection scenarios, have poor self-adaptability, and are prone to the situation where the behavior strategies made for the current scenario are no longer applicable when the scenario changes.

[0005] The embodiments of the present application provide an intelligent vehicle intersection decision-making method based on meta-reinforcement learning. The method includes: determining the first observation states of the environmental vehicles corresponding to the target vehicle in each first intersection scenario; based on a pre-trained decision-making model, obtaining the second observation states, random actions, and reward values corresponding to each of the first observation states; wherein, the second observation state is the observation state corresponding to each environmental vehicle at the next moment in the first observation state; using the second observation states, random actions, reward values corresponding to each of the first observation states, and each of the first observation states as first sample data to retrain the pre-trained decision-making model to obtain a time-series-based decision-making model; according to the time-series-based decision-making model, determining the third observation states of the environmental vehicles of the target vehicle in each second intersection scenario; according to each of the third observation states and the stochastic gradient descent method, respectively determining the descent gradients corresponding to each second intersection scenario to determine the corresponding meta-goals; according to the meta-goals, determining the sensitive parameters of the time-series-based decision-making model; based on the sensitive parameters, updating the time-series-based decision-making model so that the updated time-series-based decision-making model can determine the behavior strategy of the corresponding target vehicle according to the current observation state.

[0006] In an implementation manner of the present application, retraining the pre-trained decision-making model to obtain a time-series-based decision-making model specifically includes: determining the estimation network and the target network corresponding to the pre-trained decision-making model; wherein, the structures of the estimation network and the target network are the same; determining the first action value function value corresponding to the estimation network and the second action value function value corresponding to the target network to update the sensitive parameters of the estimation network; wherein, the first action value function value and the second action value function value respectively represent the action values obtained by the target vehicle when performing corresponding actions in the first observation state and the second observation state; based on the sensitive parameters of the updated estimation network, updating the sensitive parameters of the corresponding target network to obtain a time-series-based decision-making model.

[0007] In an implementation manner of the present application, determining the first action value function value corresponding to the estimation network and the second action value function value corresponding to the target network specifically includes: the estimation network includes an action estimation network and a state estimation network, and the target network includes an action target network and a state target network; according to the first sample data, determining the state-action pair composed of the first observation state of the target vehicle and the corresponding random action, and inputting the state-action pair into the state estimation network to determine the first action value function value corresponding to the estimation network; and determining the second action value function value corresponding to the target network based on the Bellman equation.

[0008] In an implementation manner of the present application, according to each of the third observation states, the descent gradients corresponding to each of the second intersection scenarios are respectively determined to determine the corresponding meta-objective function, which specifically includes: determining the second sample data corresponding to each of the third observation states, and the batches of each of the second sample data; for the second sample data corresponding to each of the batches, respectively determining the descent gradients corresponding to each of the second intersection scenarios; adding up the descent gradients corresponding to each of the second intersection scenarios to obtain the total descent gradient; determining the total descent gradient as the meta-objective.

[0009] In an implementation manner of the present application, before determining the first observation states of the environmental vehicles corresponding to the target vehicle in each of the first intersection scenarios, the method further includes: determining the environmental vehicles in a plurality of random intersection scenarios, and the observation state information of the environmental vehicles; wherein, the observation state information at least includes the relative position and relative speed of the environmental vehicles with respect to the target vehicle; training a corresponding decision model according to the observation state information of the environmental vehicles.

[0010] In an implementation manner of the present application, before determining the third observation states of the environmental vehicles corresponding to the target vehicle in each of the second intersection scenarios according to the time-sequence-based decision model, the method further includes: putting the first sample data into the experience pool of the pre-trained decision model for re-training to obtain the time-sequence-based decision model; emptying the experience pool of the time-sequence-based decision model when the training of the time-sequence-based decision model is completed.

[0011] The embodiments of the present application further provide an intelligent vehicle intersection decision-making device based on meta-reinforcement learning. The device includes: a processor; and a memory storing executable code thereon. When the executable code is executed, it enables the processor to: determine the first observation states of the environmental vehicles corresponding to the target vehicle in each first intersection scenario; based on a pre-trained decision model, obtain the second observation states, random actions, and reward values corresponding to each of the first observation states; wherein the second observation state is the observation state corresponding to each environmental vehicle at the next moment in the first observation state; use the second observation states, random actions, reward values corresponding to each of the first observation states, and each of the first observation states as first sample data to retrain the pre-trained decision model to obtain a time-series-based decision model; according to the time-series-based decision model, determine the third observation states of the environmental vehicles of the target vehicle in each second intersection scenario; according to each of the third observation states, respectively determine the descent gradients corresponding to each second intersection scenario to determine the corresponding meta-goals; according to the meta-goals, determine the sensitive parameters of the time-series-based decision model; based on the sensitive parameters, update the time-series-based decision model so that the updated time-series-based decision model can determine the behavior strategy of the corresponding target vehicle according to the current observation state.

[0012] In an implementation manner of the present application, at least one processor can also: determine the second sample data corresponding to each of the third observation states and the batches of each of the second sample data; for the second sample data corresponding to each batch, respectively determine the descent gradients corresponding to each second intersection scenario; sum up the descent gradients corresponding to each second intersection scenario to obtain a total descent gradient; and determine the total descent gradient value as the meta-goal.

[0013] The embodiment of the present application also provides a non - volatile computer storage medium, characterized in that computer instructions are stored thereon, and when the computer instructions are executed, the following are realized: determining the first observation states of the environmental vehicles corresponding to the target vehicle in each first intersection scene; obtaining, based on a pre - trained decision model, the second observation states, random actions and reward values corresponding to each of the first observation states; wherein the second observation state is the observation state corresponding to each environmental vehicle at the next moment in the first observation state; taking the second observation states, random actions, reward values corresponding to each of the first observation states, and each of the first observation states as first sample data to retrain the pre - trained decision model to obtain a decision model based on time series; determining the third observation states of the environmental vehicles of the target vehicle in each second intersection scene according to the decision model based on time series; determining the descent gradients corresponding to each second intersection scene respectively according to each of the third observation states to determine corresponding meta - targets; determining the sensitive parameters of the decision model based on time series according to the meta - targets; and updating the decision model based on time series based on the sensitive parameters so that the updated decision model can determine the behavior strategy of the corresponding target vehicle according to the current observation state.

[0014] The intelligent vehicle intersection decision method, device and medium based on meta - reinforcement learning provided by the embodiment of the present application, in view of the uncertain factors in complex intersection scenes, when observing the environmental vehicles around the target vehicle, instead of predicting the behavior strategy of the target vehicle according to the single - frame observation states at different times, the observation state is changed to a set of consecutive frames, thus proposing an algorithm of a decision model based on time series. By setting the training method of a random environment, the decision model can cope with the intersection scenes in random states. And, on the basis of the decision model based on time series obtained through retraining, the idea of meta - learning is further incorporated, so as to perform secondary gradient update on the decision model based on time series, which can enable the decision model to quickly adapt to the new environment of complex intersection scenes and improve the decision - making efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0016] Figure 1 It is a flowchart of the intelligent vehicle intersection decision method based on meta - reinforcement learning provided by the embodiment of the present application;

[0017] Figure 2 It is a decision flowchart of deep reinforcement learning based on quadratic gradient provided by the embodiment of the present application;

[0018] Figure 3 Schematic diagram of Test Scenario 1 provided by the embodiments of the present application;

[0019] Figure 4 Curve graph of the speed changes of each vehicle in Test Scenario 1 provided by the embodiments of the present application;

[0020] Figure 5 Curve graph of the acceleration change of the target vehicle in Test Scenario 1 provided by the embodiments of the present application;

[0021] Figure 6 Schematic diagram of Test Scenario 2 provided by the embodiments of the present application;

[0022] Figure 7 Curve graph of the speed changes of each vehicle in Test Scenario 2 provided by the embodiments of the present application;

[0023] Figure 8 Curve graph of the acceleration change of the target vehicle in Test Scenario 2 provided by the embodiments of the present application;

[0024] Figure 9 Schematic diagram of Test Scenario 3 provided by the embodiments of the present application;

[0025] Figure 10 Curve graph of the speed changes of each vehicle in Test Scenario 3 provided by the embodiments of the present application;

[0026] Figure 11 Curve graph of the acceleration change of the target vehicle in Test Scenario 3 provided by the embodiments of the present application;

[0027] Figure 12 Schematic diagram of Test Scenario 4 provided by the embodiments of the present application;

[0028] Figure 13 Curve graph of the speed changes of each vehicle in Test Scenario 4 provided by the embodiments of the present application;

[0029] Figure 14 Curve graph of the acceleration change of the target vehicle in Test Scenario 4 provided by the embodiments of the present application;

[0030] Figure 15 Comparison graph of the T-MAML-DDPG decision-making model and the T-DDPG decision-making model provided by the embodiments of the present application;

[0031] Figure 16 Schematic diagram of the structure of the intelligent vehicle intersection decision-making device based on meta-reinforcement learning provided by the embodiments of the present application. Detailed implementation manners

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without making creative efforts belong to the scope of protection of this application.

[0033] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the drawings.

[0034] Figure 1 It is a flowchart of an intelligent vehicle intersection decision-making method based on meta-reinforcement learning provided for an embodiment of this application. As Figure 1 shown, the intelligent vehicle intersection decision-making method based on meta-reinforcement learning provided for the embodiments of this application mainly includes the following steps:

[0035] S101. The server determines the first observation states of the target vehicle corresponding to each environmental vehicle in each first intersection scenario.

[0036] In a complex intersection scenario, the vehicle behavior decision-making process is a continuous time-series process. Therefore, for multiple first intersection scenarios, the server needs to determine the first observation state of the target vehicle in the first intersection scenario to retrain the decision model through the first observation state, so as to accurately predict the corresponding execution actions at other times according to the action conditions executed at the current moment.

[0037] Among them, the multiple first intersection scenarios are all different. They represent the scenarios of the same intersection at different times, or the scenarios of different intersections at the same time. The different intersection scenarios may have different numbers of environmental vehicles, or different initial positions or speed states of the environmental vehicles. Among them, the environmental vehicles are the vehicles around the target vehicle in each intersection scenario. The first observation state represents the observation state of the target vehicle at the current moment in the corresponding first intersection scenario, and at least includes the relative position and relative speed of the environmental vehicle with respect to the target vehicle.

[0038] In one embodiment, before retraining the decision model, the server needs to determine the environmental vehicles and their corresponding observation state information under multiple preset random intersection scenarios. Among them, the observation state information at least includes the relative position and relative speed of the environmental vehicle with respect to the target vehicle. Furthermore, the server trains the corresponding decision model according to the observation state information. In the embodiments of this application, a reinforcement learning (Deep Deterministic Policy Gradient, DDPG) decision model can be used as the vehicle behavior decision model.

[0039] In the intersection scenario, the ego vehicle continuously interacts with the environment, thereby changing its own actions and the state of the environment, and receiving penalties or rewards. This process can be abstractly represented as a quadruple (S, A, P, R), where S is a finite set of states, A is a finite set of control actions, P is the probability of transitioning from one state to a new state when executing an action, and R is the expected immediate reward. Through iteration, the ultimate goal is to find an optimal behavior strategy π∶S→A for the vehicle. Through this strategy, the vehicle needs to know what the best action is in each state in order to maximize the reward. The value function is represented by V π (s), which represents the expected value of the reward obtained starting from the state and following the strategy π.

[0040] The DDPG decision model is an improved Actor-Critic method. In the Actor-Critic algorithm, the policy function in the actor gives an action A based on the given current state S. The critic evaluates the corresponding action value function based on the action A output by the actor and the current state S. Through the pre-trained DDPG decision model, the corresponding random action can be output according to the current state of the target vehicle, thereby determining the behavior strategy that the target vehicle should adopt in the current observation state. However, the pre-trained DDPG decision model follows the characteristic that the current state of the target vehicle at the current moment is only related to the previous moment. In a complex intersection scenario, the behavior decision-making process of the vehicle is essentially a continuous time-series process, and the state of the vehicle at the next moment is uncertain. Therefore, it is very difficult for the DDPG model to be trained to convergence in a complex scenario, and it is also impossible to obtain accurate actions for the target vehicle states at different moments.

[0041] Therefore, the server needs to evaluate the quality of the strategy based on the DDPG decision model and combined with the estimation of the future state.

[0042] S102. The server obtains the second observation state, random action, and reward value corresponding to each first observation state based on the pre-trained decision model.

[0043] Among them, the second observation state is the observation state corresponding to each ego vehicle at the next moment of the first observation state. After the server determines the first observation state of the target vehicle at the current moment, it needs to determine the execution action corresponding to the first observation state, the second observation state corresponding after executing the corresponding action in the first observation state, and the corresponding reward value according to the pre-trained DDPG decision model. Then, based on this, the DDPG decision model is retrained to obtain a decision model combined with time series, that is, the T-DDPG decision model.

[0044] S103. Use the second observation state, random action, reward value corresponding to each of the first observation states, and each of the first observation states as first sample data to retrain the pre-trained decision model to obtain a time-series based decision model.

[0045] The server uses the first observation state, as well as the random action, reward value, and second observation state corresponding to the first observation state as first sample data, and inputs it into the experience pool of the pre-trained DDPG decision model for retraining. Specifically, let the first observation state of the target vehicle at the current moment be s, the second observation state corresponding to the next moment be s′, a be the random action corresponding to the first observation state, and r be the immediate reward value. Then the first sample data is represented as <s, a, r, s′>.

[0046] In one embodiment, the DDPG decision model includes two networks: the Actor network (action network) and the Critic network (state network). Moreover, the DDPG model has a dual-network structure. The Actor network (action estimation network) and the Critic network (state estimation network) are estimation networks, and they both have corresponding target networks, namely the Target-Actor network (action target network) and the Target-Critic network (state target network). When training the DDPG decision model, only the sensitive parameters of the Actor network and the Critic network need to be trained, while the sensitive parameters in the Target-Actor network and the Target-Critic network are copied from the sensitive parameters of the estimation network during the iterative training process of the model. Therefore, retraining the DDPG decision model according to the first sample data is essentially a process of updating the sensitive parameters to maximize the reward value corresponding to the output behavior strategy, which is specifically achieved through the following methods:

[0047] First, the server determines the estimation network and the target network corresponding to the pre-trained DDPG decision model; among them, the structures of the estimation network and the target network are the same. During the iterative training process, the sensitive parameters of the estimation network are updated first.

[0048] Second, in the DDPG algorithm, the Critic network will evaluate the corresponding first action value function value based on the current state s and the output action a. Correspondingly, it will also evaluate the second action value function value based on the state s′ at the next moment and the corresponding action a′. The server will first update the estimation network according to the first action value function value corresponding to the estimation network and the second action value function value corresponding to the target network.

[0049] Specifically, the server can determine a state-action pair <s,a> consisting of the first observation state s and the random action a corresponding to the first observation state based on the first sample data, and then input the state-action pair into the state estimation network to obtain the corresponding first action value function value.

[0050] Moreover, according to the Bellman equation, the server can obtain the corresponding second action value function:

[0051]

[0052] Among them, P(s,π,s′) is the state transition probability from state s to state s′ after executing policy π, R(s,π,s′) is the expected reward value obtained after transitioning from s to s′ according to policy π, and γ is the discount factor. Furthermore, the server can obtain the second action value function value based on the reward value R, s′, and a′.

[0053] Among them, the first action value function value and the second action value function value respectively represent the action value obtained by the target vehicle when executing the corresponding action in the first observation state and the second observation state, that is, the Q value. It should be noted that the first action value function value corresponds to the Q value of the estimation network, and the second action value function value corresponds to the Q value of the target network.

[0054] Furthermore, after the server determines the Q values of the estimation network and the target network, it will further determine the corresponding loss function according to the square of the difference between the two, and update the gradient of the Critic network according to the loss function. This is specifically implemented through the following formula:

[0055]

[0056] Among them, Q(s,a|θ Q ) represents the Q value of the estimation network, y i represents the Q value of the target network, and N represents the total number of batches.

[0057] Furthermore, after the server updates the Critic network, it can obtain a deterministic policy, and then update the Actor network through policy gradients. The server determines the policy corresponding to the target vehicle from the first observation state to the corresponding execution action based on the updated state estimation network, and calculates the gradient of the policy for the updated policy; according to the gradient of the policy, the Actor network is updated through the following formula:

[0058]

[0059] Among them, N represents the total number of batches of the first sample data, Q(s,a|θ Q ) represents the first action value function value, Represents the gradient of the Q value. Represents the gradient of the policy π.

[0060] Thus, the training of the Critic network and the Actor network and the update of the corresponding sensitive parameters are completed.

[0061] Finally, the server updates the corresponding target network according to the updated estimation network, and thus the T-DDPG decision-making model is obtained. In the T-DDPG decision-making model, the continuous action space problem is solved by the methods of experience replay and asynchronous update. The sensitive parameters of the target network are updated according to the following equations:

[0062] θ Q′ ← τθ Q +(1 - τ)θ Q′

[0063] θ μ′ ← τθ μ +(1 - τ)θ μ′

[0064] Among them, θ Q represents the sensitive parameter of the Critic network, and θ μ represents the sensitive parameter of the Actor network.

[0065] Establishing the T-DDPG decision-making model can extend the adoption of vehicle behavior strategies to the observed states of the current and past periods of time. In this way, the decision-making model trained based on the first observed state can still output accurate random actions at the moment corresponding to the second observed state. When facing the intersection scenario with high real-time performance, corresponding strategies can be obtained in a timely manner according to the environmental state, with high flexibility and accuracy.

[0066] S104. The server determines the third observed states of the environmental vehicles of the target vehicle in each second intersection scenario according to the time-sequence-based decision-making model.

[0067] Based on a single update of the sensitive parameters of the decision-making model, when adapting to a new intersection scenario, the server samples new tasks and performs secondary training on the already updated sensitive parameters to minimize the sum of the loss functions of all tasks under the updated sensitive parameters, so as to greatly improve the reward and make the decision-making model more adaptable. Among them, a new task may include achieving a new goal in a new intersection environment or successfully achieving a previously trained goal, enabling the strategy to quickly adapt to complex and changeable scenarios.

[0068] In one embodiment, when retraining the DDPG decision model, the server puts the first sample data into the experience pool. In the case where the T-DDPG decision model is trained, when adapting to a new task, the server needs to empty the experience pool, which can make the decision model applicable to the scenario where the number of surrounding vehicles is random, reduce errors, and improve the applicability of the decision-making.

[0069] In one embodiment, after emptying the experience pool, the server needs to obtain the third observation states of the environmental vehicles of the target vehicle in each second intersection scenario according to the T-DDPG decision model, so as to update the sensitive parameters of the T-DDPG decision model in the second intersection scenario through the third observation states, enabling the T-DDPG decision model to adapt to the second intersection scenario and output corresponding strategies.

[0070] S105. The server respectively determines the descent gradients corresponding to the second intersection scenarios according to the third observation states, so as to determine the corresponding meta-goals.

[0071] In one embodiment, when the server performs a secondary update on the sensitive parameters of the T-DDPG decision model, it uses the updated policy to sample the second sample data to perform a secondary update on the already updated sensitive parameters. By collecting sample data under multiple intersection scenarios and training them, the T-DDPG decision model can widely learn the internal features of all scenarios and update the parameters according to the total gradients of each scenario, achieving cross-task parameter optimization.

[0072] The embodiment of the present application adopts Model-Agnostic Meta-Learning (MAML), which is a method of learning the parameters of any standard model according to the idea of meta-learning, so as to prepare the model for quickly adapting to new scenarios, aiming to optimize the model parameters when facing new scenarios, so that a maximum-efficiency behavior strategy can be generated through one or a small number of gradient calculations. Let a parameterized function model be represented by f θ The initial parameter of the model is θ, and it becomes θ′ after update i . By performing one or more gradient calculations on each second intersection scenario and summing up the gradients of each second intersection scenario, the meta-goal in the learning process of the decision model can be obtained. The meta-goal can minimize the sum of the loss functions of all tasks of the model under θ′ i .

[0073] Specifically, the server determines the second sample data corresponding to the third observation state and the batch of the second sample data, i.e., min-batch. Then, for the second sample data corresponding to each min-batch, the descending gradients corresponding to each second intersection scenario are determined; then, the descending gradients corresponding to each second intersection scenario are summed up to obtain the total descending gradient; finally, the total descending gradient is used as the meta-goal. The meta-goal aims to minimize the sum of the loss functions under each second intersection scenario. Through the following formula, the meta-goal can be obtained:

[0074]

[0075] where represents the loss function, α is the step size, θ represents the sensitive parameter of the T-DDPG decision model, and Q(s,a'|θ) represents the action value function value.

[0076] S106. The server determines the sensitive parameter of the time-series-based decision model according to the meta-goal.

[0077] After obtaining the meta-goal, the server can update the sensitive parameter of the T-DDPG decision model through the following formula:

[0078] In the embodiments of this application, the idea of MAML is incorporated into the T-DDPG decision model. The purpose is to enable the target vehicle to quickly obtain a behavior strategy for a new scenario task by using a small amount of experience obtained through interaction with the environment before, improve the adaptability of the T-DDPG decision model to the new environment, and accelerate the model convergence speed. In the embodiments of this application, the meta-goal is the gradient of the Actor network. By summing up the gradients of the Actor network, the T-DDPG model can quickly adapt to the new intersection scenario and thus output the corresponding behavior strategy.

[0079] Figure 2 This is the decision flow chart of the deep reinforcement learning based on the quadratic gradient provided by the embodiments of this application. As Figure 2As shown in the figure, the intelligent vehicle intersection decision-making method provided by the embodiment of the present application adopts a quadratic gradient update method. First, the server initializes the model parameters θ based on the pre-trained DDPG decision-making model, and then sets the training tasks in a random intersection scenario, that is, multiple first intersection scenarios. After that, the first sample data is put into the experience pool for retraining the DDPG decision-making model, and the corresponding Actor network and Critic network are updated to obtain the T-DDPG decision-making model. Thus, one gradient update of the decision-making model based on the random intersection scenario is completed. After that, in each second intersection scenario, based on the T-DDPG decision-making model, the corresponding second sample data is determined, and the gradient calculation in multiple scenarios is performed accordingly. The initial parameters θ are optimized according to the sum of the gradients of each scenario, so as to update the initial parameters θ. Thus, the quadratic gradient update of the intelligent vehicle intersection decision-making model is completed. Based on the updated T-DDPG decision-making model, the target vehicle can quickly adapt to the random scenario and obtain a better behavior strategy.

[0080] S107. The server updates the time-series based decision-making model based on the sensitive parameters, so that the updated time-series based T-DDPG decision-making model can determine the corresponding behavior strategy according to the current observation state.

[0081] The updated T-DDPG decision-making model (T-MAML-DDPG decision-making model) can quickly adapt to the new scenario and determine the behavior strategy of the target vehicle according to the observation state of the current environmental vehicle, improving the generalization ability of the model.

[0082] Next, based on different intersection scenarios, the T-MAML-DDPG decision-making model is tested in a random scenario after 2000 rounds of training. The test results are as follows:

[0083] Scenario 1: There are 3 environmental vehicles, and the vehicle itself focuses on efficiency and tends to go first.

[0084] Figure 3 It is a schematic diagram of test scenario 1 provided by the embodiment of the present application. In this scenario, 3 environmental vehicles, namely environmental vehicle 1, environmental vehicle 2, and environmental vehicle 3, are in the same lane. The vehicle itself takes the lead strategy after environmental vehicle 1 turns left. Figure 4 It describes the speed changes of each vehicle. It can be seen that within the first 3.5 seconds, each vehicle accelerates forward. Subsequently, the vehicle itself maintains acceleration and goes through the conflict point first. At this time, environmental vehicle 2 decelerates and gives way, waiting for the vehicle itself to pass through the conflict point and then accelerating forward. Environmental vehicle 3 basically maintains its speed behind. Figure 5 It describes the output strategy of the decision-making algorithm in this scenario, and the acceleration curve rises with a trend of first steep and then gentle.

[0085] Scenario 2: There are 4 environmental vehicles, and the vehicle itself focuses on efficiency and tends to go first.

[0086] Figure 6 Schematic diagram of test scenario 2 provided by an embodiment of the present application. The environmental vehicle 1 turns right, but the right-turn route does not affect the left-turn route of the vehicle itself. The vehicle itself chooses to accelerate and go first, passes through the conflict area before the environmental vehicles 2, 3, and 4 reach the conflict area, and then reaches the target lane. Figure 7 、 8 The speed changes and acceleration changes of each vehicle are described in []. The environmental vehicle 2 first accelerates. The speed of the environmental vehicle 4 is relatively slow and it only accelerates to 6 m / s after 4 s. The vehicle itself chooses to go first according to the positions and speed states of the surrounding vehicles, keeps accelerating, and takes the lead in passing through the conflict point and driving towards the target lane. The environmental vehicle 2 chooses to yield because of the acceleration strategy of the vehicle itself, decelerates at 5.5 s, and then accelerates forward. The environmental vehicles 4 and 3 basically maintain their speeds because they are far from the conflict point.

[0087] Scenario 3: There are 5 environmental vehicles. The vehicle itself focuses on efficiency and tends to go first.

[0088] Figure 9 Schematic diagram of test scenario 3 provided by an embodiment of the present application. The environmental vehicle 1 turns left. As can be seen from Figure 10 、 11 , the environmental vehicles 4 and 5 accelerate. The environmental vehicle 2 needs to turn right, so it decelerates and waits for 4 and 5 to leave the conflict area. The environmental vehicle 3 decelerates to maintain a certain distance because of the deceleration of the environmental vehicle 2. The vehicle itself chooses to yield to the environmental vehicles 4 and 5, and then drives towards the target lane together with the environmental vehicle 2 before the environmental vehicle 3 reaches the conflict point. During the left-turn passing process in this scenario, the vehicle itself is in a probing state because there are many surrounding vehicles, and the speed curve shows a trend of repeated acceleration and deceleration. The speeds of the environmental vehicles 4 and 5 almost remain unchanged after 3 s. The speed curve of the environmental vehicle 2 is relatively fluctuating during the right-turn process. Considering the positions of the environmental vehicles 4 and 5 and the right-turn trend of 2, the vehicle itself chooses to decelerate and yield to the environmental vehicles 4 and 5, and passes through first before the environmental vehicle 3 reaches the conflict area.

[0089] Scenario 4: There are 5 environmental vehicles. The vehicle itself focuses on safety and tends to yield.

[0090] Figure 12 Schematic diagram of test scenario 4 provided by an embodiment of the present application. As can be seen from Figure 13 、 14 , the environmental vehicles 4 and 5 accelerate. The vehicle itself first chooses to yield to 4 and 5. When 4 and 5 leave the conflict area, the environmental vehicle 2 reaches the conflict area, and 5 and 3 are also about to arrive. At this time, the vehicle itself chooses to continue to yield and turns left towards the target lane after all the environmental vehicles leave the conflict area. All 5 environmental vehicles drive towards the intersection at almost a constant speed, and it can be seen from (a) that the distances between the 5 vehicles are relatively uniform. The vehicle itself cannot pass through between any two vehicles, and finally adopts a deceleration strategy and accelerates through after 12 s. At this time, the environmental vehicle 5 has passed through the conflict area.

[0091] A comparative test was conducted on the T-MAML-DDPG decision model and the T-DDPG decision model. The two decision models were respectively trained in a random complex intersection scenario, and the number of surrounding vehicles, the motion states of surrounding vehicles, and the initial position of the vehicle itself were randomly set. After every 20 rounds of training, a test round was carried out. As Figure 15 shown, it is a comparison of the test results of T-MAML-DDPG and T-DDPG.

[0092] From Figure 15 it can be seen that both methods converged to the region after 80 tests. In the first 20 tests of the T-DDPG algorithm, the episode reward value was around -1000 or below. It was not until the 30th test that the episode reward value reached around -500. An episode reward value of around -1000 indicates that a collision would occur at this time. The T-DDPG strategy did not learn to avoid collisions until after 600 rounds of training. However, the episode reward value of T-MAML-DDPG was relatively high in the first few tests, reaching around -500, which was equivalent to the effect of T-DDPG in the 30th test. Compared with the reward values of the first few tests of T-DDPG, T-MAML-DDPG increased by 66.7%, indicating that the vehicle based on the T-MAML-DDPG decision algorithm quickly adapted to the complex scenario and, on the basis of ensuring safe passage, searched for a strategy that balanced efficiency. In terms of the convergence speed, T-DDPG converged in the 60th test, while T-MAML-DDPG had tended to converge after 30 tests, and the convergence speed was increased by about 50% compared with T-DDPG. As can be seen from the figure, the episode reward value regions of the T-MAML-DDPG and T-DDPG algorithms were the same after the 50th test, indicating that the T-DDPG decision algorithm based on MAML was not much different from the T-DDPG algorithm in terms of the final performance ability, but it could help the target vehicle quickly adapt to the complex and changeable intersection environment, improve the adaptability, and the fast adaptability could enable the algorithm to possess the ability to adaptively process random tasks as early as possible, that is, to improve the generalization ability.

[0093] The above is the method embodiment proposed in this application. Based on the same idea, some embodiments of this application also provide the corresponding device and non-volatile computer storage medium for the above method. Figure 16 It is a schematic structural diagram of an intelligent vehicle intersection decision-making device based on meta-reinforcement learning provided by an embodiment of this application. As Figure 16 shown, the device includes: a processor 1601; and a memory 1602, on which executable code is stored. When the executable code is executed, the processor 1601 is enabled to execute the above-mentioned intelligent vehicle intersection decision-making method based on meta-reinforcement learning.

[0094] A non - volatile computer storage medium provided by some embodiments of the present application stores computer - executable instructions, and the computer - executable instructions are set as: the intelligent vehicle intersection decision - making method based on meta - reinforcement learning as above.

[0095] Each embodiment in the present application is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0096] The devices and media provided by the embodiments of the present application correspond one - to - one with the methods. Therefore, the devices and media also have beneficial technical effects similar to those of their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be elaborated here.

[0097] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer - usable storage media (including but not limited to disk storage, CD - ROM, optical storage, etc.) containing computer - usable program code.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processors of general - purpose computers, special - purpose computers, embedded processors, or other programmable data - processing devices to generate a machine, so that the instructions executed by the processors of the computer or other programmable data - processing devices generate means for implementing the functions specified in Figure 1 one or more flows and / or Figure 1 blocks or multiple blocks.

[0099] These computer program instructions can also be stored in a computer - readable memory that can direct a computer or other programmable data - processing device to work in a specific manner, so that the instructions stored in the computer - readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one or more flows and / or Figure 1 blocks or multiple blocks.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing the steps specified in one Figure 1 process or multiple processes and / or blocks Figure 1 steps of the functions specified in one block or multiple blocks.

[0101] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory. The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0102] Computer-readable media includes permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0103] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, commodity or device including the said element.

[0104] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

[0105] Each embodiment in the present application is described in a progressive manner. For the parts that are the same or similar among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content.

[0106] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, commodity or device including the said element.

[0107] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. An intelligent vehicle intersection decision-making method based on meta-reinforcement learning, characterized in that The method includes: Determining first observation states of environmental vehicles corresponding to a target vehicle in each first intersection scenario; Based on a pre-trained decision model, obtaining second observation states, random actions, and reward values corresponding to each of the first observation states; wherein, the second observation states are the observation states corresponding to the environmental vehicles at the next moment in the first observation states; Taking the second observation states, random actions, reward values corresponding to each of the first observation states, and each of the first observation states as first sample data, and retraining the pre-trained decision model to obtain a decision model based on time series; According to the decision model based on time series, determining third observation states of environmental vehicles of the target vehicle in each second intersection scenario; According to each of the third observation states, respectively determining descent gradients corresponding to each of the second intersection scenarios to determine corresponding meta-goals; According to the meta-goals, determining sensitive parameters of the decision model based on time series; Based on the sensitive parameters, updating the decision model based on time series so that the updated decision model based on time series can determine a behavior strategy of a corresponding target vehicle according to a current observation state.

2. The intelligent vehicle intersection decision-making method based on meta-reinforcement learning according to claim 1, wherein Retraining the pre-trained decision model to obtain a decision model based on time series specifically includes: Determining an estimation network and a target network corresponding to the pre-trained decision model; wherein, the structures of the estimation network and the target network are the same; Determining a first action-value function value corresponding to the estimation network and a second action-value function value corresponding to the target network to update sensitive parameters of the estimation network; wherein, the first action-value function value and the second action-value function value respectively represent action values obtained by the target vehicle when performing corresponding actions in the first observation state and the second observation state; Based on the sensitive parameters of the updated estimation network, updating sensitive parameters of the corresponding target network to obtain a decision model based on time series.

3. The intelligent vehicle intersection decision-making method based on meta-reinforcement learning according to claim 1, characterized in that, According to each of the third observation states and the stochastic gradient descent method, respectively determining descent gradients corresponding to each of the second intersection scenarios to determine corresponding meta-goals, specifically including: Determining second sample data corresponding to each of the third observation states and batches of each of the second sample data; For the second sample data corresponding to each batch, respectively determining descent gradients corresponding to each of the second intersection scenarios; Adding up the descent gradients corresponding to each of the second intersection scenarios to obtain a total descent gradient; and determining the total descent gradient as the meta-goal.

4. The intelligent vehicle intersection decision-making method based on meta-reinforcement learning according to claim 2, characterized in that, Determining the first action-value function value corresponding to the estimation network and the second action-value function value corresponding to the target network specifically includes: The estimation network includes an action estimation network and a state estimation network, and the target network includes an action target network and a state target network; According to the first sample data, determining a state-action pair composed of the first observation state of the target vehicle and the corresponding random action, and inputting the state-action pair into the state estimation network to determine the first action-value function value corresponding to the estimation network; Based on the Bellman equation, determine the second action value function value corresponding to the target network.

5. The intelligent vehicle intersection decision-making method based on meta-reinforcement learning according to claim 4, wherein Update the estimation network, which specifically includes: Determine the corresponding loss function according to the square of the difference between the first action value function value and the second action value function value, and update the state estimation network according to the loss function; Based on the updated state estimation network, determine the policy corresponding to the target vehicle from the first observation state to the corresponding execution action, and calculate the gradient of the policy for the updated policy; Update the action estimation network according to the gradient of the policy.

6. The intelligent vehicle intersection decision-making method based on meta-reinforcement learning according to claim 1, wherein Before determining the first observation state of each environmental vehicle corresponding to the target vehicle in each first intersection scenario, the method further includes: Determine environmental vehicles in multiple random intersection scenarios and the observation state information of the environmental vehicles; wherein, the observation state information at least includes the relative position and relative speed of the environmental vehicles relative to the target vehicle; Train a corresponding decision model according to the observation state information of the environmental vehicles.

7. The intelligent vehicle intersection decision-making method based on meta-reinforcement learning according to claim 1, characterized in that Before determining the third observation state of each environmental vehicle of the target vehicle in each second intersection scenario according to the time-series based decision model, the method further includes: Put the first sample data into the experience pool of the pre-trained decision model for retraining to obtain the time-series based decision model; When the training of the time-series based decision model is completed, empty the experience pool of the time-series based decision model.

8. A non-volatile computer storage medium stores computer-executable instructions, characterized in that, The computer-executable instructions are set to: Determine the first observation state of each environmental vehicle corresponding to the target vehicle in each first intersection scenario; based on the pre-trained decision model, obtain the second observation state, random action, and reward value corresponding to each first observation state; wherein, the second observation state is the observation state corresponding to each environmental vehicle at the next moment in the first observation state; use the second observation state, random action, reward value corresponding to each first observation state, and each first observation state as the first sample data to retrain the pre-trained decision model to obtain a time-series based decision model; determine the third observation state of each environmental vehicle of the target vehicle in each second intersection scenario according to the time-series based decision model; respectively determine the descending gradient corresponding to each second intersection scenario according to each third observation state to determine the corresponding meta-goal; determine the sensitive parameter of the time-series based decision model according to the meta-goal; update the time-series based decision model based on the sensitive parameter so that the updated time-series based decision model can determine the behavior strategy of the corresponding target vehicle according to the current observation state.

9. An intelligent vehicle intersection decision-making device based on meta-reinforcement learning, characterized in that, The device includes: A processor; and a memory storing executable code, which, when executed, enables the processor to: determine the first observation states of the environmental vehicles corresponding to the target vehicle in each first intersection scenario; obtain the second observation states, random actions, and reward values corresponding to each of the first observation states based on a pre-trained decision model; wherein the second observation state is the observation state corresponding to each environmental vehicle at the next moment in the first observation state; use the second observation states, random actions, reward values corresponding to each of the first observation states, and each of the first observation states as first sample data to retrain the pre-trained decision model to obtain a time-series-based decision model; determine the third observation states of the environmental vehicles of the target vehicle in each second intersection scenario according to the time-series-based decision model; determine the descent gradients corresponding to each second intersection scenario respectively according to each of the third observation states to determine corresponding meta-goals; determine the sensitive parameters of the time-series-based decision model according to the meta-goals; and update the time-series-based decision model based on the sensitive parameters so that the updated time-series-based decision model can determine the behavior strategy of the corresponding target vehicle according to the current observation state.

10. The intelligent vehicle intersection decision-making device based on meta-reinforcement learning according to claim 9, characterized in that, The processor can also: determine the second sample data corresponding to each of the third observation states and the batches of each of the second sample data; determine the descent gradients corresponding to each second intersection scenario respectively for the second sample data corresponding to each batch; sum up the descent gradients corresponding to each second intersection scenario to obtain a total descent gradient; and determine the total descent gradient as the meta-goal.

Images