A signal-free intersection vehicle cooperation method based on asynchronous deep reinforcement learning

Abstract

The application discloses a kind of vehicle cooperation methods of signal intersection based on asynchronous deep reinforcement learning, method includes: by formulating Markov decision framework to describe the coordination problem of vehicle in intersection area, the system problem includes observation state space, action space and reward function;Vehicle cooperation model is designed based on shared deep neural network architecture, the content of the shared deep neural network architecture design includes control output strategy and value function estimation strategy;The vehicle cooperation model is trained using asynchronous training strategy;According to the vehicle cooperation model after training is completed, vehicle information in actual scene is processed, and the vehicle cooperation scheme of signal intersection is output.The embodiment of the application can realize the effective cooperation of vehicle in signal intersection scene within limited computing resources, reduce vehicle congestion and conflict, improve the safety and traffic efficiency of traffic management, and can be widely applied to intelligent traffic technology field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a vehicle cooperation method for a signal-free intersection based on asynchronous deep reinforcement learning. BACKGROUND

[0002] With the continuous evolution of urban transportation systems and the increasing demand for residents' travel, the main bottleneck in the urban transportation network, the road intersection where multiple roads intersect, often becomes a high-risk area of traffic congestion and potential safety hazards. Although intersections with traffic signal timing can effectively plan traffic flow, traffic signals are not fully popularized at urban intersections, and traffic congestion and accidents often occur in signal-free intersection scenarios. Under the background of the rapid development of V2X technology and intelligent connected vehicles, how to develop an effective cooperation method to solve the problem of vehicle passing through a signal-free intersection is a difficult problem to be solved in the field of autonomous driving.

[0003] In the field of research on signal-free intersection cooperation, the main problems existing at present include: the central coordination system does not fully consider the space-time characteristics of traffic flow, resulting in great waste of time and space, and low efficiency of traffic management; the calculation method used by the coordination system often has great computational complexity, which is difficult to guarantee the real-time requirement of the algorithm within limited computing resources, which leads to their difficulty in application in actual deployment; the coordination system may not be able to accurately identify and model the driving state and driving intention of the vehicle, increasing the risk of collision accidents in the intersection area. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a vehicle cooperation method for a signal-free intersection based on asynchronous deep reinforcement learning, which can realize effective cooperation of vehicles in a signal-free intersection scenario within limited computing resources, reduce vehicle congestion and conflicts, and improve the safety and efficiency of traffic management.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application proposes a vehicle cooperation method for a signal-free intersection based on asynchronous deep reinforcement learning, comprising the following steps:

[0006] A Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, and the system problem includes an observation state space, an action space and a reward function;

[0007] A vehicle cooperation model is designed based on a shared deep neural network architecture, and the content of the shared deep neural network architecture design includes a control output policy and a value function estimation policy;

[0008] The vehicle cooperation model is trained using an asynchronous training strategy;

[0009] According to the vehicle cooperation model after the training is completed, vehicle information in an actual scene is processed, and a vehicle cooperation scheme for a non-signalized intersection is output.

[0010] In some embodiments, the Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, including the following steps:

[0011] According to the state space of all vehicles in series, the state space is obtained, wherein for the state s of each vehicle n , the formula s n =(x n ,y n ,vx n ,vy n ,d n ) is used, wherein x n , y n represent the horizontal and vertical coordinates of the vehicle in the intersection scene, vx n , vy n represent the projection speed of the instantaneous speed of the vehicle on the horizontal and vertical coordinate axes, and d n represents the intention of the vehicle;

[0012] According to the pure tracking algorithm, the steering angle of the vehicle is calculated, and then the lateral action of the vehicle is determined; and the longitudinal action of the vehicle is determined according to the acceleration of the vehicle, and the action space of all vehicles is obtained in series.

[0013] When the vehicle performs the action and the state value is converted, the corresponding reward value is determined according to the reward function; the reward value is composed of a collision reward, a time step reward and a success reward; wherein the collision reward is used to give a collision reward with a value of -100 when the vehicle collides, otherwise 0; the time step reward is determined according to the negative value of the accumulated time step of the current simulation environment; the success reward is used to give a success reward with a value of 100 when all vehicles successfully pass through the intersection without collision, otherwise 0.

[0014] In some embodiments, the Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, including the following steps:

[0015] When the environment is in the state value S t , the vehicles perform the action A, and their next time state S t+1 is calculated by the vehicle kinematics equation, and the calculation formula is: Where x and y represent the vehicle position coordinates, θ and v represent the vehicle body angle and speed, δ and a represent the steering angle and acceleration of the vehicle, L represents the longitudinal axis length of the vehicle, and represent the vehicle position coordinates of the next time state, and the body angle and the speed of the vehicle representing the state of the vehicle at the next moment;

[0016] The cumulative reward of the action sequence is calculated by a reward discount factor.

[0017] In some embodiments, the vehicle cooperation model is designed based on a shared deep neural network architecture, including the following steps:

[0018] An A2C deep neural network model is used to learn the overall decision of the vehicle at the intersection; the A2C deep neural network model combines an Actor model and a Critic model; the Actor model interacts by sampling actions, and updates the policy using the method of policy gradient; the Critic model provides feedback signals by estimating the value function, helping the Actor model to better update the policy;

[0019] The A2C deep neural network model is divided into a shared network and a separate network, wherein the shared network is composed of one layer of long short-term memory network LSTM and two layers of fully connected layer network FC; the LSTM is used to learn the time correlation features in the sequence data, and the fully connected layer is used to further extract the spatial correlation features in the sequence data, and the Tanh activation function is used for nonlinear conversion of the features between the networks; the separate network uses two layers of fully connected layers to learn the output features of the shared network, and outputs the action control quantity; the separate network uses one layer of fully connected layers to learn the output features of the shared network, and outputs the value function estimate value;

[0020] The network layer is updated by using generalized advantage estimation, and the advantage function is estimated by weighted combination of multi-step advantage estimation;

[0021] In the training process, the calculation of the network loss is composed of the policy loss and the value function loss;

[0022] The gradient of the network loss is calculated by the back propagation method, and then the parameters of the model are updated by using the Adam optimizer.

[0023] In some embodiments, the calculation formula of the network loss is:

[0024]

[0025] Wherein, log probs (t) represents the logarithmic probability, Ent coe represents the policy entropy regularization coefficient, Ent(t) is the policy entropy, r t is the reward function value; GAE(t) represents the generalized advantage estimation; loss policy represents the policy loss; loss value represents the value function loss; V(st ) represents the value function estimate;

[0026] wherein the calculation formula of the generalized advantage estimate is:

[0027]

[0028] wherein δ t represents the advantage estimate at time step t, λ is a hyperparameter controlling bias and variance, T represents the maximum value of the time step; γ represents the reward discount factor.

[0029] In some embodiments, the training of the vehicle collaboration model using the asynchronous training strategy includes the following steps:

[0030] The asynchronous update of the model is realized by using the multi-process concurrency feature of the operating system. Before the training starts, multiple processes are initialized as training processes, and one process is initialized as a test process. At the same time, a parameter sharing pool is created to load and update the model parameters in real time.

[0031] In some embodiments, for the training processes: each training process individually creates a simulated intersection environment using a different random seed; each time the simulation environment is reset, the model parameters of the vehicle collaboration model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition is reached; the feature sequence information of the interaction process is used for training and updating the model; all training processes share a parameter sharing pool mutex, which will be identified as a blocked state when a training process is updating the model parameters, preventing other training processes from updating the parameters; when the parameter update is complete, the mutex will be released, allowing other processes to continue updating the model parameters;

[0032] For the test process: to test the effectiveness and robustness of the model in real time during the training process, one process is initialized as a test process, which randomly initializes the intersection scene and vehicle position every 30s, loads the model parameters from the parameter sharing pool, and tests whether the current model can effectively complete the collaborative task; when the model can well complete the collaborative task in a continuous time period, it is determined that the current asynchronous training has converged, and all processes are terminated and the trained model is saved;

[0033] For the parameter sharing pool: the parameter sharing pool is shared by all processes, and the training processes and the test process load the model parameters from the parameter sharing pool for interaction in the simulation environment; the parameter sharing pool is provided with a mutex, which only allows one training process to enter the parameter pool to update the model parameters at the same time.

[0034] Another aspect of the embodiment of the application also provides a signal-free intersection vehicle collaboration system based on asynchronous deep reinforcement learning, comprising:

[0035] a first module configured to formulate a Markov decision framework to describe a coordination problem of vehicles in an intersection area, the system problem including an observation state space, an action space and a reward function;

[0036] a second module configured to design a vehicle coordination model based on a shared deep neural network architecture, the shared deep neural network architecture including a control output policy and a value function estimation policy;

[0037] a third module configured to train the vehicle coordination model using an asynchronous training strategy;

[0038] a fourth module configured to process vehicle information in an actual scene according to the trained vehicle coordination model, and output a vehicle coordination scheme for a signal-free intersection.

[0039] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0040] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0041] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method described above.

[0042] The embodiment of the present application at least has the following beneficial effects: the present application provides a signal-free intersection vehicle coordination method based on asynchronous deep reinforcement learning, which formulates a Markov decision framework to describe a coordination problem of vehicles in an intersection area, the system problem including an observation state space, an action space and a reward function; designs a vehicle coordination model based on a shared deep neural network architecture, the shared deep neural network architecture including a control output policy and a value function estimation policy; trains the vehicle coordination model using an asynchronous training strategy; and processes vehicle information in an actual scene according to the trained vehicle coordination model, and outputs a signal-free intersection vehicle coordination scheme. The embodiment of the present application can realize effective coordination of vehicles in a signal-free intersection scene within limited computing resources, reduce vehicle congestion and conflicts, and improve the safety and efficiency of traffic management. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0044] Figure 2 is a flowchart of the overall steps provided by an embodiment of the present application;

[0045] Figure 3 is an application scenario diagram provided by an embodiment of the present application;

[0046] Figure 4 is an encoding mode of a driving state quantity d n provided by an embodiment of the present application;

[0047] Figure 5 is a pure tracking algorithm schematic diagram provided by an embodiment of the present application;

[0048] Figure 6 is a vehicle cooperative network model architecture diagram provided by an embodiment of the present application;

[0049] Figure 7 is a whole framework of an asynchronous deep reinforcement learning method provided by an embodiment of the present application;

[0050] Figure 8 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0052] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0053] The terms "at least one", "multiple", "each", "any", and the like, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding plurality, and any refers to any one of the plurality.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing the embodiments of the application only and is not intended to be limiting of the application.

[0055] The vehicle cooperation method for unsignalized intersection based on asynchronous deep reinforcement learning provided by the embodiments of the application relates to the technical field of intelligent transportation. The vehicle cooperation method for unsignalized intersection based on asynchronous deep reinforcement learning provided by the embodiments of the application can be applied to a terminal, can also be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; and the software can be an application for implementing the vehicle cooperation method for unsignalized intersection based on asynchronous deep reinforcement learning, and the like, but is not limited to the above forms.

[0056] The application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0057] As Figure 1 shown is an implementation environment schematic diagram provided by the embodiments of the application. Referring toFigure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected by wireless or wired means to complete data transmission and exchange.

[0058] The server 101 can be a stand-alone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0059] In addition, the server 101 can also be a node server in a blockchain network. The blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0060] The terminal 102 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 can also be a vehicle-mounted terminal of various device types as exemplified above, but is not limited thereto. The terminal 102 and the server 101 can be directly or indirectly connected by wired or wireless communication, and the present embodiment is not limited thereto.

[0061] Exemplarily based on the implementation environment shown in the figure, the present embodiment provides a vehicle cooperation method for a signal-free intersection based on asynchronous deep reinforcement learning. The following takes the case that the vehicle cooperation method for a signal-free intersection based on asynchronous deep reinforcement learning is applied to the server 101 as an example. It can be understood that the method can also be applied to the terminal 102. Figure 1 Referring to the flowchart of the vehicle cooperation method for a signal-free intersection based on asynchronous deep reinforcement learning provided by the present embodiment and applied to the server, the execution subject of the method can be any one of the computer devices (including servers or terminals) described above. Referring to the figure,

[0062] Figure 2 The method can include the following steps: Figure 2 Figure 2 A Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area. The system problem includes an observation state space, an action space, and a reward function.

[0063] A vehicle cooperation model is designed based on a shared deep neural network architecture. The shared deep neural network architecture includes a control output policy and a value function estimation policy.

[0064] A vehicle cooperation model is designed based on a shared deep neural network architecture. The shared deep neural network architecture includes a control output policy and a value function estimation policy.​​

[0065] The vehicle cooperation model is trained using an asynchronous training strategy;

[0066] The vehicle cooperation model, after training, processes vehicle information in the actual scenario and outputs a vehicle cooperation scheme for unsignalized intersections.

[0067] In some embodiments, developing a Markov decision framework to describe the vehicle coordination problem in an intersection area includes the following steps:

[0068] The state space is obtained by concatenating the state spaces of all vehicles, where for each vehicle's state s... n Using formula s n =(x n ,y n vx n ,vy n ,d n ) indicates that, where x n ,y n Vx represents the horizontal and vertical coordinates of the vehicle at the intersection. n ,vy n d represents the projected velocity of the vehicle's instantaneous velocity on the horizontal and vertical coordinate axes. n As an indication of the vehicle's intent;

[0069] The vehicle's steering angle is calculated using a pure tracking algorithm, which in turn determines the vehicle's lateral motion. The longitudinal motion is determined based on the vehicle's acceleration, and the vehicle's motion space is obtained by concatenating the longitudinal motions of all vehicles.

[0070] When a vehicle performs an action that causes a state value transition, the corresponding reward value is determined according to the reward function. The reward value consists of a collision reward, a time step reward, and a success reward. The collision reward is given as -100 when a vehicle collides, and 0 otherwise. The time step reward is determined based on the negative value of the accumulated time step in the current simulation environment. The success reward is given as 100 when all vehicles successfully pass through the intersection without collision, and 0 otherwise.

[0071] In some embodiments, developing a Markov decision framework to describe the vehicle coordination problem in an intersection area further includes the following steps:

[0072] When the environment is in state value S t When a vehicle performs action A, its next state S is... t+1 It will be calculated from the vehicle's kinematic equations, using the following formula: where x and y represent the vehicle position coordinates, and 0 and v represent the vehicle body angle and speed, and d and a represent the vehicle steering angle and acceleration, and L represents the longitudinal axis length of the vehicle, and represent the vehicle position coordinates of the next time state, and represent the vehicle body angle and speed of the next time state;

[0073] The cumulative reward of the action sequence is calculated by a reward discount factor.

[0074] In some embodiments, the vehicle cooperation model is designed based on a shared deep neural network architecture, including the following steps:

[0075] An A2C deep neural network model is used to learn the overall decision of the vehicle at the intersection; the A2C deep neural network model combines an Actor model and a Critic model; the Actor model interacts by sampling actions, and updates the policy using the method of policy gradient; the Critic model provides feedback signals by estimating the value function, helping the Actor model to better update the policy;

[0076] The A2C deep neural network model is divided into a shared network and a separate network, wherein the shared network is composed of one layer of long short-term memory network (LSTM) and two layers of fully connected layer network (FC); the LSTM is used to learn the time correlation features in the sequence data, and the fully connected layer is used to further extract the spatial correlation features in the sequence data, and the Tanh activation function is used for nonlinear conversion of the features between the networks; the separate network uses two layers of fully connected layers to learn the output features of the shared network, and outputs the action control quantity; the separate network uses one layer of fully connected layers to learn the output features of the shared network, and outputs the value function estimate value;

[0077] Generalized advantage estimation is used to update the network layer, and the advantage function is estimated by weighted combination of multi-step advantage estimation;

[0078] In the training process, the calculation of network loss is composed of policy loss and value function loss;

[0079] The gradient of the network loss is calculated by the back propagation method, and then the parameters of the model are updated using the Adam optimizer.

[0080] In some embodiments, the calculation formula of the network loss is:

[0081]

[0082] where log probs (t) represents the logarithmic probability, Ent coedenotes the policy entropy regularization coefficient, Ent(t) is the policy entropy, r t GAE(t) denotes the generalized advantage estimation; loss policy denotes the policy loss; loss value denotes the value function loss; V(s t ) denotes the value function estimate value.

[0083] wherein the calculation formula of the generalized advantage estimation is:

[0084]

[0085] wherein δ t denotes the advantage estimation at the time step t, λ is a hyperparameter for controlling the bias and variance, T denotes the maximum value of the time step; γ denotes the reward discount factor.

[0086] In some embodiments, the training of the vehicle collaboration model using the asynchronous training strategy comprises the following steps:

[0087] The asynchronous update of the model is realized by using the multi-process concurrency feature of the operating system. Before the training starts, multiple processes are initialized as training processes, one process is initialized as a test process, and a parameter sharing pool is created to load and update the model parameters in real time.

[0088] In some embodiments, for the training processes: each training process individually creates a simulated intersection environment using a different random seed; each time the simulation environment is reset, the model parameters of the vehicle collaboration model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition is reached; the feature sequence information of the interaction process is used for training and updating the model; all training processes share a parameter sharing pool mutex, when a training process is updating the model parameters, the mutex will be identified as a blocked state, preventing other training processes from updating the parameters; when the parameter update is complete, the mutex will be released, allowing other processes to continue updating the model parameters;

[0089] For the test process: to test the effectiveness and robustness of the model in real time during the training process, one process is initialized as a test process, the intersection scene and vehicle position are randomly initialized every 30s, the model parameters are loaded from the parameter sharing pool, and it is tested whether the current model can effectively complete the collaborative task; when the model can well complete the collaborative task in a continuous time period, it is determined that the current asynchronous training has converged, all processes are terminated, and the trained model is saved;

[0090] For the parameter sharing pool: the parameter sharing pool is shared by all processes, and the training process and the test process load model parameters from the parameter sharing pool for interaction in the simulation environment; the parameter sharing pool is provided with a mutex, and only one training process is allowed to enter the parameter pool to update the model parameters at the same time.

[0091] Another aspect of the embodiment of the application also provides a vehicle cooperation system for a signal-free intersection based on asynchronous deep reinforcement learning, comprising:

[0092] A first module is configured to formulate a Markov decision framework to describe the cooperation problem of vehicles in the intersection area, and the system problem comprises an observation state space, an action space and a reward function.

[0093] A second module is configured to design a vehicle cooperation model based on a shared deep neural network architecture, and the shared deep neural network architecture includes a control output policy and a value function estimation policy.

[0094] A third module is configured to train the vehicle cooperation model using an asynchronous training strategy.

[0095] A fourth module is configured to process vehicle information in an actual scene according to the trained vehicle cooperation model, and output a vehicle cooperation scheme for the signal-free intersection.

[0096] The specific implementation process of the application will be described in detail below with a specific application scenario as an example:

[0097] In view of the problems in the prior art, the application provides a signal-free intersection cooperation method based on asynchronous deep reinforcement learning, which can realize effective cooperation of vehicles in a signal-free intersection scenario within limited computing resources, reduce vehicle congestion and conflicts, and improve the safety and efficiency of traffic management.

[0098] (1) Formulate a Markov decision framework to describe the cooperation problem of vehicles in the intersection area, which includes an observation state space, an action space and a reward function.

[0099] (2) Design a vehicle cooperation model based on a shared deep neural network architecture, including a control output policy and a value function estimation policy.

[0100] (3) Train the cooperation model using an asynchronous training strategy.

[0101] For the Markov decision framework, specifically:

[0102] Markov decision frameworks can be used to describe the interaction between environmental states and vehicle behavior. Generally, this framework can be represented by a quintuple (S, A, R, P, γ), where S represents the global state space, A represents the action space, R represents the reward function, P represents the environmental state transition probability, and γ represents the reward discount factor.

[0103] (1a) Global state space S

[0104] The state space is obtained by concatenating the state spaces of all vehicles, such as... Figure 3 As shown, S = (s1, s2, s3, s4). For each vehicle's state s n Using formula s n =(x n ,y n vx n ,vy n ,d n ) represents. Where x n ,y n Vx represents the horizontal and vertical coordinates of the vehicle at the intersection. n ,vy n d represents the projected velocity of the vehicle's instantaneous velocity on the horizontal and vertical coordinate axes. n As a representation of a vehicle's intent, the driving intent of a vehicle at an intersection can be encoded, as shown in the encoding method below. Figure 4 As shown, d is determined based on the driving route and position of vehicle n. n =d1 or d n =d2.

[0105] (1b) Action Space A

[0106] In this invention, the lateral movement of the vehicle is the steering angle. Since all driving routes at the intersection are fixed, the vehicle's steering angle δ is calculated by a pure tracking algorithm, such as... Figure 5 As shown. The longitudinal motion of a vehicle is its acceleration, and the motion space is formed by the series connection of the longitudinal motions of all vehicles, A = (a1, a2, a3, a4). Wherein, the longitudinal motion of the vehicle is a... n ∈{-2,-1,0,1,2}, unit is m / s 2 .

[0107] (1c) Reward function R

[0108] When the vehicle performs action A, the state value S is... t Convert to S t+1The reward that can be obtained later is composed of three parts, R = r1 + r2 + r3. r1 is the collision reward, if the vehicle collides, the collision reward will be given a value of -100, otherwise 0; r2 is the time step reward, the value is the negative value of the accumulated time step (s) of the current simulation environment; r3 is the success reward, when all vehicles successfully pass through the intersection without collision, the success reward is given a value of 100, otherwise 0.

[0109] (1d) Environment state transition P

[0110] When the environment is in state value S t , the vehicles perform action A, and their next time state S t+1 Will be calculated by the vehicle kinematics equation. The calculation formula is as follows,

[0111]

[0112] Where x and y represent the vehicle position coordinates, theta and v represent the vehicle body angle and speed, delta and a represent the vehicle steering angle and acceleration, and L represents the longitudinal axis length of the vehicle.

[0113] (1e) Reward discount factor gamma

[0114] Used to calculate the cumulative reward of the action sequence, solve the instantaneity of future rewards, set to 0.99.

[0115] For a vehicle cooperation network model based on a deep neural network, specifically:

[0116] The application uses an A2C (Advantage Actor-Critic) deep neural network model to learn the overall decision of the vehicle at the intersection. The A2C model combines the Actor model and the Critic model, aiming to improve the decision-making ability of the agent by learning the strategy (Actor) and the value function (Critic). The Actor model interacts by sampling actions, and uses the policy gradient method to update the policy; the Critic model estimates the value function to provide feedback signals to help the Actor model better update the policy.

[0117] The A2C model is divided into a shared network part and a separate network part, as Figure 6The shared network part is composed of one layer of long short-term memory network (LSTM) and two layers of fully connected layer network (FC), the LSTM is used to learn the time correlation features in the sequence data, the fully connected layer is used to further extract the spatial correlation features in the sequence data, and the Tanh activation function is used for nonlinear conversion of the features between the networks, so as to effectively analyze and model the intersection scene in real time. The separate Actor network part uses two layers of fully connected layers to learn the output features of the shared network, and outputs the action control A. The separate Critic network uses one layer of fully connected layer to learn the output features of the shared network, and outputs the value function estimate V.

[0118] For the A2C model, the present application updates the network layer by using generalized advantage estimation (GAE), which estimates the advantage function by weighted combination of multi-step advantage estimation, reduces the bias and variance of the advantage estimation.

[0119] δ t =r t +γV(s t+1 )-V(s t )

[0120]

[0121] Wherein, δ t represents the advantage estimation at time step t, V(s t ) is the value function estimate, λ is a hyperparameter for controlling the bias and variance, and T represents the maximum value of the time step.

[0122] The network loss calculation in the training process is composed of two parts: policy loss and value function loss, and the calculation formula is as follows,

[0123]

[0124] Wherein, log probs (t) represents the logarithmic probability, Ent coe represents the policy entropy regularization coefficient, Ent(t) is the policy entropy, and r t is the reward function calculation value.

[0125] The present application calculates the gradient of the network loss by the back propagation method, and then updates the parameters of the model by using the Adam optimizer.

[0126] For the asynchronous training strategy, specifically,

[0127] The present application uses the multi-process concurrent characteristics of the operating system to realize the asynchronous update of the model. Before the training starts, multiple processes will be initialized as training processes, one process as a test process, and a parameter sharing pool is created to load and update the model parameters in real time.

[0128] Training process: Each training process creates a simulation intersection environment with different random seeds to improve the exploration of complex environment. Each time the simulation environment is reset, the model parameters of the vehicle coordination model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition (collision or task completion) is reached. The feature sequence information of the interaction process will be used to train and update the model. All training processes share a parameter sharing pool mutex, which will be blocked when a training process is updating the model parameters, preventing other training processes from updating the parameters. When the parameter update is complete, the mutex will be released, allowing other processes to continue updating the model parameters.

[0129] Test process: To test the effectiveness and robustness of the model in real time during training, a process is initialized as a test process, which randomly initializes the intersection scene and vehicle position every 30s, loads the model parameters from the parameter sharing pool, and tests whether the current model can effectively complete the collaborative task. When the model can well complete the collaborative task in a continuous period of time, it is considered that the current asynchronous training has converged, and all processes can be terminated, and the trained model is saved.

[0130] Parameter sharing pool: The sharing pool is shared by all processes, and the training process and the test process can load the model parameters from the pool for interaction in the simulation environment. At the same time, the pool is equipped with a mutex that allows only one training process to enter the parameter pool to update the model parameters at the same time.

[0131] Based on the above technical details, the overall framework of the asynchronous deep reinforcement learning method proposed by the present application is shown in Figure 7 .

[0132] In summary, compared with the prior art, the present application has the following advantages:

[0133] First, the present application formulates a Markov decision framework for intersection coordination problems, and develops effective environment state representation, vehicle action representation, and reward function design for vehicle coordination tasks.

[0134] Second, the present application designs a vehicle coordination model based on a deep neural network architecture, fully considers the time complexity and spatial complexity of the intersection environment, and designs different neuron processing layers to effectively improve the calculation efficiency and the quality of the solution.

[0135] Third, the present application uses an asynchronous training method to asynchronously share and update the model, improving the training speed and reliability of the model.

[0136] Another aspect of the embodiment of the present application also provides a vehicle coordination system for a signal-free intersection based on asynchronous deep reinforcement learning, comprising:

[0137] a first module configured to formulate a Markov decision framework to describe a coordination problem of vehicles in an intersection area, the system problem including an observation state space, an action space and a reward function;

[0138] a second module configured to design a vehicle coordination model based on a shared deep neural network architecture, the shared deep neural network architecture including a control output policy and a value function estimation policy;

[0139] a third module configured to train the vehicle coordination model using an asynchronous training strategy;

[0140] a fourth module configured to process vehicle information in an actual scene according to the trained vehicle coordination model, and output a vehicle coordination scheme for a signal-free intersection.

[0141] It can be understood that the content in the above method embodiments is applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0142] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned signal-free intersection vehicle coordination method based on asynchronous deep reinforcement learning when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0143] It can be understood that the content in the above method embodiments is applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0144] Please refer to Figure 8 , Figure 8 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0145] The processor 801 can be implemented in the mode of a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an ASIC (Application Specific Integrated Circuit, application specific integrated circuit) or one or more integrated circuits, etc., and is used to execute a related program to implement the technical solutions provided by the embodiment of the present application.

[0146] The memory 802 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), etc. The memory 802 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 802 and are called and executed by the processor 801 to implement the vehicle cooperation method based on asynchronous deep reinforcement learning at a signal-free intersection according to the embodiments of the present application;

[0147] The input / output interface 803 is configured to realize information input and output.

[0148] The communication interface 804 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0149] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803 and the communication interface 804) of the device.

[0150] The processor 801, the memory 802, the input / output interface 803 and the communication interface 804 are connected to each other through the bus 805 to realize the communication connection between the devices.

[0151] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the vehicle cooperation method based on asynchronous deep reinforcement learning at a signal-free intersection.

[0152] It can be understood that the contents in the above method embodiments are applicable to the present storage medium embodiment. The present storage medium embodiment specifically implements the same functions as the above method embodiments and achieves the same beneficial effects as the above method embodiments.

[0153] The memory is a non-transitory computer readable storage medium and can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0154] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent is obtained first, and the collection, use, and processing of such data comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user is obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate is obtained.

[0155] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0156] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0157] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0158] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0159] The terms "first", "second", "third", "fourth" and the like used in the description of the present application and the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0160] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0161] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0162] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0163] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0164] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0165] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for vehicle cooperation at unsignalized intersections based on asynchronous deep reinforcement learning, characterized in that, Includes the following steps: A Markov decision framework is developed to describe the cooperative problem of vehicles in an intersection area, which includes the observation state space, action space, and reward function. A vehicle cooperation model is designed based on a shared deep neural network architecture, the design of which includes a control output strategy and a value function estimation strategy. The vehicle cooperation model is trained using an asynchronous training strategy; Based on the trained vehicle cooperation model, the vehicle information in the actual scene is processed to output a vehicle cooperation scheme for unsignalized intersections. The development of a Markov decision framework to describe the vehicle coordination problem in an intersection area includes the following steps: The state space is obtained by concatenating the state spaces of all vehicles, where the state of each vehicle is... Using formula It means that, among them, This represents the horizontal and vertical coordinates of the vehicle at the intersection. This represents the projected velocity of the vehicle's instantaneous speed onto the horizontal and vertical coordinate axes. As an indication of the vehicle's intent; The vehicle's steering angle is calculated using a pure tracking algorithm, which in turn determines the vehicle's lateral motion. The longitudinal motion is determined based on the vehicle's acceleration, and the vehicle's motion space is obtained by concatenating the longitudinal motions of all vehicles. When a vehicle performs an action that causes a state value transition, the corresponding reward value is determined according to the reward function. The reward value consists of a collision reward, a time step reward, and a success reward. The collision reward is given as -100 when a vehicle collides, and 0 otherwise. The time step reward is determined based on the negative value of the accumulated time step in the current simulation environment. The success reward is given as 100 when all vehicles successfully pass through the intersection without collision, and 0 otherwise. The method of developing a Markov decision framework to describe the vehicle coordination problem in an intersection area also includes the following steps: When the environment is in a state value When the vehicles perform action A, their next state is... It will be calculated from the vehicle's kinematic equations, using the following formula: ,in, and Indicates the vehicle's position coordinates. and Indicates the vehicle's body angle and speed. and This indicates the vehicle's steering angle and acceleration. Indicates the longitudinal axis length of the vehicle. and The vehicle's position coordinates represent the state at the next moment. and This indicates the vehicle's body angle and velocity at the next moment; The cumulative reward of an action sequence is calculated using a reward discount factor.

2. The method for vehicle cooperation at unsignalized intersections based on asynchronous deep reinforcement learning according to claim 1, characterized in that, The design of the vehicle collaboration model based on the shared deep neural network architecture includes the following steps: An A2C deep neural network model is used to learn the overall decision-making of vehicles at intersections. The A2C deep neural network model combines an Actor model and a Critic model. The Actor model interacts by sampling actions and updates the policy using a policy gradient method. The Critic model provides feedback signals through an estimation function to help the Actor model update the policy better. The A2C deep neural network model is divided into a shared network and a standalone network. The shared network consists of a single-layer Long Short-Term Memory (LSTM) network and two fully connected layers (FC). The LSTM is used to learn the temporal features in the sequence data, and the fully connected layers are used to further extract the spatial features in the sequence data. The Tanh activation function is used to perform non-linear transformation of features between networks. The standalone network uses two fully connected layers to learn the output features of the shared network and outputs the action control quantity. The standalone network also uses one fully connected layer to learn the output features of the shared network and outputs the value function estimate. The network layers are updated using generalized advantage estimation, and the advantage function is estimated by weighted combination of multi-step advantage estimation. During training, the network loss is calculated by consisting of policy loss and value function loss; The gradient of the network loss is calculated using the backpropagation method, and then the parameters of the model are updated using the Adam optimizer.

3. The method for vehicle cooperation at unsignalized intersections based on asynchronous deep reinforcement learning according to claim 2, characterized in that, The formula for calculating the network loss is: in, Represents logarithmic probability. This represents the policy entropy regularization coefficient. For policy entropy, Calculate the value for the reward function; This represents the generalized advantage estimate; Indicates the strategy loss; Represents the loss function of the value function; This represents the estimated value of the value function; The formula for calculating the generalized dominance estimate is as follows: in, This represents the advantage estimate at time step t. To control the hyperparameters of bias and variance, T This indicates the maximum value of the time step; This represents the reward discount factor.

4. The method for vehicle cooperation at unsignalized intersections based on asynchronous deep reinforcement learning according to claim 1, characterized in that, The process of training the vehicle cooperation model using an asynchronous training strategy includes the following steps: Asynchronous model updates are achieved by leveraging the multi-process concurrency features of the operating system. Before training begins, multiple processes are initialized as training processes, one process is used as a testing process, and a parameter sharing pool is created to load and update model parameters in real time.

5. The method for vehicle cooperation at unsignalized intersections based on asynchronous deep reinforcement learning according to claim 4, characterized in that, For the training process: Each training process creates a separate simulation intersection environment using a different random number seed; each time the simulation environment is reset, the model parameters of the vehicle cooperation model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition is reached; the feature sequence information of the interaction process is used for model training and updating; all training processes share a parameter sharing pool mutex lock, and when a training process is updating model parameters, the mutex lock will be marked as blocked, preventing other training processes from updating parameters; Once the parameters are updated, the mutex will be released, allowing other processes to continue updating the model parameters; For the testing process: In order to test the effectiveness and robustness of the model in real time during training, a process will be initialized as a test process. The intersection scene and vehicle positions will be randomly initialized every 30 seconds. The model parameters will be loaded from the parameter sharing pool to test whether the current model can effectively complete the cooperative task. When the model can complete the cooperative task well in a continuous time period, it is determined that the current asynchronous training has converged, all processes will be terminated and the trained model will be saved. For the parameter sharing pool: the parameter sharing pool is shared by all processes. The training process and the testing process load model parameters from the parameter sharing pool to interact in the simulation environment. The parameter sharing pool is equipped with a mutex lock, which allows only one training process to enter the parameter pool at a time to update the model parameters.

6. A system for implementing the asynchronous deep reinforcement learning-based vehicle cooperation method at unsignalized intersections as described in any one of claims 1-5, characterized in that, include: The first module is used to develop a Markov decision framework to describe the cooperative problem of vehicles in an intersection area, which includes an observation state space, an action space, and a reward function. The second module is used to design a vehicle cooperation model based on a shared deep neural network architecture. The design of the shared deep neural network architecture includes a control output strategy and a value function estimation strategy. The third module is used to train the vehicle cooperation model using an asynchronous training strategy; The fourth module is used to process vehicle information in the actual scene based on the vehicle cooperation model after training, and output a vehicle cooperation scheme for unsignalized intersections.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images