Non-signalized intersection vehicle cooperation method based on asynchronous deep reinforcement learning

Abstract

The invention discloses a non-signalized intersection vehicle cooperation method based on asynchronous deep reinforcement learning, and the method comprises the steps: describing the cooperation problem of vehicles in an intersection region through formulating a Markov decision framework, and the system problem comprises an observation state space, an action space and a reward function; designing a vehicle cooperation model based on a shared deep neural network architecture, wherein the content of the shared deep neural network architecture design comprises a control output strategy and a value function estimation strategy; using an asynchronous training strategy to train the vehicle cooperation model; and processing vehicle information in an actual scene according to the trained vehicle cooperation model, and outputting a vehicle cooperation scheme of a non-signalized intersection. According to the embodiment of the invention, effective cooperation of vehicles in a non-signalized intersection scene can be realized in limited computing resources, vehicle congestion and conflicts are reduced, the safety and traffic efficiency of traffic management are improved, and the method can be widely applied to the technical field of intelligent traffic.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a vehicle cooperation method at an unsignalized intersection based on asynchronous deep reinforcement learning. Background Art

[0002] With the continuous evolution of urban transportation systems and the increasing travel needs of residents, the main bottleneck in urban transportation networks - road intersections where multiple roads intersect and converge - often become high-incidence areas of traffic congestion and potential safety hazards. Although intersections with traffic signal timing can effectively plan traffic flow, traffic lights at urban intersections are not fully popularized, and traffic congestion and accidents often occur at unsignaled intersections. Against the background of the rapid development of V2X technology and intelligent connected vehicle technology, how to develop effective collaborative methods to solve the problem of vehicle traffic at unsignaled intersections is a difficult problem that needs to be solved urgently in the field of autonomous driving.

[0003] In the research field of coordination in unsignalized intersection scenarios, the main problems currently exist include: the central coordination system does not fully consider the spatiotemporal characteristics of traffic flow, resulting in huge waste of time and space, and inefficient traffic management; the calculation methods used by the coordination system often have large computational complexity, and it is difficult to ensure the real-time requirements of the algorithm within limited computing resources, which makes them difficult to apply in actual deployment; the coordination system may not be able to accurately identify and model the driving status and driving intention of the vehicle, increasing the risk of collision accidents in the intersection area. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to propose a vehicle collaboration method for unsignalized intersections based on asynchronous deep reinforcement learning, which can achieve effective collaboration of vehicles in unsignalized intersection scenarios within limited computing resources, reduce vehicle congestion and conflicts, and improve the safety and traffic efficiency of traffic management.

[0005] To achieve the above object, an embodiment of the present invention provides a method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning, comprising the following steps:

[0006] A Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, wherein the system problem includes observation state space, action space and reward function;

[0007] Designing a vehicle cooperation model based on a shared deep neural network architecture, wherein the shared deep neural network architecture design includes a control output strategy and a value function estimation strategy;

[0008] Training the vehicle cooperation model using an asynchronous training strategy;

[0009] The vehicle information in the actual scene is processed according to the trained vehicle cooperation model, and a vehicle cooperation plan for an unsignalized intersection is output.

[0010] In some embodiments, the method of formulating a Markov decision framework to describe the coordination problem of vehicles in an intersection area includes the following steps:

[0011] The state space is obtained by concatenating the state spaces of all vehicles, where for each vehicle's state s n , using the formula n =(x n ,y n ,vx n ,vy n ,d n ) means, where x n ,y n Indicates the horizontal and vertical coordinates of the vehicle in the intersection scene, vx n ,vy n It represents the projection speed of the vehicle's instantaneous speed on the horizontal and vertical axes, d n As a vehicle’s expression of intent;

[0012] The steering angle of the vehicle is calculated based on the pure tracking algorithm, and then the lateral movement of the vehicle is determined; the longitudinal movement is determined based on the acceleration of the vehicle, and the movement space of the vehicle is obtained by connecting the longitudinal movements of all vehicles in series;

[0013] When the vehicle performs an action that causes the state value to change, the corresponding reward value is determined according to the reward function; the reward value is composed of a collision reward, a time step reward and a success reward; wherein the collision reward is used to give a collision reward of -100 when a vehicle collides, otherwise it is 0; the time step reward is determined according to the negative value of the accumulated time step of the current simulation environment; the success reward is used to give a success reward of 100 when all vehicles successfully pass through the intersection without collision, otherwise it is 0.

[0014] In some embodiments, the formulation of a Markov decision framework to describe the coordination problem of vehicles in an intersection area further includes the following steps:

[0015] When the environment is in state value S t When the vehicles perform action A, their next state S t+1 It will be calculated by the vehicle kinematic equation, the calculation formula is: Among them, x and y represent the vehicle position coordinates, θ and v represent the vehicle body angle and speed, δ and a represent the vehicle steering angle and acceleration, and L represents the longitudinal axis length of the vehicle. and The vehicle position coordinates representing the next moment state, and The body angle and speed of the vehicle representing the state at the next moment;

[0016] The cumulative reward for an action sequence is calculated using a reward discount factor.

[0017] In some embodiments, the designing of a vehicle cooperation model based on a shared deep neural network architecture comprises the following steps:

[0018] An A2C deep neural network model is used to learn the overall decision of vehicles at intersections; the A2C deep neural network model combines an Actor model and a Critic model; the Actor model interacts by sampling actions and updates the policy using a policy gradient method; the Critic model provides feedback signals through an estimated value function to help the Actor model better update its policy;

[0019] The A2C deep neural network model is divided into a shared network and a separate network. The shared network consists of a long short-term memory network LSTM and two fully connected layer networks FC. LSTM is used to learn time-related features in sequence data, and the fully connected layer is used to further extract spatial-related features in sequence data, and the Tanh activation function is used to perform nonlinear transformation of features between networks. The separate network uses two fully connected layers to learn the output features of the shared network and output the action control amount. The separate network uses one fully connected layer to learn the output features of the shared network and output the estimated value of the value function.

[0020] The network layers are updated using generalized advantage estimation, and the advantage function is estimated by weighted combination of multi-step advantage estimates.

[0021] During the training process, the calculation of network loss consists of policy loss and value function loss;

[0022] The gradient of the network loss is calculated by the back-propagation method, and then the parameters of the model are updated using the Adam optimizer.

[0023] In some embodiments, the network loss is calculated as follows:

[0024]

[0025] Among them, log probs (t) represents the logarithmic probability, Ent coe represents the policy entropy regularization coefficient, Ent(t) is the policy entropy, r t is the value calculated for the reward function; GAE(t) means generalized advantage estimation; loss policy Indicates strategy loss; loss value represents the value function loss; V(st ) represents the estimated value of the value function;

[0026] The calculation formula of generalized advantage estimation is:

[0027]

[0028] Among them, δ t represents the advantage estimate at time step t, λ is the hyperparameter for controlling bias and variance, T represents the maximum time step; γ represents the reward discount factor.

[0029] In some embodiments, the using an asynchronous training strategy to train the vehicle cooperation model comprises the following steps:

[0030] The multi-process concurrency feature of the operating system is used to implement asynchronous updates of the model. Before training begins, multiple processes will be initialized as training processes and one process will be initialized as a testing process. At the same time, a parameter sharing pool will be created to load and update model parameters in real time.

[0031] In some embodiments, for the training process: each training process separately creates a simulated intersection environment using different random number seeds; each time the simulation environment is reset, the model parameters of the vehicle cooperation model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition is reached; the characteristic sequence information of the interaction process is used to train and update the model; all training processes share a parameter sharing pool mutex lock, and when a training process is updating the model parameters, the mutex lock will be marked as blocked, preventing other training processes from updating the parameters; when the parameter update is completed, the mutex lock will be released, allowing other processes to continue to update the model parameters;

[0032] For the test process: In order to test the effectiveness and robustness of the model in real time during the training process, a process will be initialized as the test process. The intersection scene and vehicle position will be randomly initialized every 30 seconds, and the model parameters will be loaded from the parameter sharing pool to test whether the current model can effectively complete the collaborative task. If the model can complete the collaborative task well in a continuous time period, it is determined that the current asynchronous training has converged, and all processes will be terminated and the trained model will be saved.

[0033] For the parameter sharing pool: the parameter sharing pool is shared by all processes, and the training process and the test process load the model parameters from the parameter sharing pool to interact in the simulation environment; the parameter sharing pool is equipped with a mutex lock, which only allows one training process to enter the parameter pool at the same time to update the model parameters.

[0034] Another aspect of the embodiment of the present invention further provides a vehicle cooperation system for an unsignalized intersection based on asynchronous deep reinforcement learning, comprising:

[0035] The first module is used to formulate a Markov decision framework to describe the coordination problem of vehicles in the intersection area, and the system problem includes an observation state space, an action space and a reward function;

[0036] The second module is used to design a vehicle cooperation model based on a shared deep neural network architecture, wherein the content of the shared deep neural network architecture design includes a control output strategy and a value function estimation strategy;

[0037] A third module is used to train the vehicle cooperation model using an asynchronous training strategy;

[0038] The fourth module is used to process the vehicle information in the actual scene according to the trained vehicle cooperation model and output the vehicle cooperation plan for the unsignalized intersection.

[0039] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0040] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0041] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0042] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a vehicle cooperation method for unsignalized intersections based on asynchronous deep reinforcement learning, which describes the vehicle cooperation problem in the intersection area by formulating a Markov decision framework, wherein the system problem includes observation state space, action space and reward function; a vehicle cooperation model is designed based on a shared deep neural network architecture, wherein the content of the shared deep neural network architecture design includes control output strategy and value function estimation strategy; the vehicle cooperation model is trained using an asynchronous training strategy; vehicle information in the actual scene is processed according to the vehicle cooperation model after training, and a vehicle cooperation solution for unsignalized intersections is output. The embodiments of the present invention can realize effective cooperation of vehicles in unsignalized intersection scenarios within limited computing resources, reduce vehicle congestion and conflict, and improve the safety and traffic efficiency of traffic management. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0044] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0045] Figure 3 is an application scenario diagram provided by an embodiment of the present invention;

[0046] Figure 4 is the driving state quantity d provided in the embodiment of the present invention n The encoding method;

[0047] Figure 5 is a schematic diagram of a pure tracking algorithm provided by an embodiment of the present invention;

[0048] Figure 6 is a diagram of the vehicle cooperation network model architecture provided by an embodiment of the present invention;

[0049] Figure 7 It is the overall framework of the asynchronous deep reinforcement learning method provided by the embodiment of the present invention;

[0050] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0055] The vehicle cooperation method for an unsignalized intersection based on asynchronous deep reinforcement learning provided in an embodiment of the present invention relates to the field of intelligent transportation technology. The vehicle cooperation method for an unsignalized intersection based on asynchronous deep reinforcement learning provided in an embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a vehicle-mounted terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the vehicle cooperation method for an unsignalized intersection based on asynchronous deep reinforcement learning, etc., but is not limited to the above forms.

[0056] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0057] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0058] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0059] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0060] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0061] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a vehicle cooperation method at an unsignalized intersection based on asynchronous deep reinforcement learning. The following is an example of applying the vehicle cooperation method at an unsignalized intersection based on asynchronous deep reinforcement learning to a server 101. It can be understood that the method can also be applied to a terminal 102.

[0062] Reference Figure 2 , Figure 2 The flowchart of the method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning applied to a server provided in an embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0063] A Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, wherein the system problem includes observation state space, action space and reward function;

[0064] Designing a vehicle cooperation model based on a shared deep neural network architecture, wherein the shared deep neural network architecture design includes a control output strategy and a value function estimation strategy;

[0065] Training the vehicle cooperation model using an asynchronous training strategy;

[0066] The vehicle information in the actual scene is processed according to the trained vehicle cooperation model, and a vehicle cooperation plan for an unsignalized intersection is output.

[0067] In some embodiments, the method of formulating a Markov decision framework to describe the coordination problem of vehicles in an intersection area includes the following steps:

[0068] The state space is obtained by concatenating the state spaces of all vehicles, where for each vehicle's state s n , using the formula n =(x n ,y n ,vx n ,vy n ,d n ) means, where x n ,y n Indicates the horizontal and vertical coordinates of the vehicle in the intersection scene, vx n ,vy n It represents the projection speed of the vehicle's instantaneous speed on the horizontal and vertical axes, d n As a vehicle’s expression of intent;

[0069] The steering angle of the vehicle is calculated based on the pure tracking algorithm, and then the lateral movement of the vehicle is determined; the longitudinal movement is determined based on the acceleration of the vehicle, and the movement space of the vehicle is obtained by connecting the longitudinal movements of all vehicles in series;

[0070] When the vehicle performs an action that causes the state value to change, the corresponding reward value is determined according to the reward function; the reward value is composed of a collision reward, a time step reward and a success reward; wherein the collision reward is used to give a collision reward of -100 when a vehicle collides, otherwise it is 0; the time step reward is determined according to the negative value of the accumulated time step of the current simulation environment; the success reward is used to give a success reward of 100 when all vehicles successfully pass through the intersection without collision, otherwise it is 0.

[0071] In some embodiments, the formulation of a Markov decision framework to describe the coordination problem of vehicles in an intersection area further includes the following steps:

[0072] When the environment is in state value S t When the vehicles perform action A, their next state S t+1 It will be calculated by the vehicle kinematic equation, the calculation formula is: Among them, x and y represent the vehicle position coordinates, θ and v represent the vehicle body angle and speed, δ and a represent the vehicle steering angle and acceleration, and L represents the longitudinal axis length of the vehicle. and The vehicle position coordinates representing the next moment state, and The body angle and speed of the vehicle representing the state at the next moment;

[0073] The cumulative reward for an action sequence is calculated using a reward discount factor.

[0074] In some embodiments, the designing of a vehicle cooperation model based on a shared deep neural network architecture comprises the following steps:

[0075] An A2C deep neural network model is used to learn the overall decision of vehicles at intersections; the A2C deep neural network model combines an Actor model and a Critic model; the Actor model interacts by sampling actions and updates the policy using a policy gradient method; the Critic model provides feedback signals through an estimated value function to help the Actor model better update its policy;

[0076] The A2C deep neural network model is divided into a shared network and a separate network. The shared network consists of a long short-term memory network (LSTM) and two fully connected layers (FC). LSTM is used to learn time-related features in sequence data, and the fully connected layer is used to further extract spatial-related features in sequence data, and the Tanh activation function is used to perform nonlinear transformation of features between networks. The separate network uses two fully connected layers to learn the output features of the shared network and output the action control amount. The separate network uses one fully connected layer to learn the output features of the shared network and output the estimated value of the value function.

[0077] The network layers are updated using generalized advantage estimation, and the advantage function is estimated by weighted combination of multi-step advantage estimates.

[0078] During the training process, the calculation of network loss consists of policy loss and value function loss;

[0079] The gradient of the network loss is calculated by the back-propagation method, and then the parameters of the model are updated using the Adam optimizer.

[0080] In some embodiments, the network loss is calculated as follows:

[0081]

[0082] Among them, log probs (t) represents the logarithmic probability, Ent coerepresents the policy entropy regularization coefficient, Ent(t) is the policy entropy, r t is the value calculated for the reward function; GAE(t) means generalized advantage estimation; loss policy Indicates strategy loss; loss value represents the value function loss; V(s t ) represents the estimated value of the value function;

[0083] The calculation formula of generalized advantage estimation is:

[0084]

[0085] Among them, δ t represents the advantage estimate at time step t, λ is the hyperparameter for controlling bias and variance, T represents the maximum time step; γ represents the reward discount factor.

[0086] In some embodiments, the using an asynchronous training strategy to train the vehicle cooperation model comprises the following steps:

[0087] The multi-process concurrency feature of the operating system is used to implement asynchronous updates of the model. Before training begins, multiple processes will be initialized as training processes and one process will be initialized as a testing process. At the same time, a parameter sharing pool will be created to load and update model parameters in real time.

[0088] In some embodiments, for the training process: each training process separately creates a simulated intersection environment using different random number seeds; each time the simulation environment is reset, the model parameters of the vehicle cooperation model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition is reached; the characteristic sequence information of the interaction process is used to train and update the model; all training processes share a parameter sharing pool mutex lock, and when a training process is updating the model parameters, the mutex lock will be marked as blocked, preventing other training processes from updating the parameters; when the parameter update is completed, the mutex lock will be released, allowing other processes to continue to update the model parameters;

[0089] For the test process: In order to test the effectiveness and robustness of the model in real time during the training process, a process will be initialized as the test process. The intersection scene and vehicle position will be randomly initialized every 30 seconds, and the model parameters will be loaded from the parameter sharing pool to test whether the current model can effectively complete the collaborative task. If the model can complete the collaborative task well in a continuous time period, it is determined that the current asynchronous training has converged, and all processes will be terminated and the trained model will be saved.

[0090] For the parameter sharing pool: the parameter sharing pool is shared by all processes, and the training process and the test process load the model parameters from the parameter sharing pool to interact in the simulation environment; the parameter sharing pool is equipped with a mutex lock, which only allows one training process to enter the parameter pool at the same time to update the model parameters.

[0091] Another aspect of the embodiment of the present invention further provides a vehicle cooperation system for an unsignalized intersection based on asynchronous deep reinforcement learning, comprising:

[0092] The first module is used to formulate a Markov decision framework to describe the coordination problem of vehicles in the intersection area, and the system problem includes an observation state space, an action space and a reward function;

[0093] The second module is used to design a vehicle cooperation model based on a shared deep neural network architecture, wherein the content of the shared deep neural network architecture design includes a control output strategy and a value function estimation strategy;

[0094] A third module is used to train the vehicle cooperation model using an asynchronous training strategy;

[0095] The fourth module is used to process the vehicle information in the actual scene according to the trained vehicle cooperation model and output the vehicle cooperation plan for the unsignalized intersection.

[0096] The following describes the specific implementation process of the present invention in detail by taking a specific application scenario as an example:

[0097] In view of the problems existing in existing research, this paper proposes a signal-free intersection cooperation method based on asynchronous deep reinforcement learning, which can achieve effective cooperation of vehicles in signal-free intersection scenarios within limited computing resources, reduce vehicle congestion and conflict, and improve the safety and traffic efficiency of traffic management. The steps are as follows:

[0098] (1) A Markov decision framework is developed to describe the coordination problem of vehicles in the intersection area, which includes the observation state space, action space, and reward function;

[0099] (2) Design a vehicle collaboration model based on a shared deep neural network architecture, including control output strategy and value function estimation strategy;

[0100] (3) Use asynchronous training strategy to train the collaborative model.

[0101] For the Markov decision framework, specifically:

[0102] The Markov decision framework can be used to describe the interaction between the environment state and the vehicle behavior. In general, the framework can be represented by a five-tuple (S, A, R, P, γ), where S represents the global state space, A represents the action space, R represents the reward function, P represents the environment state transition probability, and γ represents the reward discount factor.

[0103] (1a) Global state space S

[0104] The state space is obtained by concatenating the state spaces of all vehicles, such as Figure 3 As shown, S = (s1, s2, s3, s4). For each vehicle’s state s n , using the formula n =(x n ,y n ,vx n ,vy n ,d n ) is represented by n ,y n Indicates the horizontal and vertical coordinates of the vehicle in the intersection scene, vx n ,vy n It represents the projection speed of the vehicle's instantaneous speed on the horizontal and vertical axes, d n As a representation of the vehicle's intention, the vehicle's driving intention at the intersection can be encoded in the following way: Figure 4 As shown, it is determined according to the driving route and position of vehicle n, d n =d1 or d n =d2.

[0105] (1b) Action space A

[0106] In the present invention, the lateral movement of the vehicle is the steering angle. Since all driving routes at the intersection are fixed, the steering angle δ of the vehicle is calculated by a pure tracking algorithm, such as Figure 5 The longitudinal motion of the vehicle is the acceleration of the vehicle, and the motion space is composed of the longitudinal motions of all vehicles in series, A = (a1, a2, a3, a4). n ∈{-2,-1,0,1,2}, unit is m / s 2 .

[0107] (1c) Reward function R

[0108] When the vehicle performs action A, the state value S t Convert to S t+1The reward that can be obtained after the intersection is composed of three parts, R = r1 + r2 + r3. r1 is the collision reward. If the vehicle collides, a collision reward of -100 will be given, otherwise it is 0; r2 is the time step reward, the value is the negative value of the accumulated time step (s) of the current simulation environment; r3 is the success reward. When all vehicles successfully pass through the intersection without collision, a success reward of 100 will be given, otherwise it is 0.

[0109] (1d) Environmental state transition P

[0110] The environment is in state value S t When the vehicles perform action A, their next state S t+1 It will be calculated by the vehicle kinematic equation. The calculation formula is as follows,

[0111]

[0112] Among them, x and y represent the vehicle position coordinates, θ and v represent the vehicle body angle and speed, δ and a represent the vehicle steering angle and acceleration, and L represents the longitudinal axis length of the vehicle.

[0113] (1e) Reward discount factor γ

[0114] It is used to calculate the cumulative reward of the action sequence and solve the immediacy problem of future rewards. It is set to 0.99.

[0115] For the vehicle cooperation network model based on deep neural network, specifically:

[0116] The present invention uses the A2C (Advantage Actor-Critic) deep neural network model to learn the overall decision-making of vehicles at intersections. The A2C model combines the Actor model and the Critic model, aiming to improve the decision-making ability of the intelligent agent by learning the strategy (Actor) and the value function (Critic). The Actor model interacts through sampling actions and updates the strategy using the policy gradient method; the Critic model provides feedback signals by estimating the value function to help the Actor model better update the strategy.

[0117] The A2C model is divided into a shared network part and a separate network part, such as Figure 6As shown. The shared network consists of a long short-term memory network (LSTM) and two fully connected layers (FC). LSTM is used to learn time-related features in sequence data, and the fully connected layer is used to further extract spatial-related features in sequence data. The Tanh activation function is used to perform nonlinear transformation of features between networks, thereby effectively performing real-time analysis and modeling of intersection scenes. The separate Actor network uses two fully connected layers to learn the output features of the shared network and output the action control amount A. The separate Critic network uses one fully connected layer to learn the output features of the shared network and output the value function estimate V.

[0118] For the A2C model, the present invention adopts generalized advantage estimation (GAE) to update the network layer. This method estimates the advantage function through a weighted combination of multi-step advantage estimation, thereby reducing the bias and variance of the advantage estimation.

[0119] δ t =r t +γV(s t+1 )-V(s t )

[0120]

[0121] Among them, δ t represents the advantage estimate at time step t, V(s t ) is the estimated value of the value function, λ is the hyperparameter for controlling the deviation and variance, and T represents the maximum time step.

[0122] The network loss calculation during training consists of two parts: strategy loss and value function loss. The calculation formula is as follows:

[0123]

[0124] Among them, log probs (t) represents the logarithmic probability, Ent coe represents the policy entropy regularization coefficient, Ent(t) is the policy entropy, r t Calculates the value for the reward function.

[0125] The present invention calculates the gradient of network loss through the back-propagation method, and then uses the Adam optimizer to update the parameters of the model.

[0126] For the asynchronous training strategy, specifically:

[0127] The present invention utilizes the multi-process concurrent characteristics of the operating system to realize asynchronous updating of the model. Before the training starts, multiple processes will be initialized as training processes, one process will be initialized as a testing process, and a parameter sharing pool will be created to load and update model parameters in real time.

[0128] Training process: Each training process creates a simulated intersection environment with different random number seeds to improve the exploration of complex environments. Each time the simulation environment is reset, the model parameters of the vehicle cooperation model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition is reached (collision occurs or the task is completed). The characteristic sequence information of the interaction process will be used to train and update the model. All training processes share a parameter sharing pool mutex. When a training process is updating the model parameters, the mutex will be marked as blocked, preventing other training processes from updating the parameters. When the parameters are updated, the mutex will be released, allowing other processes to continue to update the model parameters.

[0129] Test process: In order to test the effectiveness and robustness of the model in real time during the training process, a process will be initialized as a test process. Every 30 seconds, the intersection scene and vehicle position will be randomly initialized, and the model parameters will be loaded from the parameter sharing pool to test whether the current model can effectively complete the collaborative task. When the model can complete the collaborative task well in a continuous period of time, it is considered that the current asynchronous training has converged, and all processes can be terminated to save the trained model.

[0130] Parameter sharing pool: This shared pool is shared by all processes. Both the training process and the test process can load model parameters from the pool to interact in the simulation environment. At the same time, the pool is equipped with a mutex lock, which only allows one training process to enter the parameter pool at the same time to update the model parameters.

[0131] Based on the above technical details, the overall framework of the asynchronous deep reinforcement learning method proposed in this invention is as follows: Figure 7 shown.

[0132] In summary, compared with the prior art, the present invention has the following advantages:

[0133] First, the present invention develops a Markov decision framework for the intersection cooperation problem and develops effective environment state representation, vehicle action representation and reward function design for the vehicle cooperation task.

[0134] Second, the present invention designs a vehicle cooperation model based on a deep neural network architecture, fully considers the time complexity and spatial complexity of the intersection environment, and designs different neural processing layers, which effectively improves the computational efficiency and the quality of the solution.

[0135] Third, the present invention uses an asynchronous training method to asynchronously share and update the model, thereby improving the training speed and reliability of the model.

[0136] Another aspect of the embodiment of the present invention further provides a vehicle cooperation system for an unsignalized intersection based on asynchronous deep reinforcement learning, comprising:

[0137] The first module is used to formulate a Markov decision framework to describe the coordination problem of vehicles in the intersection area, and the system problem includes an observation state space, an action space and a reward function;

[0138] The second module is used to design a vehicle cooperation model based on a shared deep neural network architecture, wherein the content of the shared deep neural network architecture design includes a control output strategy and a value function estimation strategy;

[0139] A third module is used to train the vehicle cooperation model using an asynchronous training strategy;

[0140] The fourth module is used to process the vehicle information in the actual scene according to the trained vehicle cooperation model and output the vehicle cooperation plan for the unsignalized intersection.

[0141] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0143] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0145] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0146] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the vehicle cooperation method for an unsignalized intersection based on asynchronous deep reinforcement learning in the embodiment of the present invention;

[0147] Input / output interface 803, used to implement information input and output;

[0148] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0149] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0150] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0151] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method of vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning.

[0152] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0155] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0156] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0157] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0159] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0160] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0161] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0165] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A vehicle cooperation method at an unsignalized intersection based on asynchronous deep reinforcement learning, characterized in that: The following steps are involved: A Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, wherein the system problem includes observation state space, action space and reward function; Designing a vehicle cooperation model based on a shared deep neural network architecture, wherein the shared deep neural network architecture design includes a control output strategy and a value function estimation strategy; Training the vehicle cooperation model using an asynchronous training strategy; The vehicle information in the actual scene is processed according to the trained vehicle cooperation model, and a vehicle cooperation plan for an unsignalized intersection is output.

2. The method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning according to claim 1, characterized in that: The Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, including the following steps: The state space is obtained by concatenating the state spaces of all vehicles, where for each vehicle's state s n , using the formula n =(x n ,y n ,vx n ,vy n ,d n ) means, where x n ,y n Indicates the horizontal and vertical coordinates of the vehicle in the intersection scene, vx n ,vy n It represents the projection speed of the vehicle's instantaneous speed on the horizontal and vertical axes, d n As a vehicle’s expression of intent; The steering angle of the vehicle is calculated based on the pure tracking algorithm, and then the lateral movement of the vehicle is determined; the longitudinal movement is determined based on the acceleration of the vehicle, and the movement space of the vehicle is obtained by connecting the longitudinal movements of all vehicles in series; When the vehicle performs an action that causes the state value to change, the corresponding reward value is determined according to the reward function; the reward value is composed of a collision reward, a time step reward and a success reward; wherein the collision reward is used to give a collision reward of -100 when a vehicle collides, otherwise it is 0; the time step reward is determined according to the negative value of the accumulated time step of the current simulation environment; the success reward is used to give a success reward of 100 when all vehicles successfully pass through the intersection without collision, otherwise it is 0.

3. The method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning according to claim 2, characterized in that: The Markov decision framework is formulated to describe the coordination problem of vehicles in the intersection area, and further comprises the following steps: When the environment is in state value S t When the vehicles perform action A, their next state S t+1 It will be calculated by the vehicle kinematic equation, the calculation formula is: Among them, x and y represent the vehicle position coordinates, θ and v represent the vehicle body angle and speed, δ and a represent the vehicle steering angle and acceleration, and L represents the longitudinal axis length of the vehicle. and The vehicle position coordinates representing the next moment state, and The body angle and speed of the vehicle representing the state at the next moment; The cumulative reward for an action sequence is calculated using a reward discount factor.

4. The method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning according to claim 1, characterized in that: The method of designing a vehicle cooperation model based on a shared deep neural network architecture includes the following steps: An A2C deep neural network model is used to learn the overall decision of vehicles at intersections; the A2C deep neural network model combines an Actor model and a Critic model; the Actor model interacts by sampling actions and updates the policy using a policy gradient method; the Critic model provides feedback signals through an estimated value function to help the Actor model better update its policy; The A2C deep neural network model is divided into a shared network and a separate network. The shared network consists of a long short-term memory network LSTM and two fully connected layer networks FC. LSTM is used to learn time-related features in sequence data, and the fully connected layer is used to further extract spatial-related features in sequence data, and the Tanh activation function is used to perform nonlinear transformation of features between networks. The separate network uses two fully connected layers to learn the output features of the shared network and output the action control amount. The separate network uses one fully connected layer to learn the output features of the shared network and output the estimated value of the value function. The network layers are updated using generalized advantage estimation, and the advantage function is estimated by weighted combination of multi-step advantage estimates. During the training process, the calculation of network loss consists of policy loss and value function loss; The gradient of the network loss is calculated by the back-propagation method, and then the parameters of the model are updated using the Adam optimizer.

5. The method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning according to claim 4, characterized in that: The network loss calculation formula is: Among them, log probs (t) represents the logarithmic probability, Ent coe represents the policy entropy regularization coefficient, Ent(t) is the policy entropy, r t is the value calculated for the reward function; GAE(t) means generalized advantage estimation; loss policy Indicates strategy loss; loss value represents the value function loss; V(s t ) represents the estimated value of the value function; The calculation formula of generalized advantage estimation is: Among them, δ t represents the advantage estimate at time step t, λ is the hyperparameter for controlling bias and variance, T represents the maximum time step; γ represents the reward discount factor.

6. The method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning according to claim 1, characterized in that: The method of training the vehicle cooperation model using an asynchronous training strategy includes the following steps: The multi-process concurrency feature of the operating system is used to implement asynchronous updates of the model. Before training begins, multiple processes will be initialized as training processes and one process will be initialized as a testing process. At the same time, a parameter sharing pool will be created to load and update model parameters in real time.

7. The method for vehicle cooperation at an unsignalized intersection based on asynchronous deep reinforcement learning according to claim 6, characterized in that: For the training process: each training process creates a simulated intersection environment with different random number seeds; each time the simulation environment is reset, the model parameters of the vehicle cooperation model are loaded from the parameter sharing pool, and then continuous interaction is performed until the termination condition is reached; the characteristic sequence information of the interaction process is used to train and update the model; all training processes share a parameter sharing pool mutex lock. When a training process is updating the model parameters, the mutex lock will be marked as blocked, preventing other training processes from updating the parameters; When the parameters are updated, the mutex will be released, allowing other processes to continue updating the model parameters; For the test process: In order to test the effectiveness and robustness of the model in real time during the training process, a process will be initialized as the test process. The intersection scene and vehicle position will be randomly initialized every 30 seconds, and the model parameters will be loaded from the parameter sharing pool to test whether the current model can effectively complete the collaborative task. If the model can complete the collaborative task well in a continuous time period, it is determined that the current asynchronous training has converged, and all processes will be terminated and the trained model will be saved. For the parameter sharing pool: the parameter sharing pool is shared by all processes, and the training process and the test process load the model parameters from the parameter sharing pool to interact in the simulation environment; the parameter sharing pool is equipped with a mutex lock, which only allows one training process to enter the parameter pool at the same time to update the model parameters.

8. A vehicle cooperation system for unsignalized intersections based on asynchronous deep reinforcement learning, characterized in that: include: The first module is used to formulate a Markov decision framework to describe the coordination problem of vehicles in the intersection area, and the system problem includes an observation state space, an action space and a reward function; The second module is used to design a vehicle cooperation model based on a shared deep neural network architecture, wherein the content of the shared deep neural network architecture design includes a control output strategy and a value function estimation strategy; A third module is used to train the vehicle cooperation model using an asynchronous training strategy; The fourth module is used to process the vehicle information in the actual scene according to the trained vehicle cooperation model and output the vehicle cooperation plan for the unsignalized intersection.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images