On-ramp vehicle afflux strategy generation method, device, equipment and medium
Through the deep reinforcement learning model, the vehicle inlet strategy on the ramp is generated, which solves the traffic bottleneck problem in the ramp entrance area and realizes the intelligent management of efficient vehicle in the main road and traffic system.
Patent Information
- Application Number
- CN202411830801.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-02-28
AI Technical Summary
The ramp entrance area frequently becomes a traffic bottleneck, resulting in traffic congestion, increased accident risk and extended pass times. It is difficult for the existing technology to fundamentally achieve efficient and intelligent traffic management.
The deep reinforcement learning model is adopted to obtain preset traffic data, build state space and action space, and train the strategy network and value network to generate vehicle inlet strategies on the ramp, and optimize vehicle inlet control.
It realizes that vehicles are efficient and smoothly transported into the main road, effectively alleviating traffic bottlenecks, and fundamentally improving the overall performance and driving experience of the transportation system.
Smart Images
Figure CN119541214A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent transportation, and in particular to a method, device, equipment and medium for generating a merging strategy for vehicles entering a ramp. Background Art
[0002] Nowadays, urban and inter-city transportation networks, especially viaducts and highway systems, are important arteries connecting major cities and economic regions. Their traffic efficiency directly affects the speed of economic operation and the quality of life of the people. However, in these efficient transportation systems, the ramp entrance area frequently becomes a traffic bottleneck, especially during peak hours. The process of vehicles merging into the main road often leads to traffic congestion, increased accident risks and extended travel time, which seriously affects the overall road capacity and driving experience.
[0003] As a typical bottleneck of elevated roads and highways, the problem with ramp entrances is that limited road space must meet the needs of a large number of vehicles merging into the main road quickly and safely. Traditional methods to solve such problems rely on measures such as traffic signal control, increasing the number of lanes or optimizing road design, but these methods are often limited by factors such as land resources, construction costs and environmental impact, making it difficult to fundamentally achieve efficient and intelligent traffic management. Summary of the invention
[0004] In view of this, the purpose of this application is to provide a method, device, equipment and medium for generating a vehicle merging strategy for entering a ramp, which can enable vehicles to merge into the main road efficiently and smoothly, effectively alleviate traffic bottlenecks, fundamentally realize efficient and intelligent traffic management, and improve the overall performance of the traffic system. The specific scheme is as follows:
[0005] In a first aspect, the present application provides a method for generating a ramp vehicle merging strategy, comprising:
[0006] Acquire preset traffic data, determine a state space set and an action space set based on the preset traffic data, and construct a deep reinforcement learning model; the state space set includes the vehicle's own state and the surrounding environment state; the action space set includes the vehicle's executable actions; the deep reinforcement learning model includes a policy network and a value network;
[0007] For the current state in the state space set, using the current policy network in the deep reinforcement learning model to generate probabilities corresponding to each of the executable actions in the action space set, and using the executable action with the highest probability as the execution action of the target vehicle in the current state, so as to obtain a current state-action pair;
[0008] The current expected reward of the current state-action pair is determined according to the current value network in the deep reinforcement learning model, and the current policy network and the current value network are trained based on the current expected reward until both policy network parameters and value network parameters satisfy preset stability conditions, and a corresponding on-ramp vehicle merging strategy is determined based on the latest current policy network, so as to perform corresponding vehicle merging control using the on-ramp vehicle merging strategy.
[0009] Optionally, the obtaining of preset traffic data includes:
[0010] Acquire preset traffic data and perform data preprocessing operations on the preset traffic data; the preset traffic data includes vehicle driving conditions, road conditions, weather conditions, and traffic signals; the data preprocessing operations include data cleaning, data standardization, data conversion, and data segmentation.
[0011] Optionally, the method for generating a ramp vehicle merging strategy further includes:
[0012] Creating a target reward function so that the current value network determines the current expected return of the current state-action pair using the target reward function;
[0013] The target reward function is a function composed of a preset driving safety function, a preset time efficiency function, and a preset riding comfort function.
[0014] Optionally, the value network in the deep reinforcement learning model includes a first value network and a second value network;
[0015] Accordingly, determining the current expected return of the current state-action pair according to the current value network includes:
[0016] Based on the current first value network and the current second value network, the current state-action pair is evaluated respectively to obtain the current first expected return and the current second expected return;
[0017] An average of the current first expected reward and the current second expected reward is determined, and the average is used as the current expected reward of the current state-action pair.
[0018] Optionally, training the current value network based on the current expected return includes:
[0019] The current first value network is trained using the current first expected reward, and the current second value network is trained using the current second expected reward.
[0020] Optionally, the method for generating a ramp vehicle merging strategy further includes:
[0021] Verify the on-ramp vehicle merging strategy network in a controlled environment with vehicle-road collaboration capabilities to obtain verification results;
[0022] If the verification result indicates that the verification is passed, the on-ramp vehicle merging strategy network is deployed to the actual traffic system, and the on-ramp vehicle merging strategy network is continuously optimized according to the actual operation feedback result;
[0023] If the verification result indicates that the verification has not passed, the on-ramp vehicle merging strategy network continues to be trained.
[0024] Optionally, the preset traffic data is simulated traffic data obtained through data simulation;
[0025] Furthermore, before training the policy network in the deep reinforcement learning model based on the preset traffic data, it also includes:
[0026] Obtain real traffic data;
[0027] The policy network in the deep reinforcement learning model is initially trained using the real traffic data to obtain an initially trained deep reinforcement learning model, so that the policy network in the initially trained deep reinforcement learning model can be further trained based on the preset traffic data.
[0028] In a second aspect, the present application provides a device for generating a ramp vehicle merging strategy, comprising:
[0029] A model building module, used to obtain preset traffic data, determine a state space set and an action space set based on the preset traffic data, and build a deep reinforcement learning model; the state space set includes the vehicle's own state and the surrounding environment state; the action space set includes the vehicle's executable actions; the deep reinforcement learning model includes a policy network and a value network;
[0030] an action determination module, for generating, for a current state in the state space set, a probability corresponding to each of the executable actions in the action space set using the current policy network in the deep reinforcement learning model, and taking the executable action with the highest probability as the execution action of the target vehicle in the current state, so as to obtain a current state action pair;
[0031] A policy network training module is used to determine the current expected return of the current state-action pair according to the current value network in the deep reinforcement learning model, and train the current policy network and the current value network based on the current expected return until both the policy network parameters and the value network parameters meet the preset stability conditions, and determine the corresponding on-ramp vehicle merging strategy based on the latest current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control.
[0032] In a third aspect, the present application provides an electronic device, including:
[0033] Memory, used to store computer programs;
[0034] The processor is used to execute the computer program to implement the aforementioned method for generating a vehicle merging strategy for entering a ramp.
[0035] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned method for generating a ramp vehicle merging strategy.
[0036] In the present application, preset traffic data is obtained, a state space set and an action space set are determined based on the preset traffic data, and a deep reinforcement learning model is constructed; the state space set includes the vehicle's own state and the surrounding environment state; the action space set includes the vehicle's executable actions; the deep reinforcement learning model includes a policy network and a value network; for the current state in the state space set, the current policy network in the deep reinforcement learning model is used to generate probabilities corresponding to each of the executable actions in the action space set, and the executable action with the highest probability is used as the executable action of the target vehicle in the current state to obtain a current state-action pair; the current expected return of the current state-action pair is determined according to the current value network in the deep reinforcement learning model, and the current policy network and the current value network are trained based on the current expected return until both the policy network parameters and the value network parameters meet the preset stability conditions, and the corresponding on-ramp vehicle merging strategy is determined based on the latest current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control. As can be seen from the above, the present application constructs a deep reinforcement learning model, and based on the preset traffic data obtained by the deep reinforcement learning model, continuously trains the policy network and the value network in the deep reinforcement learning model until the policy network parameters and the value network parameters meet the preset stability conditions, and obtains the latest current policy network after training, and determines the corresponding on-ramp vehicle merging strategy based on the latest current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control, so that vehicles can merge into the main road efficiently and smoothly, effectively alleviate traffic bottlenecks, fundamentally realize efficient and intelligent traffic management, and improve the overall performance of the transportation system. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0038] Figure 1 A flow chart of a ramp-entering vehicle merging strategy training method disclosed in this application;
[0039] Figure 2 A schematic diagram of a specific on-ramp vehicle merging strategy training method disclosed in this application;
[0040] Figure 3 This is a schematic diagram of the structure of a ramp-entering vehicle merging strategy training device disclosed in the present application;
[0041] Figure 4This is a schematic diagram of the structure of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0043] Nowadays, urban and inter-city transportation networks, especially viaducts and highway systems, serve as important arteries connecting major cities and economic regions. However, in these efficient transportation systems, the ramp entrance area frequently becomes a traffic bottleneck, especially during peak hours. The process of vehicles merging into the main road often leads to traffic congestion, increased accident risks, etc., which seriously affects the overall road capacity and driving experience; the ramp entrance is a typical bottleneck of elevated roads and highways. The problem is that the limited road space needs to meet the needs of a large number of vehicles merging into the main road quickly and safely. In traditional methods, solving such problems mostly relies on measures such as traffic signal control, increasing the number of lanes or optimizing road design, but these methods are often limited by factors such as land resources, construction costs and environmental impacts, and it is difficult to fundamentally achieve efficient and intelligent traffic management. To this end, the present application provides a method for generating a vehicle merging strategy for entering a ramp, which can fundamentally achieve efficient and intelligent traffic management and improve the overall performance of the transportation system.
[0044] See also Figure 1 As shown, an embodiment of the present invention discloses a method for generating a ramp vehicle merging strategy, comprising:
[0045] Step S11, obtaining preset traffic data, determining a state space set and an action space set based on the preset traffic data, and constructing a deep reinforcement learning model; the state space set includes the vehicle's own state and the surrounding environment state; the action space set includes the vehicle's executable actions; the deep reinforcement learning model includes a policy network and a value network.
[0046] In this embodiment, preset traffic data is first obtained, and the preset traffic data is simulated traffic data obtained through data simulation. For example, a deep reinforcement learning model training environment is constructed using SUMO (Simulation of Urban MObility) traffic simulation software and Python (Python Programming Language, a high-level programming language) programming language, and data simulation operations are performed based on SUMO to obtain simulated traffic data. The above-mentioned simulated traffic data includes but is not limited to vehicle driving conditions, road conditions, weather conditions, and traffic signals.
[0047] Furthermore, a deep reinforcement learning model is constructed.
[0048] Specifically, the SAC (Soft Actor-Critic) algorithm is used as the core architecture of the deep reinforcement learning model. The SAC algorithm combines the policy gradient method and maximum entropy reinforcement learning, aiming to learn a strategy that maximizes the expected return and is as random as possible, so as to enhance the exploration ability of the deep reinforcement learning model and avoid the model falling into the local optimal solution too early.
[0049] The state space set and action space set are determined using preset traffic data. The state space set includes the vehicle's own state (such as speed, position), surrounding environment state (such as neighboring vehicle state, road conditions), etc. The action space set includes the vehicle's executable actions, such as acceleration, deceleration, maintaining speed, steering wheel steering angle, etc.
[0050] A policy network is constructed based on a deep neural network (DNN). The policy network generates an action probability distribution according to the current state in the above state space set, that is, it outputs each executable action and its corresponding probability. For example, for the current state, the probability of acceleration is 0.3, the probability of deceleration is 0.2, the probability of maintaining speed is 0.3, the probability of slight steering adjustment is 0.1, and the probability of lane change is 0.1. The sum of these probabilities is 1. It should be noted that in actual situations, state information is complex and diverse, and may be a vector containing many variables. Therefore, it is necessary to ensure that the policy network can handle multi-dimensional inputs. For example, the state of a vehicle may include the vehicle's own three-dimensional position coordinates, velocity vector, position and velocity information of multiple surrounding vehicles, etc. These information are combined to form a high-dimensional input vector. It should be ensured that the policy network can process the above high-dimensional input vector to make decisions. On the other hand, the action space set defines all possible executable actions of the vehicle and whether these actions are continuous or discrete. For merging actions, such as acceleration, deceleration, lane change, etc., they are discrete, while for steering adjustment actions, they may be continuous within a certain range. The output of the policy network needs to conform to the definition of the action space, that is, for discrete actions, the probability of each action is output, and for continuous actions, the output conforms to the result of the continuous probability distribution.
[0051] Based on DNN, two value networks, namely Q networks, are constructed to evaluate the expected return of the current state-action pair, namely the Q value; the two value networks are trained independently to improve the robustness and generalization ability of the model.
[0052] Create a target reward function to encourage safe, efficient, and comfortable ramp merging behavior, and enable the value network to use the target reward function to determine the expected return of the state-action pair; the target reward function can be composed of a preset driving safety function, a preset time efficiency function, and a preset ride comfort function. The function value of the target reward function is the target reward value, and the target reward value is used as the expected return of the state-action pair. For example, the target reward function is as follows:
[0053] ;
[0054] in, represents the target reward function; t represents the current time; represents a preset driving safety function; represents the preset time efficiency function; Represents a preset ride comfort function; the target reward value is jointly constituted by a function value based on a preset driving safety function, a function value based on a preset time efficiency function, and a function value based on a preset ride comfort function.
[0055] The function value of the preset driving safety function can be measured by detecting the potential collision risk. For example, create a function , which returns the minimum distance between the vehicle and the nearest obstacle, such as other vehicles, at the current time t. The preset driving safety function is as follows:
[0056] ;
[0057] in, represents the weight coefficient; It represents the set safety distance threshold. When the minimum distance is less than the safety distance threshold, the function value returned by the preset driving safety function is negative to encourage collision avoidance. When the minimum distance is not less than the safety distance threshold, the function value returned by the preset driving safety function is zero.
[0058] The function value of the preset time efficiency function can be measured by minimizing the total time it takes for a vehicle to enter the main road from the ramp entrance. is the total time required for the vehicle to complete the merge, the preset time efficiency function is as follows:
[0059] ;
[0060] in, Represents the weight coefficient, which is used to adjust the proportion of time efficiency in the target reward value.
[0061] The function value of the preset ride comfort function can be measured by the change of acceleration. is the acceleration of the vehicle at the current time t, the preset ride comfort function is as follows:
[0062] ;
[0063] in, Represents the weight coefficient, which is used to adjust the proportion of ride comfort in the target reward. The smaller the acceleration change, the higher the reward value to encourage smooth vehicle driving behavior; Represents a small change in time. Indicates a small change in acceleration over time; | | indicates the absolute value symbol.
[0064] After building the deep reinforcement learning model, it is necessary to train the policy network in the deep reinforcement learning model based on the preset traffic data. It should be noted that before training the policy network in the deep reinforcement learning model based on the preset traffic data, the policy network in the deep reinforcement learning model can be initially trained based on the initial traffic data. Specifically, first, real traffic data can be obtained as the initial traffic data, and then the policy network in the deep reinforcement learning model can be initially trained using the real traffic data to obtain a deep reinforcement learning model after initial training, so that the policy network in the deep reinforcement learning model after initial training can be further trained based on the preset traffic data; wherein, the NGSIM (Next Generation Simulation, a highway driving data set) data set or real-time data obtained from the vehicle-road cooperative system (V2X) including but not limited to vehicle direction, road status (such as lane width, speed limit, traffic density), weather conditions, traffic signals, etc. can be used as real traffic data.
[0065] After obtaining the preset traffic data, data preprocessing operations can be performed on the preset traffic data, such as data cleaning, data standardization, data conversion, data segmentation, etc.; accordingly, before performing initial training on the policy network in the deep reinforcement learning model based on the initial traffic data, data preprocessing operations can also be performed on the initial traffic data to ensure that the quality and format of the input data meet the training requirements, so as to provide high-quality, adaptive data for the policy network training that helps to improve the training effect and generalization ability.
[0066] Step S12: for the current state in the state space set, use the current policy network in the deep reinforcement learning model to generate the probabilities corresponding to the executable actions in the action space set, and use the executable action with the highest probability as the execution action of the target vehicle in the current state to obtain a current state-action pair.
[0067] In this embodiment, the policy network in the deep reinforcement learning model is trained based on the preset traffic data. Specifically, after the deep reinforcement learning model is constructed, the current policy network in the deep reinforcement learning model receives relevant information, such as vehicle speed, surrounding environment, etc., for the current state obtained from the state space set, and generates corresponding probabilities for each executable action in the action space set based on the current policy network parameters and modes that have been learned by itself. For example, for the current state, the probability of acceleration is 0.3, the probability of deceleration is 0.2, the probability of maintaining speed is 0.4, and the probability of changing lanes is 0.1; after obtaining the probabilities corresponding to each executable action, the executable action with the highest probability is selected as the action to be actually performed by the target vehicle in the current state. For example, if the probability of maintaining speed is the highest, maintaining speed is used as the action to be actually performed by the target vehicle in the current state. The current state and the determined execution action are combined together to form a "current state action pair", and the expected return of the state action pair is subsequently evaluated through the value network to continuously optimize the policy network, so that the target vehicle can make better decisions in different states and achieve better learning effects.
[0068] Step S13: determining the current expected reward of the current state-action pair according to the current value network in the deep reinforcement learning model, and training the current policy network and the current value network based on the current expected reward until both policy network parameters and value network parameters meet preset stability conditions, and determining a corresponding on-ramp vehicle merging strategy based on the latest current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control.
[0069] In this embodiment, the current value network uses the target reward function to determine the target reward value of the current state-action pair, and uses the above target reward value as the expected return of the current state-action pair. The policy network and the value network are continuously trained using the expected return of the current state-action pair, and the policy network parameters and the value network parameters are continuously iterated and updated during the training process until the policy network parameters and the value network parameters meet the preset stability conditions, and then the current policy network and the current value network are stopped from being trained; wherein, the preset stability conditions are used to measure whether the policy network and the value network are trained, and the preset stability conditions can be that the policy network parameters and the value network parameters tend to a better value; it should be noted that if the action probability distribution generated by the policy network in the same state changes very little after multiple rounds of iterations, or the expected return fluctuation of the value network for the same state-action pair is very small, the current policy network and the current value network are also considered to be trained, and the current policy network and the current value network are stopped from being trained.
[0070] It can be seen from step S11 that two value networks are constructed based on DNN, that is, the value network in the deep reinforcement learning model includes a first value network and a second value network; accordingly, determining the current expected return of the current state-action pair based on the current value network can include: first, evaluating the current state-action pair based on the current first value network and the current second value network, respectively, to obtain the current first expected return and the current second expected return; then determining the average of the current first expected return and the current second expected return, and using the average as the current expected return of the current state-action pair; further, using the current first expected return to train the current first value network, and using the current second expected return to train the current second value network.
[0071] This embodiment implements deep reinforcement learning model training in Python based on the SAC method. By continuously trying in the SUMO environment and receiving feedback from the environment, the policy network parameters and value network parameters are gradually optimized to obtain the optimized current policy network, and the corresponding on-ramp vehicle merging strategy is determined based on the optimized current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control. In the early stage of deep reinforcement learning model training, a higher exploration rate can be set for the model to encourage the model to widely explore different behaviors in the SUMO environment to promote a comprehensive understanding of the environment; as the training process deepens, the exploration rate can be gradually reduced to increase the efficiency of using existing knowledge to formulate better strategies.
[0072] After the policy network training in the deep reinforcement learning model is completed, the trained policy network can be tested offline in a simulation environment to use the target reward function in step S11 to comprehensively evaluate the performance of the on-ramp vehicle merging strategy determined by the current policy network under different traffic conditions and scenarios from multiple dimensions to ensure its theoretical stability and efficiency.
[0073] Furthermore, the strategy network is field tested. This step not only requires the test site to support vehicle-road communication, but also requires continuous collection of actual operation data during the test, which is used as a basis to further verify the effectiveness and security of the strategy network and ensure its feasibility and reliability in the real environment. Specifically, the on-ramp vehicle merging strategy network is verified in a controlled environment with vehicle-road collaboration capabilities to obtain verification results; if the verification result indicates that the verification is passed, the on-ramp vehicle merging strategy network is deployed to the actual traffic system, and the on-ramp vehicle merging strategy network is continuously optimized based on the actual operation feedback results, forming a continuous iterative cycle to ensure that the strategy can continuously adapt to the changing traffic environment and always maintain its optimal performance; if the verification result indicates that the verification is not passed, the on-ramp vehicle merging strategy network continues to be trained.
[0074] For example Figure 2 As shown, this embodiment first performs data collection and processing, that is, obtains initial traffic data and performs data cleaning, data standardization, data conversion, data segmentation and other processing on the initial traffic data; then constructs a deep reinforcement learning model, including defining the model architecture, state space, action space, policy network, value network, and reward function; further performs model training and optimization, that is, uses SUMO to build a simulation environment, and performs policy optimization based on the SAC algorithm, that is, trains the policy network in the deep reinforcement learning model, and determines the corresponding on-ramp vehicle merging strategy based on the trained policy network; finally, the on-ramp vehicle merging strategy is verified and deployed.
[0075] As can be seen from the above, the core of this embodiment is to build a deep reinforcement learning model that can receive multi-dimensional input data from the vehicle-road cooperative system. In this way, by building a deep reinforcement learning model and simulating various traffic scenarios and conditions based on real traffic data or simulated traffic data generated by SUMO, the policy network and value network in the deep reinforcement learning model are continuously trained until the policy network parameters and the value network parameters meet the preset stability conditions, and the latest current policy network after training is obtained. Based on the latest current policy network, the corresponding on-ramp vehicle merging strategy is determined, which is used to improve the efficiency of ramp merging, enhance traffic safety, and achieve optimal allocation of traffic resources. The strategy performs well in many aspects such as safety, comfort and efficiency. The strategy is used to perform corresponding vehicle merging control, so that the vehicle merges into the main road efficiently and smoothly, effectively alleviating traffic bottlenecks, fundamentally realizing efficient and intelligent traffic management, and improving the overall performance of the transportation system. At the same time, the on-ramp vehicle merging strategy generated in this embodiment has significant intelligent adaptability, efficiency, safety and scalability, which improves the overall performance and safety of the transportation system.
[0076] See also Figure 3 As shown, the embodiment of the present application also discloses a device for generating a vehicle merging strategy for entering a ramp, comprising:
[0077] The model building module 11 is used to obtain preset traffic data, determine a state space set and an action space set based on the preset traffic data, and build a deep reinforcement learning model; the state space set includes the vehicle's own state and the surrounding environment state; the action space set includes the vehicle's executable actions; the deep reinforcement learning model includes a policy network and a value network;
[0078] An action determination module 12 is used to generate a probability corresponding to each executable action in the action space set using the current policy network in the deep reinforcement learning model for the current state in the state space set, and use the executable action with the highest probability as the execution action of the target vehicle in the current state to obtain a current state action pair;
[0079] The policy network training module 13 is used to determine the current expected return of the current state-action pair according to the current value network in the deep reinforcement learning model, and train the current policy network and the current value network based on the current expected return until both the policy network parameters and the value network parameters meet the preset stability conditions, and determine the corresponding on-ramp vehicle merging strategy based on the latest current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control.
[0080] As can be seen from the above, the present application constructs a deep reinforcement learning model, and based on the preset traffic data obtained by the deep reinforcement learning model, continuously trains the policy network and the value network in the deep reinforcement learning model until the policy network parameters and the value network parameters meet the preset stability conditions, and obtains the latest current policy network after training, and determines the corresponding on-ramp vehicle merging strategy based on the latest current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control, so that vehicles can merge into the main road efficiently and smoothly, effectively alleviate traffic bottlenecks, fundamentally realize efficient and intelligent traffic management, and improve the overall performance of the transportation system.
[0081] In some specific embodiments, the model building module 11 includes:
[0082] The data processing unit is used to obtain preset traffic data and perform data preprocessing operations on the preset traffic data; the preset traffic data includes vehicle driving conditions, road conditions, weather conditions, and traffic signals; the data preprocessing operations include data cleaning, data standardization, data conversion, and data segmentation.
[0083] In some specific implementations, the ramp-entering vehicle merging strategy generating device further includes:
[0084] A function creation unit, configured to create a target reward function so that the current value network determines a current expected return of a current state-action pair using the target reward function;
[0085] The target reward function is a function composed of a preset driving safety function, a preset time efficiency function, and a preset riding comfort function.
[0086] In some specific embodiments, the value network in the deep reinforcement learning model includes a first value network and a second value network;
[0087] Accordingly, the strategy network training module 13 includes:
[0088] A first reward determination unit, configured to evaluate the current state-action pair based on the current first value network and the current second value network, respectively, to obtain a current first expected reward and a current second expected reward;
[0089] The second reward determination unit is used to determine an average value of the current first expected reward and the current second expected reward, and use the average value as the current expected reward of the current state-action pair.
[0090] In some specific implementations, the strategy network training module 13 includes:
[0091] A network training unit is used to train the current first value network using the current first expected return, and to train the current second value network using the current second expected return.
[0092] In some specific implementations, the ramp-entering vehicle merging strategy generating device further includes:
[0093] A verification unit, used to verify the merging strategy network of on-ramp vehicles in a controlled environment with vehicle-road collaboration capabilities to obtain a verification result;
[0094] A first processing unit is configured to deploy the on-ramp vehicle merging strategy network into an actual traffic system if the verification result indicates that the verification is passed, and continuously optimize the on-ramp vehicle merging strategy network according to actual operation feedback results;
[0095] The second processing unit is configured to continue training the on-ramp vehicle merging strategy network if the verification result indicates that the verification has failed.
[0096] In some specific implementations, the preset traffic data is simulated traffic data obtained through data simulation;
[0097] Furthermore, before training the policy network in the deep reinforcement learning model based on the preset traffic data, it also includes:
[0098] A data acquisition unit, used for acquiring real traffic data;
[0099] A model training unit is used to use the real traffic data to perform initial training on the policy network in the deep reinforcement learning model to obtain the deep reinforcement learning model after initial training, so as to continue training the policy network in the deep reinforcement learning model after initial training based on the preset traffic data.
[0100] Furthermore, the present application also discloses an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0101] Figure 4 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the ramp vehicle merging strategy training method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0102] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0103] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0104] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the on-ramp vehicle merging strategy training method executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.
[0105] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed method for training a vehicle merging strategy for entering a ramp is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.
[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0107] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0108] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0109] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0110] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for generating a vehicle merging strategy for entering a ramp, characterized in that: include: Acquire preset traffic data, determine a state space set and an action space set based on the preset traffic data, and construct a deep reinforcement learning model; the state space set includes the vehicle's own state and the surrounding environment state; the action space set includes the vehicle's executable actions; the deep reinforcement learning model includes a policy network and a value network; For the current state in the state space set, using the current policy network in the deep reinforcement learning model to generate probabilities corresponding to each of the executable actions in the action space set, and using the executable action with the highest probability as the execution action of the target vehicle in the current state, so as to obtain a current state-action pair; The current expected reward of the current state-action pair is determined according to the current value network in the deep reinforcement learning model, and the current policy network and the current value network are trained based on the current expected reward until both policy network parameters and value network parameters satisfy preset stability conditions, and a corresponding on-ramp vehicle merging strategy is determined based on the latest current policy network, so as to perform corresponding vehicle merging control using the on-ramp vehicle merging strategy.
2. The method for generating a ramp vehicle merging strategy according to claim 1, characterized in that: The obtaining of preset traffic data includes: Acquire preset traffic data and perform data preprocessing operations on the preset traffic data; the preset traffic data includes vehicle driving conditions, road conditions, weather conditions, and traffic signals; the data preprocessing operations include data cleaning, data standardization, data conversion, and data segmentation.
3. The method for generating a ramp vehicle merging strategy according to claim 1, characterized in that: Also includes: Creating a target reward function so that the current value network determines the current expected return of the current state-action pair using the target reward function; The target reward function is a function composed of a preset driving safety function, a preset time efficiency function, and a preset riding comfort function.
4. The method for generating a ramp vehicle merging strategy according to claim 1, characterized in that: The value network in the deep reinforcement learning model includes a first value network and a second value network; Accordingly, determining the current expected return of the current state-action pair according to the current value network includes: Based on the current first value network and the current second value network, the current state-action pair is evaluated respectively to obtain the current first expected return and the current second expected return; An average of the current first expected reward and the current second expected reward is determined, and the average is used as the current expected reward of the current state-action pair.
5. The method for generating a ramp vehicle merging strategy according to claim 4, characterized in that: Training the current value network based on the current expected return includes: The current first value network is trained using the current first expected reward, and the current second value network is trained using the current second expected reward.
6. The method for generating a ramp vehicle merging strategy according to any one of claims 1 to 5, characterized in that: Also includes: Verify the on-ramp vehicle merging strategy network in a controlled environment with vehicle-road collaboration capabilities to obtain verification results; If the verification result indicates that the verification is passed, the on-ramp vehicle merging strategy network is deployed to the actual traffic system, and the on-ramp vehicle merging strategy network is continuously optimized according to the actual operation feedback result; If the verification result indicates that the verification has not passed, the on-ramp vehicle merging strategy network continues to be trained.
7. The method for generating a ramp vehicle merging strategy according to claim 1, characterized in that: The preset traffic data is simulated traffic data obtained through data simulation; Furthermore, before training the policy network in the deep reinforcement learning model based on the preset traffic data, it also includes: Obtain real traffic data; The policy network in the deep reinforcement learning model is initially trained using the real traffic data to obtain an initially trained deep reinforcement learning model, so that the policy network in the initially trained deep reinforcement learning model can be further trained based on the preset traffic data.
8. A vehicle merging strategy training device for entering a ramp, characterized in that: include: A model building module, used to obtain preset traffic data, determine a state space set and an action space set based on the preset traffic data, and build a deep reinforcement learning model; the state space set includes the vehicle's own state and the surrounding environment state; the action space set includes the vehicle's executable actions; the deep reinforcement learning model includes a policy network and a value network; an action determination module, for generating, for a current state in the state space set, a probability corresponding to each of the executable actions in the action space set using the current policy network in the deep reinforcement learning model, and taking the executable action with the highest probability as the execution action of the target vehicle in the current state, so as to obtain a current state action pair; A policy network training module is used to determine the current expected return of the current state-action pair according to the current value network in the deep reinforcement learning model, and train the current policy network and the current value network based on the current expected return until both the policy network parameters and the value network parameters meet the preset stability conditions, and determine the corresponding on-ramp vehicle merging strategy based on the latest current policy network, so as to use the on-ramp vehicle merging strategy to perform corresponding vehicle merging control.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the method for generating a ramp vehicle merging strategy as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the method for generating a ramp vehicle merging strategy as described in any one of claims 1 to 7.