Ramp merge strategy learning method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a ramp merging strategy learning method and device, electronic equipment and storage medium, relating to the technical field of autonomous driving. Among them, the strategy learning method comprises: configuring cumulative expected return in the training environment, and the cumulative expected return is used to represent the cumulative value of the expected return of continuously executing the merging strategy to complete the ramp merging process; constructing a strategy evaluation model and a strategy improvement model based on the cumulative expected return, the strategy evaluation model is used to evaluate the first expected return obtained by following the merging strategy from the specified driving state, and the strategy improvement model is used to evaluate the second expected return obtained by adopting the specified driving action; alternately iteratively learning the strategy evaluation model and the strategy improvement model to obtain a distributed driving action; and performing collision avoidance correction operation on the distributed driving action based on a safety shield model to generate a corrected target merging strategy. Through the technical scheme of the present disclosure, the safety of the output strategy can be improved when applied to the scene of merging onto the highway from the ramp.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of automatic driving, and particularly relates to a ramp merging strategy learning method and device, electronic equipment and a computer readable storage medium. BACKGROUND

[0002] Automobile intelligent technology capable of effectively improving driving performance has become a main development trend of the automobile industry in recent years, and autonomous decision-making is one of the core technologies of intelligent technology.

[0003] Since the ramp is an important part of the road, in order to realize automatic driving of structured roads, autonomous lane changing between the ramp and the main road needs to be realized, and due to the dynamic, random and game nature of the ramp merging scene, ramp merging has become one of the road sections with high incidence of vehicle accidents, and intelligent vehicle autonomous merging decision is a key technology to realize structured autonomous lane changing automatic driving, and the success rate and safety of merging in a dynamic environment are main difficulties of autonomous decision-making.

[0004] In related technologies, a reinforcement learning algorithm is applied to lane changing and upper-level behavior decision-making in the scene of merging from a ramp to a highway, which has the potential to surpass the level of human drivers and is an effective method to solve complex decision-making problems. Since there is a contradiction between exploration and trial learning and state safety in the scheme, it has the defect of low safety when applied to the scene of merging from a ramp to a highway.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the present disclosure is to provide a ramp merging strategy learning method and device, electronic equipment and a computer readable storage medium, which can at least partially improve the problem of low safety when the automatic driving merging strategy in related technologies is applied to the scene of merging from a ramp to a highway.

[0007] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a ramp merging strategy learning method is provided, including: configuring a cumulative expected return in a training environment, the cumulative expected return being used to represent a cumulative value of expected returns of continuously executing a merging strategy to complete a ramp merging process; constructing a strategy evaluation model and a strategy improvement model based on the cumulative expected return, the strategy evaluation model being used to evaluate a first expected return of following the merging strategy from a specified driving state, and the strategy improvement model being used to evaluate a second expected return of adopting a specified driving action, the specified driving action being a driving action selected based on the merging strategy in the specified driving state; performing alternating iterative learning on the strategy evaluation model and the strategy improvement model with a strategy optimization target of optimizing the cumulative expected return, and obtaining a to-be-corrected merging strategy based on a result of the alternating iterative learning, the to-be-corrected merging strategy being used to output a distributed driving action in ramp merging; and performing collision avoidance correction operation on the distributed driving action based on a safety shield model in an application environment to generate a corrected target merging strategy, so that a vehicle performs ramp merging operation based on the target merging strategy.

[0009] In one embodiment, the configuring a cumulative expected return in a training environment includes: obtaining an immediate return model for safety state evaluation and a strategy entropy model of the merging strategy, the immediate return model being used to evaluate safety of entering a corresponding driving state by executing the merging strategy, and the strategy entropy model being used to measure uncertainty of the merging strategy; and configuring the cumulative expected return based on the immediate return model and the strategy entropy model.

[0010] In one embodiment, the performing alternating iterative learning on the strategy evaluation model and the strategy improvement model with a strategy optimization target of optimizing the cumulative expected return includes: obtaining a first association relationship between the strategy improvement model and the strategy evaluation model; and performing alternating iterative learning on the strategy evaluation model and the strategy improvement model based on the first association relationship and with the strategy optimization target of maximizing the cumulative expected return.

[0011] In one embodiment, the performing alternating iterative learning on the strategy evaluation model and the strategy improvement model based on the first association relationship and with the strategy optimization target of maximizing the cumulative expected return includes: configuring a corresponding dynamic programming equation based on the first association relationship, the dynamic programming equation being used to express a relationship between expected returns between adjacent two driving states; and alternately calculating a strategy evaluation result output by the strategy evaluation model and an improved strategy output by the strategy improvement model based on the dynamic programming equation, wherein the improved strategy is adapted to maximize the corresponding strategy evaluation result of the distributed driving action under an initial distribution.

[0012] In an embodiment, the cumulative expected return is configured with a temperature coefficient adapted to adaptively adjust a speed of convergence of the stochastic policy to the deterministic policy, the policy evaluation model is configured to output a policy evaluation result based on the dynamic programming equation, and the policy improvement model is configured to output an improved policy based on the dynamic programming equation, and the method further comprises: determining the deterministic policy as the to-be-corrected merge-in policy, wherein the improved policy converges to the deterministic policy from the stochastic policy.

[0013] In an embodiment, before the policy evaluation result and the improved policy are alternately calculated based on the dynamic programming equation, the method further comprises: interacting with a deployed driving environment using the merge-in policy from the specified driving state, to generate a sequence comprising the driving state, the driving action, and an immediate reward, to determine a discounted cumulative return of the immediate reward based on the sequence; determining a second association between the discounted cumulative return and the policy improvement model; modeling a distribution of the discounted cumulative return based on the second association to obtain a policy improvement distribution model, the policy improvement distribution model being adapted to optimize the policy evaluation result; and configuring the dynamic programming equation based on the policy improvement distribution model, to alternately calculate the policy evaluation result and the improved policy based on the dynamic programming equation.

[0014] In an embodiment, the alternately calculating the policy evaluation result and the improved policy comprises: parameterizing the improved policy to obtain a parameterized policy; parameterizing the policy improvement model and the policy evaluation model to obtain a parameterized model; and alternately updating the first expected return and the second expected return based on updates to parameters in the parameterized policy and the parameterized model, to output the to-be-corrected merge-in policy.

[0015] In an embodiment, the parameterizing the improved policy to obtain a parameterized policy comprises: parameterizing the merge-in policy using a mean and a variance to obtain the parameterized policy comprising policy parameters.

[0016] In an embodiment, the parameterizing the policy improvement model and the policy evaluation model to obtain a parameterized model comprises: representing a value function distribution of the policy improvement model based on the policy improvement distribution model; and approximating the policy improvement distribution model based on a neural network to obtain the parameterized model comprising neural network parameters.

[0017] In an embodiment, the updating the first expected return and the second expected return alternately based on the parameter updating in the parameterized policy and the parameterized model to output the to-be-corrected merge-in policy comprises: updating the policy parameters by using a gradient descent algorithm to maximize the first expected return as a first updating target to obtain an updated policy; determining an expected return distribution of a real driving action based on an experience data sample; inputting the updated policy into the parameterized model to optimize the neural network parameters to obtain a learned target policy evaluation model and a target policy improvement model, with minimizing a gap between the value function distribution and the expected return distribution of the real driving action as a second updating target.

[0018] In an embodiment, the configuring the cumulative expected return based on the instant return model and the policy entropy model further comprises: adjusting a temperature coefficient of the cumulative expected return based on minimizing a difference between an output of the policy entropy model and an expected policy entropy.

[0019] In an embodiment, the safety shield model comprises a state constraint module, and the collision avoidance correction operation on the distribution driving action based on the safety shield model in the application environment comprises: inputting the distribution driving action into a vehicle longitudinal and lateral dynamic model in the application environment to output an estimated driving state corresponding to any driving action; constructing the state constraint module of the safety state based on vehicle-vehicle collision avoidance constraints and vehicle-road collision avoidance constraints; and performing a collision avoidance correction operation on the driving action when it is detected that the estimated driving state does not satisfy a constraint condition in the state constraint module.

[0020] In an embodiment, the constructing the state constraint module of the safety state based on the vehicle-vehicle collision avoidance constraints and the vehicle-road collision avoidance constraints comprises: generating first contour information of a ego vehicle and second contour information of a surrounding vehicle based on a six-circle model; determining a first critical safety distance between the ego vehicle and the surrounding vehicle and a second critical safety distance between the ego vehicle and a road based on the first contour information and the second contour information; generating a vehicle-vehicle collision avoidance constraint condition based on the first critical safety distance; generating a vehicle-road collision avoidance constraint condition based on the second critical safety distance; and constructing the state constraint module based on the vehicle-vehicle collision avoidance constraint condition, the vehicle-road collision avoidance constraint condition, and the driving state.

[0021] In an embodiment, the safety shield model further comprises an action correction module, and the constructing the state constraint module based on the vehicle-vehicle collision avoidance constraints, the vehicle-road collision avoidance constraints, and the driving state further comprises: setting a conservative adjustment variable based on the action correction module to adjust conservativeness of the state constraint module based on the conservative adjustment variable.

[0022] According to another aspect of the present disclosure, there is provided a ramp merging strategy learning device, comprising: a configuration module configured to configure a cumulative expected return in a training environment, the cumulative expected return being configured to represent a cumulative value of expected returns of continuously performing a merging strategy to complete a ramp merging process; a construction module configured to construct a strategy evaluation model and a strategy improvement model based on the cumulative expected return, the strategy evaluation model being configured to evaluate an expected return of following the merging strategy from a specified driving state, and the strategy improvement model being configured to evaluate an expected return of adopting a specified driving action, the specified driving action being a driving action selected based on the merging strategy in the specified driving state; a learning module configured to alternately and iteratively learn the strategy evaluation model and the strategy improvement model with a strategy optimization target of optimizing the cumulative expected return, and obtain a to-be-corrected merging strategy based on a result of the alternately and iteratively learning, the to-be-corrected merging strategy being configured to output a distributed driving action in ramp merging; and a correction module configured to perform a collision avoidance correction operation on the distributed driving action based on a safety shield model in an application environment, and generate a corrected target merging strategy, so that a vehicle performs a ramp merging operation based on the target merging strategy.

[0023] According to still another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the ramp merging strategy learning method of the above-mentioned another aspect via execution of the executable instructions.

[0024] According to yet another aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the ramp merging strategy learning method of any of the above-mentioned aspects.

[0025] The ramp merging strategy learning scheme provided by the embodiments of the present disclosure can take into account trial-and-error learning and state safety in a strategy learning process by configuring a cumulative expected return in a training environment, can comprehensively consider strategy performance, strategy uncertainty and state safety in the strategy learning process, and can improve the accuracy of strategy evaluation by using a value distribution principle, and further can improve the safety of driving when the merging strategy is executed in the scene of merging from a ramp to a highway by driving according to a distributed driving action, performing a collision avoidance correction operation in a timely manner based on a safety shield model when it is detected that the driving action is not safe, and the like.

[0026] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings described below are only some embodiments of the present disclosure, and other drawings can be obtained by those of ordinary skill in the art without creative effort on the basis of these drawings.

[0028] Figure 1 A schematic diagram showing a ramp merging strategy learning system structure in an embodiment of the present disclosure is shown.

[0029] Figure 2 A flowchart showing a ramp merging strategy learning method in an embodiment of the present disclosure is shown.

[0030] Figure 3 A flowchart showing another ramp merging strategy learning method in an embodiment of the present disclosure is shown.

[0031] Figure 4 A flowchart showing still another ramp merging strategy learning method in an embodiment of the present disclosure is shown.

[0032] Figure 5 A flowchart showing still another ramp merging strategy learning method in an embodiment of the present disclosure is shown.

[0033] Figure 6 A flowchart showing still another ramp merging strategy learning method in an embodiment of the present disclosure is shown.

[0034] Figure 7 A schematic block diagram showing still another ramp merging strategy learning scheme in an embodiment of the present disclosure is shown.

[0035] Figure 8 A neural network approximation schematic diagram of a safe autonomous merging strategy in an embodiment of the present disclosure is shown.

[0036] Figure 9 A neural network approximation schematic diagram of another safe autonomous merging strategy in an embodiment of the present disclosure is shown.

[0037] Figure 10 A six-circle-based vehicle representation diagram in an embodiment of the present disclosure is shown.

[0038] Figure 11 A schematic diagram showing a collision constraint between a host vehicle and a surrounding vehicle in an embodiment of the present disclosure is shown.

[0039] Figure 12 A direct state constraint schematic diagram in an embodiment of the present disclosure is shown.

[0040] Figure 13 A schematic diagram showing a gate function state constraint trend envelope in an embodiment of the present disclosure is shown.

[0041] Figure 14 FIG. 1 shows a schematic diagram of an offline training parallel structure of a ramp merging strategy according to an embodiment of the present disclosure;

[0042] Figure 15 FIG. 2 shows a schematic diagram of a ramp merging strategy learning device according to an embodiment of the present disclosure;

[0043] Figure 16 FIG. 3 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those skilled in the art. Features described in the description, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0045] In addition, the accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure. The same reference numbers in different drawings represent the same or similar elements, and thus their repetitive description will be omitted. Some of the block diagrams in the drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0046] The scheme provided in the present application can consider trial and error learning and state safety in the policy learning process by accumulating expected return in the training environment configuration period, can comprehensively consider policy performance, policy uncertainty and state safety in the policy learning process, and improve the accuracy of policy evaluation by using the value distribution principle, and further drive according to the distribution driving action, through the interaction between the driving environment, based on the safety shield model, when the driving action is detected to be unsafe, collision avoidance correction operation is performed in time, and thus the safety of driving when the merging strategy is executed can be improved when applied to the scene of merging from the ramp to the highway.

[0047] Figure 1 FIG. 1 shows a schematic diagram of an offline training parallel structure of a ramp merging strategy according to an embodiment of the present disclosure;

[0048] The terminal 120 can be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the like, or the terminal 120 can also be a PC (Personal Computer) such as a laptop computer and a desktop computer, and the like.

[0049] The terminal 120 can be installed with an application program for providing the ramp merging strategy learning.

[0050] The terminal 120 and the server cluster 140 are connected through a communication network. The communication network can be a wired network or a wireless network.

[0051] The server cluster 140 can be a single server, or a plurality of servers, or a virtualization platform, or a cloud computing service center. The server cluster 140 is configured to provide a background service for the ramp merging strategy learning application program. Optionally, the server cluster 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or the server cluster 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or the terminal 120 and the server cluster 140 adopt a distributed computing architecture to perform collaborative computing.

[0052] In some optional embodiments, the server cluster 140 is configured to store the ramp merging strategy learning model, and the like.

[0053] Optionally, the application program clients installed in different terminals 120 are the same, or the application program clients installed on the two terminals 120 are the same type of application program clients of different control system platforms. Based on the difference in the terminal platform, the specific form of the application program client can also be different, for example, the application program client can be a mobile phone client, a PC client, or a Web (World Wide Web) client, and the like.

[0054] It can be appreciated by those skilled in the art that the number of the terminals 120 can be more or less. For example, the terminals can be only one, or the terminals can be dozens or hundreds, or more. The number of the terminals and the type of the devices are not limited in the embodiments of the present application.

[0055] Optionally, the system can further include a management device (not shown) configured to manage the terminals 120 and the server cluster 140. Figure 1The management device is connected to the server cluster 140 through a communication network (not shown). Optionally, the communication network is a wired network or a wireless network.

[0056] Optionally, the wireless network or wired network described above uses standard communication techniques and / or protocols. The network is typically the Internet, but can also be any network, including, but not limited to, a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile, wired or wireless network, a private network or any combination of virtual private networks. In some embodiments, techniques and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, all or some links can be encrypted using conventional encryption techniques such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In other embodiments, custom and / or proprietary data communication techniques can be used instead of or in addition to the data communication techniques described above.

[0057] In the following, each step in the ramp merging policy learning method in the example embodiment will be described in more detail in conjunction with the accompanying drawings and embodiments.

[0058] Figure 2 A flowchart of a ramp merging policy learning method in an embodiment of the present disclosure is shown. The method provided in the embodiment of the present disclosure can be executed by any electronic device with computing processing capability, for example, the terminal 120 and / or the server cluster 140 in Figure 1 The following example is illustrated with the terminal 120 as the execution subject.

[0059] As shown in Figure 2 The server cluster 140 executes the ramp merging policy learning method, including the following steps:

[0060] Step S202, configure cumulative expected return in a training environment, the cumulative expected return is used to represent the cumulative value of the expected return of continuously executing the merging policy to complete the ramp merging process.

[0061] The cumulative expected return can be understood as a learning goal of strategy learning, and the output strategy is made to reach the cumulative expected return, so that the merging strategy can realize the merging of the ramp.

[0062] In one embodiment, configuring the cumulative expected return in the training environment includes: obtaining an immediate return model for safety state evaluation and a strategy entropy model of the merging strategy, the immediate return model is used to evaluate the safety of executing the merging strategy into the corresponding driving state, and the strategy entropy model is used to measure the uncertainty of the merging strategy.

[0063] The immediate return model is used for strategy learning with safety state as the target, and the strategy entropy model is used for continuous trial and error based on distributed driving actions to complete the strategy learning.

[0064] The cumulative expected return is configured in the training environment based on the immediate return model and the strategy entropy model.

[0065] By configuring the cumulative expected return in the training environment based on the immediate return model and the strategy entropy model, the obtained cumulative expected return can take into account the evaluation of the safety state after the strategy execution and the exploration of the uncertainty of the merging strategy at the same time, thereby ensuring the effectiveness of the output merging strategy.

[0066] Specifically, the agent starts from the initial state and ends at the terminal state. The distributed review algorithm finds the optimal strategy by maximizing the cumulative return expectation through periodic interaction with the environment. Currently, the cumulative return is single, and only the performance of task completion is considered. In order to improve the strategy exploration, the distributed review defines the cumulative return expectation of the distributed review algorithm based on the maximum entropy principle, so that the strategy evaluation considers the cumulative strategy uncertainty. The strategy evaluation return function r mod Combined with the cumulative return of the distributed review, that is, the entropy H of the strategy, the cumulative return of the entropy and safety enhancement is obtained Also known as cumulative soft return, as shown in equation (1):

[0067]

[0068] Where t represents the driving time, i represents the ordinal number of the driving step, that is, each driving action performed corresponds to a driving step, H is a strategy entropy model, indicating the uncertainty of an action a under the strategy π, r mod is the immediate return considering state safety, γ is a discount factor, which is a coefficient, 0≤γ<1, if it is the immediate return after i-t steps, it should be multiplied by γ i-tThe factor of the actual effective return is obtained, the strategy entropy model is inversely proportional to the probability of selecting an action, which means that the smaller the probability of the action being currently selected, the greater the uncertainty, the greater the cumulative soft return, the higher the strategy evaluation score, and the greater the possibility of exploring the action in the future. The representation of H is shown in formula (2).

[0069] H = - a log p (a | s) (2)

[0070] Wherein, a is a temperature coefficient, a represents a driving action, and s represents a corresponding driving state obtained.

[0071] Since the off-ramp merging decision of the intelligent vehicle has the characteristics of repetitive interaction, the strategy learning process can be defined as a periodic task, for example, taking a frame as a cycle unit, outputting the merging strategy and the expected value of the merging strategy through iteration, realizing strategy learning, and reaching the target cumulative expected return.

[0072] Step S204, constructing a strategy evaluation model and a strategy improvement model based on the cumulative expected return, the strategy evaluation model is used to evaluate the first expected return obtained by following the merging strategy from the specified driving state, and the strategy improvement model is used to evaluate the second expected return obtained by using the specified driving action. The specified driving action refers to the driving action selected based on the merging strategy in the specified driving state.

[0073] Wherein, the merging strategy refers to the mapping from state to behavior.

[0074] Specifically, the strategy evaluation model is a soft state value function for evaluating the strategy, which is defined as the expected cumulative soft return starting from the state s t , and is recorded as the first expected return:

[0075]

[0076] Wherein, p refers to the merging strategy, and the expected return.

[0077] The strategy improvement model is a soft action value function, which is represented as q soft (s, a), which represents the expected return that can be obtained by selecting the driving action a t according to the merging strategy p t , and is recorded as the second expected return, and the specific representation is:

[0078]

[0079] Wherein, r0 refers to the initial state return.

[0080] In step S206, the strategy evaluation model and the strategy improvement model are alternately iteratively learned with the optimization of the cumulative expected return as the strategy optimization target. Based on the result of the alternately iterative learning, a to-be-corrected merging strategy is obtained, and the to-be-corrected merging strategy is used to output the distributed driving action when the ramp merges.

[0081] The strategy iteration is repeatedly performed between the strategy evaluation and the strategy improvement.

[0082] The to-be-corrected merging strategy refers to the optimal merging strategy output by the strategy learning.

[0083] In step S208, the distributed driving action is collision avoidance corrected based on the safety shield model in the application environment, and a corrected target merging strategy is generated, so that the vehicle performs the ramp merging operation based on the target merging strategy.

[0084] In this embodiment, by configuring the cumulative expected return in the training environment, the trial-and-error learning and the state safety can be considered in the strategy learning process. The strategy performance, the strategy uncertainty and the state safety can be considered comprehensively in the strategy learning process. The accuracy of the strategy evaluation is improved by using the value distribution principle. Further, according to the distributed driving action, the interaction between the driving action and the driving environment is performed. When it is detected that the driving action is not safe based on the safety shield model, the collision avoidance correction operation is performed in time, and thus the safety of the driving when the merging strategy is executed can be improved when the merging strategy is applied to the scene of merging from the ramp to the highway.

[0085] In one embodiment, the alternately iteratively learning of the strategy evaluation model and the strategy improvement model with the optimization of the cumulative expected return as the strategy optimization target comprises:

[0086] The first association relationship between the strategy improvement model and the strategy evaluation model is obtained.

[0087] The relationship between the strategy improvement model and the strategy evaluation model is denoted as the first association relationship, and is specifically represented as:

[0088] v soft (s)=∑ a∈A π(a|s)(q soft (s,a)-αlogπ(a|s)) (5)

[0089] A is a set of driving actions.

[0090] Based on the first association relationship, the strategy evaluation model and the strategy improvement model are alternately iteratively learned with the optimization of the cumulative expected return as the strategy optimization target.

[0091] As Figure 3As shown, as a specific embodiment, in one embodiment, based on the first correlation relationship, and taking the maximum cumulative expected return as the strategy optimization target, the strategy evaluation model and the strategy improvement model are alternately iterated and learned, including:

[0092] In step S302, the corresponding dynamic programming equation is configured based on the first correlation relationship, and the dynamic programming equation is used to express the relationship between the expected returns between two adjacent driving states.

[0093] Among them, as a specific implementation, the Bellman equation is used as the dynamic programming equation, that is, the Bellman equation is used to repeatedly update the strategy evaluation model.

[0094] In step S304, based on the dynamic programming equation, the strategy evaluation result output by the strategy evaluation model and the improved strategy output by the strategy improvement model are alternately calculated.

[0095] Among them, the improved strategy is suitable for maximizing the corresponding strategy evaluation result of the initial distribution of the distributed driving action.

[0096] Specifically, when a given strategy π is given, the dynamic programming equation is repeatedly applied to update the process of the strategy improvement model, as shown in equation (6):

[0097]

[0098] Among them, s' is the next driving state corresponding to the current driving state, a' indicates the next driving action, T indicates the update operation, and D indicates the repeated operation.

[0099] In the process of strategy improvement, a better strategy π than the current strategy π is ensured to be found new The new strategy can be directly obtained by maximizing the average soft strategy evaluation model under the initial distribution, as shown in equation (7):

[0100]

[0101] As shown in equation (7), by introducing the cumulative strategy entropy model, the randomness of the merging strategy corresponding to each driving state is different, and the merging strategy has more reasonable randomness.

[0102] Further, the optimal strategy π * under this principle satisfies equation (8):

[0103]

[0104] Among them, the optimal term is represented by "*", the optimal action value is , the optimal state value is , and α is the temperature coefficient.

[0105] In this embodiment, the cumulative expected return is obtained by introducing policy entropy and safety evaluation, the cumulative return of the strategy changes, and the gradient of the strategy update also changes, by setting an immediate return model for safety state evaluation, the action with higher safety state is evaluated higher, and the probability of being selected in the future is larger, by setting a policy entropy model for evaluating the uncertainty of the merged strategy, the exploration term based on the policy entropy model makes the action with larger current selection probability evaluated lower, to encourage the selection of unexplored actions in the future, and make the selectability of the merged strategy higher.

[0106] In one embodiment, a temperature coefficient a is configured in the cumulative expected return, and the temperature coefficient a is suitable for adaptively adjusting the speed of convergence of the random strategy to the deterministic strategy.

[0107] In this embodiment, the performance and exploration in the return function are controlled by the temperature coefficient a, when a is 0, the strategy becomes a deterministic strategy, so the speed of convergence of the random strategy to the deterministic strategy can be controlled by adaptive adjustment of a.

[0108] Based on the dynamic programming equation, the policy evaluation result output by the policy evaluation model and the improved strategy output by the policy improvement model are alternately calculated, and the method further comprises:

[0109] In step S306, the improved strategy converges from the random strategy to the deterministic strategy, and the deterministic strategy is determined as the to-be-corrected merged strategy.

[0110] As Figure 4 shown, in one embodiment, before alternately calculating the policy evaluation result output by the policy evaluation model and the improved strategy output by the policy improvement model based on the dynamic programming equation, the method further comprises:

[0111] In step S402, the merged strategy is used to interact with the deployed driving environment from a specified driving state, to generate a sequence comprising the driving state, the driving action and the immediate feedback, so as to determine the discounted cumulative return of the immediate feedback based on the sequence.

[0112] Among them, accurate policy evaluation is the guarantee for the policy improvement process to find the optimal strategy, in order to better estimate the policy, the distributed evaluation algorithm introduces the value function distribution theory, specifically, the autonomous vehicle starts from state s, uses the merged strategy π to interact with the environment, and obtains the state-action-immediate return sequence, the discounted cumulative return of the immediate return of these sequences is represented by Z π , which can be understood as a special action value function, that is, a special policy improvement model, as shown in equation (9):

[0113]

[0114] Among them, r represents the return obtained after taking a certain action a in a certain state s, and pπ a policy-based transition state.

[0115] In step S404, a second association relationship between the discounted cumulative return and the policy improvement model is determined.

[0116] where the state transition probability The return function R and the randomness of the policy π, Z π (s, a) is a random variable, compared with the definition formula of the general policy improvement model, the second association relationship is determined as:

[0117]

[0118] In step S406, the distribution of the discounted cumulative return is modeled based on the second association relationship, and a policy improvement distribution model is obtained, which is suitable for optimizing the policy evaluation result.

[0119] Specifically, in order to more accurately evaluate the policy, the distribution of Z π (s, a) is modeled, and the modeling result is shown in formula (11):

[0120]

[0121] Formula (11) can be understood as the probability of mapping from the action state pair (s, a) to the corresponding value, which is called the policy improvement model distribution, and P refers to the transition operator.

[0122] In step S408, a dynamic programming equation is configured based on the policy improvement distribution model, so as to alternately calculate the policy evaluation result and the improved policy based on the dynamic programming equation.

[0123] Further, the dynamic programming equation under the value distribution can be rewritten as formula (12):

[0124]

[0125] where is the Bellman operator of the value distribution, r ~ R(.|s, a), s' ~ p, a' ~ π, denotes that two random variables A and B have the same probability distribution, assuming where denotes the distribution of Z

[0126]

[0127] where d is a measure function for measuring the distance between two distributions, such as the KL divergence D KL , etc.

[0128] In one embodiment, the alternately calculating the strategy evaluation result and the improved strategy comprises: parameterizing the improved strategy to obtain a parameterized strategy; parameterizing the strategy improvement model and the strategy evaluation model to obtain parameterized models; alternately updating the first expected return and the second expected return based on the parameter updating in the parameterized strategy and the parameterized models to output the to-be-corrected merged strategy.

[0129] In this embodiment, for a decision problem with a large-scale state, a large amount of state and action information needs to be stored, and the value function of each state needs to be updated separately, and the updating process is very slow. However, for a problem with a large-scale discrete state and action 1, or a decision problem like autonomous driving with a continuous state space and action space, the tabularization method will bring a dimension explosion, requiring a large amount of memory and being very time-consuming to calculate. Therefore, the value function is parameterized with the strategy by using an approximate function, and the value function of all states is updated by updating the parameters of the approximate function.

[0130] In one embodiment, the parameterizing the improved strategy to obtain a parameterized strategy comprises: parameterizing the merged strategy by using mean and variance to obtain a parameterized strategy including strategy parameters.

[0131] In this embodiment, in order to improve the exploratory of the strategy, a randomized merged strategy is selected, and the value distribution function of a state s in a certain state under the randomized strategy is regarded as a Gaussian distribution. As shown in Figure 8 , a neural network is used to approximate the distribution function to output the mean value μ and the variance σ 2 of the action distribution. The parameterized strategy is represented by formula (14):

[0132]

[0133] wherein θ is a strategy parameter, and the strategy parameter includes two, which are parameters of the mean function and the variance function.

[0134] In one embodiment, the parameterizing the strategy improvement model and the strategy evaluation model to obtain parameterized models comprises: representing the value function distribution of the strategy improvement model based on a strategy improvement distribution model; and approximating the strategy improvement distribution model based on a neural network to obtain a parameterized model including neural network parameters.

[0135] As shown in Figure 9 , an approximate schematic diagram of the value function distribution de obtained by approximating the strategy (s, a) improvement distribution model based on a neural network. 2 .

[0136] In this embodiment, the parameterized value function distribution is denoted as The parameters of the neural network are w, and in order to make the value network or the action state value network more and more accurately represent the corresponding true value, it is necessary to find suitable neural network parameters w. Therefore, the update of the parameters is to minimize the difference between the parameterized value function distribution and the true distribution as the optimization objective:

[0137]

[0138] where d(s) is the distribution of the value state, and is usually the experience data sample collected in training. The update of the neural network parameters w is performed using the gradient descent method until the optimal solution is found, as shown in equation (16):

[0139]

[0140] where β is the gradient parameter, J() is the gradient descent function, represents the gradient.

[0141] As Figure 5 shown, in one embodiment, based on the update of the parameters in the parameterized policy and the parameterized model, the first expected return and the second expected return are alternately updated to output a specific implementation of the to-be-corrected merging policy, which includes:

[0142] Step S502, using the gradient descent algorithm to update the policy parameters with the first update target of maximizing the first expected return, to obtain an updated policy.

[0143] The update of the parameters is to maximize the value function as the objective function, and the gradient descent method is used for updating, as shown in equation (17):

[0144]

[0145] Step S504, determining the expected return distribution of the real driving action based on the experience data sample.

[0146] Step S506, inputting the updated policy into the parameterized model, taking the second update target of minimizing the difference between the value function distribution and the expected return distribution of the real driving action, and optimizing the updated neural network parameters to obtain the learned target policy evaluation model and the target policy improvement model.

[0147] Specifically, taking the difference between the value function distribution and the expected return distribution of the real driving action as the target, the parameters of the value function distribution model are updated based on the gradient descent method until the merging policy that meets the ramp merging is generated.

[0148] By continuously updating the parameters of the policy network and the policy evaluation network, the true value function and policy are continuously approached. In order to ensure the stability of the learning process, the corresponding target policy evaluation model and target policy improvement model are also introduced, and the parameters are The update rule is shown in equation (18).

[0149]

[0150] where ∈ and τ are rule parameters, and the value distribution network parameters are updated to utilize the value function distribution updated by one step of dynamic programming equation as the target distribution. In order to make the value distribution network more and more accurately represent the true distribution of the value function, the parameter update of the neural network takes the KL divergence between the approximate value distribution and the target distribution as the objective function, as shown in equation (19):

[0151]

[0152] where, The process of parameterizing a probability distribution.

[0153] is the previous sampling experience data buffer, and c is a constant.

[0154] Further, the gradient of the objective function of the evaluation network with respect to the neural network parameter ω is:

[0155]

[0156] It can be seen that when is a continuous Gaussian distribution, the gradient is easy to explode because as

[0157] In order to solve this problem, it is necessary to make close to the expected Q of the current value return w (s, a). Thus, the gradient expression of the distributed evaluation network can be rewritten as:

[0158]

[0159] where,

[0160] In the formula, clip[x, A, B] represents that x is modified to the interval [A, B], and b is the modified boundary value.

[0161] In addition, the update of the parameters of the policy network takes the maximum average soft policy evaluation model as the objective function:

[0162]

[0163] To maximize the objective function, in order to find the gradient of the objective function, especially the gradient of Z(s, a) with respect to θ, the random action a and the random return Z(s, a) are reparameterized, wherein the action a will be generated according to the policy πθ The randomly selected action a can be represented by a function f that is differentiable with respect to the policy network parameters θ:

[0164] a = f θ (ξ, s) = μ + ξσ 2 (23)

[0165] where ξ is a noise sampled from a Gaussian distribution, μ and σ 2 are the mean and variance of the policy network output, similarly, the random return Z(s, a) can be represented by a function g that is differentiable with respect to the policy network parameters θ:

[0166] Z(s, a) = g θ (ξ Z , s, a) (24)

[0167] where ξ Z is a noise sampled from a Gaussian distribution, after reparameterization of the action and the random return, the gradient of the objective function can be:

[0168]

[0169] In an embodiment, the cumulative expected return is adjusted based on the temperature coefficient of the policy entropy model and the immediate return model in the training environment configuration, and the temperature coefficient of the cumulative expected return is further adjusted based on minimizing the difference between the output of the policy entropy model and the expected policy entropy.

[0170] In addition, the temperature coefficient α of the policy entropy model and the cumulative return is updated by minimizing the difference between the expected policy entropy and the difference between the output of the policy entropy model and the expected policy entropy.

[0171]

[0172] Generally, reinforcement learning introduces a neural network to approximate the policy evaluation model υ(s) and the policy improvement model q(s, a) as the network for policy evaluation.

[0173] The distributed evaluation algorithm selected in the present disclosure is based on the value distribution principle, that is, the policy is evaluated by using the value function distribution. Unlike the traditional value function approximation, the distributed evaluation algorithm uses the mean and variance of the output distribution of the neural network. Assuming that the distribution of the value function satisfies the Gaussian distribution, the distribution of the value function is approximated by using the neural network, such as Figure 7As shown, the ramp merging strategy learning scheme in the present disclosure includes an offline training module and an online application module. The offline training module includes a policy evaluation network and a policy network, i.e., a policy improvement model. The policy evaluation network evaluates the policy based on performance indicators and safety indicators, and then further improves the policy based on the evaluation results, so as to realize iterative optimization of the merging strategy, and obtain the policy 0 to the policy N. The merging strategy outputs a distribution driving action through the policy network. After the sampled distribution driving action is input into a simulation environment, experience data is output. The experience data is input into the policy evaluation network, so as to realize alternative policy iteration between policy evaluation and policy improvement.

[0174] As shown in the figure, Figure 6 In one embodiment, the safety shield model includes a state constraint module. In step S208, one specific implementation manner of the collision avoidance correction operation of the distribution driving action based on the safety shield model in the application environment includes:

[0175] In step S602, the distribution driving action is input into a vehicle longitudinal and lateral dynamic model in the online application environment, so as to output an estimated driving state corresponding to any driving action.

[0176] In step S604, a state constraint module of a safe state is constructed based on the vehicle-vehicle collision avoidance constraint and the vehicle-road collision avoidance constraint.

[0177] In step S606, when it is detected that the estimated driving state does not satisfy the constraint condition in the state constraint module, the driving action is subjected to a collision avoidance correction operation.

[0178] In one embodiment, constructing the state constraint module of the safe state based on the vehicle-vehicle collision avoidance constraint and the vehicle-road collision avoidance constraint includes: Figure 10 As shown, the first contour information of the ego vehicle and the second contour information of the surrounding vehicle are generated based on the six-circle model. Assuming that the vehicle body length is L and the vehicle body width is W, the radius of a corresponding circle is W / 4 or L / 6.

[0179] The first critical safety distance between the vehicles and the second critical safety distance between the vehicle and the road are determined based on the first contour information and the second contour information. The vehicle-vehicle collision avoidance constraint condition is generated based on the first critical safety distance. The vehicle-road collision avoidance constraint condition is generated based on the second critical safety distance. The state constraint module is constructed based on the vehicle-vehicle collision avoidance constraint condition, the vehicle-road collision avoidance constraint condition, and the driving state.

[0180] In one embodiment, the safety shield model further includes an action correction module. The state constraint module is constructed based on the vehicle-vehicle collision avoidance constraint, the vehicle-road collision avoidance constraint, and the driving state. Further, a conservative adjustment variable is set based on the action correction module, so as to adjust the conservativeness of the state constraint module based on the conservative adjustment variable.

[0181] Specifically, the safety shield in the online application stage regards the process of mapping to the safety decision space as solving a constrained optimization problem. In order to intervene as little as possible, only when the distributed review algorithm decision output is an unsafe action, find the closest safe action to it, the objective function of the optimization problem is the distance between the new action and the distributed review algorithm output action a * The safety shield is based on rules or local environment models, in order to ensure that the longitudinal and lateral actions output by the decision model can reasonably avoid collision, based on the kinematic model of the vehicle, combined with safety state constraints, the disclosure constructs an optimization problem to modify the unsafe action output by the decision.

[0182] From the above, the safety nature of the distributed review algorithm imposes safety constraints on the state, and the action output by the safety decision should make the state safe and meet the collision avoidance condition, both avoiding surrounding obstacles and avoiding collision with the road. Common collision constraints are based on relative distance, and also consider relative distance and relative speed, this disclosure directly judges the collision state by using real-time relative distance. The distance collision constraint depends on the description method of the vehicle contour. Common vehicle contour description methods can be divided into three categories: single circle, multiple circles and rectangle. According to the rectangular vehicle contour description method, it is difficult to uniformly represent the constraints in different directions due to the different sizes of the longitudinal and lateral safety distances. According to the single circle method of vehicle contour description, the lateral collision avoidance safety constraint is too conservative. Therefore, considering the anisotropy of the collision avoidance safety constraint, this disclosure uses a six-circle model to describe the contour information of the vehicle, as shown in Figure 10 The ego vehicle and surrounding vehicles are represented by six covering circles, the radius of the covering circle is one sixth of the vehicle length, the critical safety distance between vehicles is the sum of the radii of the two vehicles, and the critical safety distance between the vehicle and the road is the radius of the ego vehicle.

[0183] The vehicle-vehicle collision avoidance condition is that the distance between the ego vehicle and the surrounding vehicle needs to be greater than the critical safety distance, as shown in equation set (27):

[0184] d j,k ≥d safe

[0185] d safe =r e +r k

[0186] j,k={1,…,6} (27)

[0187] Where d j,k represents the distance between the center of the jth circle of the ego vehicle and the center of the kth circle of any surrounding vehicle, as shown by L1 in Figure 11

[0188] d safe is the critical safety distance between vehicles, and r k is the sum of the radii of the covering circle of the surrounding vehicle and the covering circle of the ego vehicle.​e To ensure a vehicle avoids collisions within its lane, the distance d from either side of the road should be maintained. r With d l Greater than the safe distance, as shown in equation (28):

[0189] d j,r ≥d safe

[0190] d j,l ≥d safe

[0191] d safe =r e (28)

[0192] Where, d j,r and d j,l It is the shortest distance from the center of the j-th circle of the car to the left and right curbs, d safe This represents the critical safe distance between the vehicle and the road. Based on the aforementioned vehicle-to-vehicle and vehicle-to-road collision avoidance constraints, if we consider a cycle number of n vehicles and know their future trajectories, we can obtain the vehicle state x from the current moment to the next future moment. e y e The direct constraints are as follows: at any given time, there are 36N vehicle-to-vehicle constraints and 12 vehicle-to-road constraints. Let h(X) be the vehicle-to-vehicle constraint between the center of the j-th circle of the vehicle at time t and the center of the k-th circle of the n-th surrounding vehicle. t ) v,j,n,k The vehicle-road constraint for the j-th circle center is h(X). t ) j,r and h(X) t ) j,l Its expression is shown in equation group (29):

[0193]

[0194] Where n∈N, x∈[x e ,y e The state x is called the constrained state, which may differ from the state s in the decision problem. e (t) and y e (t) represents the centroid position of the vehicle at time t, a and b are variables related to the length and width of the vehicle and the surrounding vehicles at times j and k, and c, d, e, and f are variables related to the position of the road edge at time j and the length and width of the vehicle.

[0195] Action correction in the online application phase is real-time and does not consider global optimality; it is a single-step optimization problem. The vehicle's lateral and longitudinal dynamics models can be used to predict the next state under a given action, as shown in equation (30):

[0196]

[0197] where a is the input of the model's state equation f(.), which needs to be consistent with the action selected by the distributed review. If the direct state constraint is taken as the constraint of the action modification optimization problem, the action optimization problem at time r is shown in equation set (31) as follows:

[0198]

[0199] However, under the direct state constraint, the current selected safe action may be more aggressive, making the next time safe state no action solution, such as Figure 12 .

[0200] The present disclosure proposes a state constraint that satisfies the exponential gate function condition, which constrains the trend of h(x), improves the conservatism of the safe action, and reduces the situation of infeasible solution in the future. For a discrete-time system, the safe state set at time t can be constructed by the direct state constraint.

[0201] S safe (t)={x t |h(x t )≤0} (32)

[0202] For a discrete-time system with control input, the safe state set is an invariant set, which means that given an initial safe state, there exists a sequence of control quantities a such that the future state is still in the safe state set, denoted by equation (33):

[0203]

[0204] In order to make the current safe action more conservative, the present disclosure uses the exponential gate function condition to convert the direct state constraint h(·) in the action optimization problem into a more conservative constraint, so that the change of h(·) satisfies the exponential gate function condition, as shown in equation set (34):

[0205]

[0206] h(x t+1 )≤(1-λ)h(x t ) (34)

[0207] Where λ is a variable that adjusts the conservatism. From the above, it can be seen that the direct state constraint satisfies equation (35) in the entire time domain:

[0208] h(x t+i )≤(1-λ) i h(x t )≤0,i∈{1,...∞} (35)

[0209] The state constraint at any time has an upper limit (1-λ) i h(xt ), called self-closing property under the exponential barrier function condition, where the upper bound (1-λ) i h(x t ) has a trend related to the value of λ, as shown in Figure 13

[0210] It can be seen that when λ = 1, the formula h(x t+i )≤(1-λ) i h(x t )≤0,i∈{1,…∞} is a one-step direct state constraint, which is the least conservative case; when 0 < λ < 1, the upper limit of the direct state constraint is between (h(x t ), 0); when λ = 0, the upper limit is h(x t ), and when λ < 0, the upper limit is (1-λ) i h(x t ), which is too conservative. Therefore, the value range of λ in the exponential barrier function condition of the direct state is (0, 1], and a smaller value can be preferred, and when there is no feasible solution under this constraint, it can be appropriately relaxed. Combined with the above state constraint barrier function condition, the optimization problem expression of the action correction at time t is shown in formula (36):

[0211]

[0212] The optimization problem in the above action correction is a quadratic problem with constraints, and common solvers can support solving. Specifically, the optimize module of SciPy is used to solve the nonlinear optimization problem with constraints.

[0213] In order to improve the offline training efficiency of the learning autonomous decision, the present disclosure adopts an asynchronous parallel framework, as shown in Figure 14 . The offline training process is divided into policy learning and action execution, which are performed asynchronously in parallel by multiple agents. The sample data generated by each agent performing an action at each time step is randomly transmitted to a certain experience data buffer, and the experience data buffer continuously stores data. The data sampled at each step is randomly transmitted to each intelligent agent for policy learning. Each agent updates the parameters of the shared policy evaluation and policy improvement modules through learning, and then synchronizes the parameters to the agents for policy learning and action execution.

[0214] ​In addition, the distributed critic algorithm employs a parallel framework to improve the efficiency of offline training. The framework includes 6 agents for learning, 6 agents for performing actions, and 4 agents for experience buffer replay. For both the random policy and the value distribution policy algorithms, the present disclosure utilizes a Gaussian distribution approximation with the neural network outputting a mean and a variance, where the variance is a diagonal matrix. Specifically, a multilayer perceptron is employed to map the outputted state quantity to the mean and variance of the Gaussian distribution, the multilayer perceptron being a type of neural network composed of an input layer, a hidden layer, and an output layer, adjusting the interconnection relationship of each node inside, training the network, adjusting the weight, and constructing a model of the input and output being associated with each other. For the action evaluation network, a fully connected layer with 5 hidden layers is employed, each layer having 256 neurons, and the activation function being a Gaussian error linear unit. The optimizer for parameter updating is an Adam method with a cosine annealing learning rate. Different algorithms employ the same neural network structure and hyperparameters.

[0215] In order to make the random policy comply with the physical characteristics of the vehicle lateral and longitudinal actions, it is necessary to limit the selected action within a certain range[a min ,a max ]. Let the unconstrained Gaussian distribution be ζ. Wherein represents an action randomly selected from the distribution ζ, and is projected into the expected action a within the limit range by using equation (37).

[0216]

[0217] Wherein represents the Hadamard inner product. The probability density of a is defined as equation (38):

[0218]

[0219] The log-likelihood function of π(a|s) is represented as equation (39):

[0220]

[0221] Specifically, intelligent vehicles capable of effectively improving driving performance have been booming in recent years, and autonomous decision-making is one of the core technologies to achieve intelligence. Ramps are an important part of roads, and the dynamic, random and game nature of the merging scene makes ramps a high-accident road section. Autonomous merging decision-making of intelligent vehicles is a key technology to achieve automatic driving on structured roads, and the success rate and safety of merging in a dynamic environment are difficult points of autonomous decision-making. However, the existing rule-based decision-making method has difficulty in setting the critical value of the behavior selection condition and is difficult to cope with complex scenes. The decision-making method based on supervised learning takes the approximation of human driver data as the index, and the decision-making model is difficult to surpass the level of human beings. The decision-making based on reinforcement learning has the potential to surpass the level of human drivers and is an effective method to solve complex decision-making problems. However, there is a contradiction between exploration learning and application safety of the reinforcement learning type decision-making. To improve the safety of the reinforcement learning type decision-making algorithm in complex dynamic driving scenes, the present disclosure takes the decision-making method of intelligent vehicles as the research object, proposes a learning type safe autonomous decision-making algorithm for offline training and online application, constructs a decision-making model for ramp merging, realizes continuous lateral and longitudinal action decision-making, and improves driving safety. The present invention designs a safety decision-making method based on distributed comments. In the online application stage, based on the vehicle-vehicle and vehicle-road collision constraints, the action modification optimization problem of the safety shield is designed with the constraint of the gate function condition. In the offline learning stage, the performance of the strategy, the uncertainty of the strategy and the state safety are considered comprehensively in the strategy evaluation stage of the distributed comments, and the accuracy of the strategy evaluation is improved by using the value distribution principle.

[0222] It should be noted that the above-described diagrams are only schematic representations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to limit the purpose. It is easy to understand that the processes shown in the above-described diagrams do not indicate or limit the time sequence of the processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0223] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0224] The ramp merging strategy learning device 1500 according to this embodiment of the present application will be described below with reference to Figure 15 The ramp merging strategy learning device 1500 shown is only an example and should not limit the function and use range of the embodiments of the present application. Figure 15 The ramp merging strategy learning device 1500 shown is only an example and should not limit the function and use range of the embodiments of the present application.

[0225] The ramp merging strategy learning device 1500 is in the form of a hardware module. The components of the ramp merging strategy learning device 1500 can include, but are not limited to, a configuration module 1502 configured to configure a cumulative expected return in a training environment, the cumulative expected return being configured to represent a cumulative value of expected returns of continuously executing a merging strategy to complete a ramp merging process; a construction module 1504 configured to construct a strategy evaluation model and a strategy improvement model based on the cumulative expected return, the strategy evaluation model being configured to evaluate an expected return of following the merging strategy from a specified driving state, and the strategy improvement model being configured to evaluate an expected return of adopting a specified driving action, the specified driving action being a driving action selected based on the merging strategy in the specified driving state; a learning module 1508 configured to perform alternating iterative learning on the strategy evaluation model and the strategy improvement model with optimization of the cumulative expected return as a strategy optimization target, and based on a result of the alternating iterative learning, obtain a to-be-corrected merging strategy, the to-be-corrected merging strategy being configured to output a distributed driving action in ramp merging; and a correction module 1506 configured to perform collision avoidance correction operation on the distributed driving action based on a safety shield model in an application environment, and generate a corrected target merging strategy, so that the vehicle performs ramp merging operation based on the target merging strategy.

[0226] In one embodiment, the configuration module 1502 is further configured to: obtain an immediate return model for safety state evaluation and a strategy entropy model of the merging strategy, the immediate return model being configured to evaluate safety of entering a corresponding driving state by executing the merging strategy, and the strategy entropy model being configured to measure uncertainty of the merging strategy; and configure the cumulative expected return in the training environment based on the immediate return model and the strategy entropy model.

[0227] In one embodiment, the learning module 1506 is further configured to: obtain a first association relationship between the strategy improvement model and the strategy evaluation model; and perform alternating iterative learning on the strategy evaluation model and the strategy improvement model based on the first association relationship and with maximization of the cumulative expected return as a strategy optimization target.

[0228] In one embodiment, the learning module 1506 is further configured to: configure a corresponding dynamic programming equation based on the first association relationship, the dynamic programming equation being configured to express a relationship between expected returns between two adjacent driving states; and alternatingly calculate a strategy evaluation result output by the strategy evaluation model and an improved strategy output by the strategy improvement model based on the dynamic programming equation, wherein the improved strategy is adapted to maximize the corresponding strategy evaluation result of the distributed driving action under the initial distribution.

[0229] In one embodiment, a temperature coefficient is configured in the cumulative expected return, the temperature coefficient being adapted to adaptively adjust a speed of convergence of a random strategy to a deterministic strategy, and the learning module 1506 is further configured to: the improved strategy is converged from the random strategy to the deterministic strategy, and the deterministic strategy is determined as the to-be-corrected merging strategy.

[0230] In an embodiment, the learning module 1506 is further configured to: interact with the deployed driving environment using the roll-in strategy from the specified driving state, generate a sequence comprising the driving state, the driving action, and the immediate reward, determine a discounted cumulative return of the immediate reward based on the sequence, determine a second association between the discounted cumulative return and the policy improvement model, model a distribution of the discounted cumulative return based on the second association to obtain a policy improvement distribution model, the policy improvement distribution model being adapted to optimize the policy evaluation result, configure a dynamic programming equation based on the policy improvement distribution model, and alternately calculate the policy evaluation result and the improved policy based on the dynamic programming equation.

[0231] In an embodiment, the learning module 1506 is further configured to: parameterize the improved policy to obtain a parameterized policy, parameterize the policy improvement model and the policy evaluation model to obtain a parameterized model, and alternately update the first expected return and the second expected return based on the update of the parameters in the parameterized policy and the parameterized model to output the to-be-corrected roll-in strategy.

[0232] In an embodiment, the learning module 1506 is further configured to: parameterize the roll-in strategy using the mean and the variance to obtain a parameterized policy comprising policy parameters.

[0233] In an embodiment, the learning module 1506 is further configured to: represent a value function distribution of the policy improvement model based on the policy improvement distribution model, and approximate the policy improvement distribution model based on a neural network to obtain a parameterized model comprising neural network parameters.

[0234] In an embodiment, the learning module 1506 is further configured to: update the policy parameters using a gradient descent algorithm to obtain an updated policy, determine an expected return distribution of the real driving action based on the experience data sample, and input the updated policy into the parameterized model to optimize the neural network parameters to obtain the learned target policy evaluation model and the target policy improvement model, with the first update target being to maximize the first expected return and the second update target being to minimize the gap between the value function distribution and the expected return distribution of the real driving action.

[0235] In an embodiment, the learning module 1506 is further configured to: adjust a temperature coefficient of the cumulative expected return based on minimizing the difference between the output of the policy entropy model and the expected policy entropy.

[0236] In an embodiment, the safety shield model comprises a state constraint module, and the correction module 1508 is further configured to: input the distribution driving action into a vehicle longitudinal and lateral dynamics model to output an estimated driving state corresponding to any driving action in the online application environment; construct a state constraint module of a safe state based on the vehicle-vehicle collision avoidance constraint and the vehicle-road collision avoidance constraint; and perform a collision avoidance correction operation on the driving action when it is detected that the estimated driving state does not satisfy the constraint condition in the state constraint module.

[0237] In an embodiment, the correction module 1508 is further configured to: generate first contour information of the ego vehicle and second contour information of the surrounding vehicle based on the six-circle model; determine a first critical safety distance between the ego vehicle and the surrounding vehicle and a second critical safety distance between the ego vehicle and the road based on the first contour information and the second contour information; generate a vehicle-vehicle collision avoidance constraint condition based on the first critical safety distance; generate a vehicle-road collision avoidance constraint condition based on the second critical safety distance; and construct the state constraint module based on the vehicle-vehicle collision avoidance constraint condition, the vehicle-road collision avoidance constraint condition, and the driving state.

[0238] In an embodiment, the safety shield model further comprises an action correction module, and the correction module 1508 is further configured to: set a conservative adjustment variable based on the action correction module, and adjust the conservativeness of the state constraint module based on the conservative adjustment variable.

[0239] The electronic device 1600 according to this embodiment of the present application will be described below with reference to Figure 16 Figure 16 The electronic device 1600 shown is merely an example and should not impose any limitation on the function and scope of use of the embodiments of the present application.

[0240] As Figure 16 shown, the electronic device 1600 is in the form of a general computing device. The components of the electronic device 1600 can include, but are not limited to, the at least one processing unit 1610 described above, the at least one storage unit 1620 described above, and a bus 1630 connecting different system components, including the storage unit 1620 and the processing unit 1610.

[0241] The storage unit stores program code that can be executed by the processing unit 1610, so that the processing unit 1610 performs the steps described in the above "Exemplary Method" section according to various exemplary embodiments of the present application. For example, the processing unit 1610 can perform the steps S202 to S208 as shown in Figure 2 and other steps defined in the ramp metering strategy learning method of the present disclosure.

[0242] ​Storage 1620 can include a readable medium in the form of volatile memory, such as random access memory (RAM) 16201 and / or cache memory 16202, and can further include read only memory (ROM) 16203.

[0243] Storage 1620 can also include program / utility 16204 having a set (at least one) of program modules 16205, examples of which include an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which implement aspects of the functionality of the network environment.

[0244] Bus 1630 can represent one or more of several types of bus structures, including a storage bus or bus controller, peripheral bus, graphics bus, processor or local bus using any of a variety of bus architectures.

[0245] Electronic device 1600 can also communicate with one or more external devices 1660 such as a keyboard or pointing device, using one or more communication interfaces 1640. Communication interfaces 1640 can also enable electronic device 1600 to communicate with one or more devices that enable user interaction with electronic device 1600 (for example, remote control devices) and / or communication such as with one or more other computing devices. Such communication can be facilitated, for example, by an I / O interface 1640. Electronic device 1600 can communicate with one or more networks, such as a local area network (LAN), a general area network (WAN), or the public networks, such as the Internet, using a network adapter 1650. As shown, network adapter 1650 can be communicatively coupled to the other components of electronic device 1600 through bus 1630. It will be appreciated that other hardware and / or software modules can be used in conjunction with electronic device 1600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0246] Those skilled in the art will readily recognize that the example embodiments described herein can be implemented using software and / or software modules, and that the example embodiments described herein can be implemented using software and / or software modules in conjunction with necessary hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the methods according to the embodiments of the present disclosure.

[0247] In the exemplary embodiments of the present disclosure, a computer readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the above-mentioned "Exemplary Methods" section of the present specification according to various exemplary embodiments of the present application when the program product is run on the terminal device.

[0248] The program product for implementing the above-mentioned method according to the embodiments of the present application can take a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited to this, and in the present document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus or device.

[0249] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which the readable program codes are borne. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic signal, optical signal or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a readable storage medium, which can send, propagate or transmit the program for use by or in conjunction with an instruction execution system, apparatus or device.

[0250] The program codes contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0251] The program codes for performing the operations of the present application can be written in any combination of one or more programming languages, including an object-oriented programming language, such as Java, C++, etc., and a conventional procedural programming language, such as "C" language or similar programming languages. The program codes can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).

[0252] It should be noted that, although several modules or units of the devices for action execution are mentioned in the above detailed description, the division into such modules or units is not mandatory. Indeed, according to an embodiment of the disclosure, the features and functionalities of two or more of the above-described modules or units can be embodied in one module or unit. Conversely, the features and functionalities of one of the above-described modules or units can be further divided into several modules or units.

[0253] Moreover, although the various steps of the methods in the disclosure are described in a particular order in the drawings, this is not required or implied as to the order of performing the steps, nor is it required that all of the illustrated steps be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, one step can be broken into multiple steps, etc.

[0254] From the above description of the embodiments, those skilled in the art will readily perceive that the example embodiments described herein can be implemented by software and / or by hardware coupled with software. Accordingly, the technical solutions according to the embodiments of the disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or network, and includes a number of instructions to make a computing device (which can be a personal computer, server, mobile terminal, or network device, etc.) execute the methods according to the embodiments of the disclosure.

[0255] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features of the disclosure as disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A method for learning ramp merging strategies, characterized in that, include: In the training environment, a cumulative expected return is configured, which represents the cumulative value of the expected return for continuously executing the merging strategy to complete the ramp merging process; Based on the cumulative expected return, a strategy evaluation model and a strategy improvement model are constructed. The strategy evaluation model is used to evaluate the first expected return obtained by following the merging strategy from a specified driving state. The strategy improvement model is used to evaluate the second expected return obtained by using a specified driving action. The specified driving action refers to the driving action selected based on the merging strategy in the specified driving state. With the optimization of the cumulative expected return as the strategy optimization objective, the strategy evaluation model and the strategy improvement model are subjected to alternating iterative learning. Based on the results of alternating iterative learning, a merging strategy to be corrected is obtained. The merging strategy to be corrected is used to output the distributed driving actions when merging into the ramp. In the application environment, collision avoidance correction operations are performed on the distributed driving actions based on a safety shield model. The safety shield model includes a state constraint module. In this application environment, the distributed driving actions are input into the vehicle's lateral and longitudinal dynamic models to output the estimated driving state corresponding to any driving action. The state constraint module is constructed based on vehicle-to-vehicle collision avoidance constraints and vehicle-to-infrastructure collision avoidance constraints. When it is detected that the estimated driving state does not meet the constraints in the state constraint module, a collision avoidance correction operation is performed on the driving action to generate a corrected target merging strategy, enabling the vehicle to perform a ramp merging operation based on the target merging strategy.

2. The ramp merging strategy learning method according to claim 1, characterized in that, The configuration of cumulative expected return in the training environment includes: Obtain an instant reward model for safety status assessment and a policy entropy model for the inflow strategy. The instant reward model is used to assess the safety when the inflow strategy is executed to enter the corresponding driving state, and the policy entropy model is used to measure the uncertainty of the inflow strategy. Configure the cumulative expected return based on the instant return model and the strategy entropy model.

3. The ramp merging strategy learning method according to claim 2, characterized in that, The step of iteratively learning the strategy evaluation model and the strategy improvement model, with the optimization objective of optimizing the cumulative expected return, includes: Obtain the first correlation between the strategy improvement model and the strategy evaluation model; Based on the first correlation, and with maximizing the cumulative expected return as the strategy optimization objective, the strategy evaluation model and the strategy improvement model are subjected to alternating iterative learning.

4. The ramp merging strategy learning method according to claim 3, characterized in that, The step of performing alternating iterative learning on the strategy evaluation model and the strategy improvement model based on the first correlation and with maximizing the cumulative expected return as the strategy optimization objective includes: Based on the first association relationship, a corresponding dynamic programming equation is configured, which is used to express the relationship between the expected returns of two adjacent driving states. Based on the dynamic programming equation, the strategy evaluation result output by the strategy evaluation model and the improved strategy output by the strategy improvement model are calculated alternately, wherein the improved strategy is suitable for maximizing the strategy evaluation result corresponding to the distributed driving action under the initial distribution.

5. The ramp merging strategy learning method according to claim 4, characterized in that, The cumulative expected return includes a temperature coefficient, which is suitable for adaptively adjusting the speed at which the stochastic policy converges to a deterministic policy. Based on the dynamic programming equation, the method of alternately calculating the strategy evaluation result output by the strategy evaluation model and the improved strategy output by the strategy improvement model further includes: The improved strategy converges from the random strategy to the deterministic strategy, and the deterministic strategy is determined as the merged strategy to be modified.

6. The ramp merging strategy learning method according to claim 4, characterized in that, Before alternately calculating the strategy evaluation result output by the strategy evaluation model and the improved strategy output by the strategy improvement model based on the dynamic programming equation, the process further includes: Starting from the specified driving state, the system interacts with the deployed driving environment using the inflow strategy to generate a sequence including the driving state, the driving action, and immediate feedback, so as to determine the cumulative discount reward of the immediate feedback based on the sequence. Determine a second correlation between the cumulative return of the discount and the strategy improvement model; The distribution of the cumulative discount return is modeled based on the second association relationship to obtain a strategy improvement distribution model, which is suitable for optimizing the strategy evaluation results; The dynamic programming equation is configured based on the strategy improvement distribution model to alternately calculate the strategy evaluation result and the improved strategy based on the dynamic programming equation.

7. The ramp merging strategy learning method according to claim 6, characterized in that, The alternating calculation of the strategy evaluation result and the improved strategy includes: The improved strategy is parameterized to obtain a parameterized strategy; The strategy improvement model and the strategy evaluation model are parameterized to obtain a parameterized model; Based on the parameter updates in the parameterization strategy and the parameterization model, the first expected return and the second expected return are updated alternately to output the import strategy to be corrected.

8. The ramp merging strategy learning method according to claim 7, characterized in that, The parameterization process of the improved strategy to obtain the parameterized strategy includes: The input strategy is parameterized using mean and variance to obtain the parameterized strategy including strategy parameters.

9. The ramp merging strategy learning method according to claim 8, characterized in that, The parameterization process for the strategy improvement model and the strategy evaluation model to obtain the parameterized model includes: The value function distribution of the policy improvement model is represented by the policy improvement distribution model; The distribution model is improved based on the strategy of approximating the neural network, resulting in the parameterized model that includes the neural network parameters.

10. The ramp merging strategy learning method according to claim 9, characterized in that, The step of alternately updating the first expected return and the second expected return based on parameter updates in the parameterized strategy and the parameterized model to output the import strategy to be corrected includes: With maximizing the first expected return as the first update objective, the policy parameters are updated using the gradient descent algorithm to obtain the updated policy; Determine the expected return distribution of real driving actions based on empirical data samples; The update strategy is input into the parameterized model, with the second update objective being to minimize the gap between the value function distribution and the expected reward distribution of the actual driving action. The neural network parameters are then optimized and updated to obtain the learned target policy evaluation model and target policy improvement model.

11. The ramp merging strategy learning method according to claim 10, characterized in that, The configuration of the cumulative expected return based on the instant return model and the strategy entropy model further includes: The temperature coefficient of the cumulative expected return is adjusted based on minimizing the difference between the output of the policy entropy model and the expected policy entropy.

12. The ramp merging strategy learning method according to claim 1, characterized in that, The state constraint module for constructing the safe state based on vehicle-to-vehicle collision avoidance constraints and road-to-vehicle collision avoidance constraints includes: The first contour information of the self-car and the second contour information of the peripheral car are generated based on the six-circle model. Based on the first contour information and the second contour information, a first critical safety distance between vehicles and a second critical safety distance between vehicles and roads are determined. Vehicle collision avoidance constraints are generated based on the first critical safety distance; Vehicle-road collision avoidance constraints are generated based on the second critical safety distance; The state constraint module is constructed based on the vehicle-to-vehicle collision avoidance constraints, the vehicle-to-road collision avoidance constraints, and the driving state.

13. The ramp merging strategy learning method according to claim 1, characterized in that, The safety shield model also includes an action correction module. The state constraint module, which constructs the state constraint based on the vehicle-to-vehicle collision avoidance constraint, the vehicle-to-infrastructure collision avoidance constraint, and the driving state, further includes: The conservative adjustment variable is set based on the action correction module, so as to adjust the conservatism of the state constraint module based on the conservative adjustment variable.

14. A ramp merging strategy learning device, characterized in that, include: A configuration module is used to configure the cumulative expected return in the training environment, wherein the cumulative expected return represents the cumulative value of the expected return for continuously executing the merging strategy to complete the ramp merging process; The construction module is used to construct a strategy evaluation model and a strategy improvement model based on the cumulative expected return. The strategy evaluation model is used to evaluate the expected return obtained by following the merging strategy from a specified driving state. The strategy improvement model is used to evaluate the expected return obtained by using a specified driving action. The specified driving action refers to the driving action selected based on the merging strategy in the specified driving state. The learning module is used to perform alternating iterative learning on the strategy evaluation model and the strategy improvement model with the optimization of the cumulative expected return as the strategy optimization objective. Based on the results of the alternating iterative learning, a merging strategy to be corrected is obtained. The merging strategy to be corrected is used to output the distributed driving actions when merging into the ramp. The correction module is used to perform collision avoidance correction operations on the distributed driving actions based on a safety shield model in the application environment. The safety shield model includes a state constraint module. In the application environment, the distributed driving actions are input into the vehicle's lateral and longitudinal dynamic models to output the estimated driving state corresponding to any driving action. The state constraint module constructs the safe state based on vehicle-to-vehicle collision avoidance constraints and vehicle-to-infrastructure collision avoidance constraints. When it is detected that the estimated driving state does not meet the constraints in the state constraint module, a collision avoidance correction operation is performed on the driving action to generate a corrected target merging strategy, enabling the vehicle to perform a ramp merging operation based on the target merging strategy.

15. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the ramp merging strategy learning method according to any one of claims 1 to 13 by executing the executable instructions.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the ramp merging strategy learning method according to any one of claims 1 to 13.

Images