Automatic driving behavior decision method, driving decision device and readable storage medium

Abstract

The invention provides an automatic driving behavior decision-making method, driving decision-making equipment and a readable storage medium, and relates to the technical field of automatic driving. According to the method, the driving behavior strategy network and the driving cost evaluation network obtained through training based on the adaptive dynamic planning technology are called, and the Monte Carlo tree search is carried out based on the real-time driving environment state of the target vehicle, so that the adaptive dynamic planning technology is integrated into each decision stage of the Monte Carlo tree search algorithm; the Monte Carlo tree search algorithm is guided to focus on effective search of high-value decisions, the workload of invalid search is reduced, the resource loss burden and the time loss are calculated, and the decision efficiency and the decision accuracy of the Monte Carlo tree search algorithm are improved. Therefore, the optimal driving action of the target vehicle in the complex driving environment at the current moment can be quickly and reliably decided, so that a driving behavior decision function which is quick, good in robustness and high in decision interpretation can be realized under the complex driving environment of other traffic participants.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to an autonomous driving behavior decision method, a driving decision device, and a readable storage medium. Background Art

[0002] With the development of automobile intelligence and assisted driving technology, autonomous driving, as an advanced stage of assisted driving, has become an important means to solve people's travel problems in the future and has become a new round of research focus and hotspot in the world. Especially in recent years, autonomous driving technology has flourished and has made milestone progress in the history of human transportation development.

[0003] At present, the technical architecture of autonomous driving is mainly divided into three main modules: environmental perception, driving decision planning, and control execution. The driving decision planning module can make decisions on the optimal driving behavior of the autonomous driving vehicle in a dynamic and complex environment based on the external environmental conditions and internal state of the vehicle perceived by the environmental perception module, so as to provide the control execution module with execution, thereby meeting the preset requirements on the basis of ensuring driving safety. Therefore, the performance of driving decision planning will directly affect the final autonomous driving effect. It is worth noting that the existing autonomous driving behavior decision-making methods are difficult to take into account decision efficiency, environmental adaptability, and decision interpretability at the same time in a complex and dynamic environment, and cannot quickly provide autonomous driving vehicles with driving behavior decision results that are adapted to the vehicle driving environment and have strong decision interpretability. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide an autonomous driving behavior decision-making method, a driving decision-making device and a readable storage medium, which can integrate adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm in the Monte Carlo tree search process based on the vehicle driving environment conditions, so as to effectively reduce the invalid search workload and improve the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm, thereby realizing a fast, robust and highly interpretable driving behavior decision-making function in a complex driving environment taking into account other traffic participants.

[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, the present application provides an autonomous driving behavior decision-making method, the method comprising:

[0007] Obtain the actual driving environment state of the target vehicle at the current moment;

[0008] The driving behavior strategy network and the driving cost evaluation network are called to perform a Monte Carlo tree search based on the actual driving environment state, and when the search termination condition is met, the optimal child node of the root node in the corresponding Monte Carlo tree that meets the driving behavior cost minimization requirement is determined; wherein all nodes in the Monte Carlo tree each represent a driving environment state, the root node represents the actual driving environment state, the initial driving behavior costs of all nodes are obtained by the driving cost evaluation network based on the driving behavior prediction result of the node and the driving environment state evaluation, and the driving behavior prediction result of all nodes is obtained by the driving behavior strategy network based on the driving environment state estimation of the node; the driving behavior strategy network and the driving cost evaluation network are obtained based on adaptive dynamic programming structure training;

[0009] The driving behavior action between the optimal child node and the root node is used as the optimal driving action of the target vehicle at the current moment.

[0010] In an optional implementation, the step of calling the driving behavior strategy network and the driving cost evaluation network to perform a Monte Carlo tree search based on the actual driving environment state includes:

[0011] In each Monte Carlo tree search process, according to the actual driving behavior cost and the actual number of visits of each node in the Monte Carlo tree, the optimal child node is selected layer by layer starting from the root node to determine the optimal node path in this Monte Carlo tree search process, wherein the benchmark driving action of each node in the Monte Carlo tree is the estimated driving behavior result of the node, and the driving behavior action between each non-root node and its parent node on the optimal node path is obtained by adding action noise to the benchmark driving action of the corresponding parent node;

[0012] Expanding the end node on the optimal node path to a sub-node, and using the estimated driving behavior result of the end node as a benchmark driving action from the end node to the expanded target sub-node;

[0013] Perform a driving preview according to the driving environment state and driving behavior estimation result of the target subnode, and set the actual number of visits to the target subnode to 1;

[0014] Based on the initial driving behavior cost of the target child node, the actual number of visits and the actual driving behavior cost of each node on the optimal node path are backtracked and updated.

[0015] In an optional embodiment, the method further comprises:

[0016] Acquire multiple driving network training samples, wherein each driving network training sample includes a historical driving environment state, a historical driving action, and a comprehensive attenuated driving cost of a sample vehicle at a corresponding historical moment, and a historical driving environment state of the sample vehicle at a target moment associated with the historical moment; the target moment is separated from the historical moment by a first preset number of control moments, the comprehensive attenuated driving cost is the sum of comprehensive driving cost attenuation values ​​of the sample vehicle at a second preset number of consecutive control moments starting from the historical moment, and the second preset number is obtained by adding one to the first preset number;

[0017] According to the multiple driving network training samples, an adaptive dynamic programming structure including an execution network and an evaluation network is iteratively trained to obtain the driving behavior strategy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network.

[0018] In an optional embodiment, the step of iteratively training the adaptive dynamic programming structure including the execution network and the evaluation network according to the multiple driving network training samples to obtain the driving behavior strategy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network includes:

[0019] For each iterative training operation of the execution network, determine the initial network weight of the execution network in this iterative training operation, and iteratively optimize the evaluation network based on the initial network weight and the multiple driving network training samples to obtain an evaluation network that meets the iterative optimization termination condition of this iterative training operation;

[0020] Based on the multiple driving network training samples and the evaluation network that meets the iterative optimization termination condition, the network weight of the execution network is optimized to obtain the target network weight of the execution network in this iterative training operation;

[0021] Detecting whether the target network weights meet the iterative training end condition;

[0022] If it is detected that the target network weight meets the iterative training termination condition, the execution network with the target network weight is directly used as the driving behavior strategy network, and the evaluation network that meets the iterative optimization termination condition of this iterative training operation is used as the driving cost evaluation network, otherwise the next iterative training operation is performed on the execution network, wherein the initial network weight of the execution network in the next iterative training operation is the target network weight of this iterative training operation.

[0023] In an optional embodiment, during each iterative training operation of the execution network, the network iterative optimization process of the evaluation network is expressed by the following equation:

[0024]

[0025] Among them, γ is used to represent the discount factor of the evaluation network, s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u are all comprehensive weights of costs, J j+1 (s t ,u t ) is used to represent the evaluation network after the j+1th iteration optimization for the historical driving environment state s t and historical driving actions t Predicted driving behavior cost, u(s t+n+1 ) is used to represent the execution network for the historical driving environment state s t+n+1 Predicted estimated driving behavior, J j (s t+n+1 ,u(s t+n+1 )) is used to represent the evaluation network after the j-th iteration optimization for the historical driving environment state s t+n+1 and estimated driving behavior u(s t+n+1 ) predicted driving behavior cost.

[0026] In an optional embodiment, the step of iteratively optimizing the evaluation network based on the initial network weights and the plurality of driving network training samples to obtain an evaluation network that satisfies an iterative optimization termination condition of this iterative training operation includes:

[0027] For each iterative optimization operation of the evaluation network during the iterative training operation of the execution network, determining a reference network weight of the evaluation network in the iterative optimization operation;

[0028] Based on the reference network weight, the initial network weight and the plurality of driving network training samples, the network weight of the evaluation network is updated to obtain the expected network weight of the evaluation network in this iterative optimization operation;

[0029] Detecting whether the expected network weights meet the iterative optimization termination condition;

[0030] If it is detected that the expected network weight meets the iterative optimization termination condition, the next iterative optimization operation is stopped and the evaluation network with the expected network weight is output. Otherwise, the next iterative optimization operation is performed on the evaluation network, wherein the reference network weight of the evaluation network in the next iterative optimization operation is the expected network weight of this iterative optimization operation.

[0031] In an optional embodiment, the network weight update process of the evaluation network in the j+1th iterative optimization operation during the i-th iterative training operation of the execution network is expressed by the following equation:

[0032]

[0033] Among them, W c,j+1 is used to represent the expected network weight of the evaluation network in the j+1th iteration optimization operation during the i-th iteration training operation of the execution network, W c,j is used to represent the expected network weight of the evaluation network during the jth iteration optimization operation of the execution network during the i-th iteration training operation, α c It is used to represent the learning rate of the evaluation network, W c is used to represent the network weight of the evaluation network, γ is used to represent the discount factor of the evaluation network, and s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, u i (s l |W a) is used to indicate that the execution network uses the initial network weight W during the i-th iteration training operation. a For historical driving environment status s l Predicted estimated driving behavior, J(s t ,u i (s t |W a )|W c ) is used to represent the evaluation network for the historical driving environment state s t and estimated driving behavior u i (s t |W a ) predicts the network weight W c The driving behavior cost, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u All are comprehensive weights of costs.

[0034] In an optional embodiment, the network weight optimization process of executing the network in the i-th iteration training operation is expressed by the following equation:

[0035]

[0036] Among them, γ is used to represent the discount factor of the evaluation network, s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, u i (s t ) is used to represent the execution network for the historical driving environment state s when completing the i-th iteration training operation t The predicted driving behavior, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u are all comprehensive weights of costs, u i (s t+n+1 ) is used to represent the execution network for the historical driving environment state s when completing the i-th iteration training operation t+n+1 Predicted estimated driving behavior, It is used to indicate that the evaluation network meets the iterative optimization termination condition during the i-th iterative training operation for the historical driving environment state s t+n+1 and estimated driving behavior u(s t+n+1 ) predicted driving behavior cost, W a is used to represent the network weight to be optimized of the execution network, and σ(·) is used to represent the tanh activation function.

[0037] In a second aspect, the present application provides a driving decision-making device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the autonomous driving behavior decision method described in any one of the aforementioned embodiments.

[0038] In a third aspect, the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a computer device, the autonomous driving behavior decision method described in any one of the aforementioned embodiments is implemented.

[0039] In this case, the beneficial effects of the embodiments of the present application may include the following:

[0040] The present application calls the driving behavior strategy network and the driving cost evaluation network trained based on the adaptive dynamic programming technology, and performs a Monte Carlo tree search based on the actual driving environment state of the target vehicle at the current moment, so that the initial driving behavior cost of each node in the Monte Carlo tree is obtained by the driving cost evaluation network based on the driving behavior prediction result and the driving environment state evaluation of the node, and the driving behavior prediction result of each node is obtained by the driving behavior strategy network based on the driving environment state estimation of the node, so as to integrate the adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm, guide the Monte Carlo tree search algorithm to focus on the effective search of high-value decisions, reduce the workload of invalid searches, the burden of computing resource loss and time loss, and improve the decision efficiency and accuracy of the Monte Carlo tree search algorithm, so as to quickly and reliably decide the optimal driving action of the target vehicle in a complex driving environment at the current moment through the organic combination of the Monte Carlo tree search algorithm with strong decision-making interpretability and the adaptive dynamic programming technology with strong environmental adaptability, so as to realize a fast, robust and decision-making interpretability driving behavior decision-making function in a complex driving environment considering other traffic participants.

[0041] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of the components of a driving decision-making device provided in an embodiment of the present application;

[0044] Figure 2 One of the flowcharts of the autonomous driving behavior decision-making method provided in an embodiment of the present application;

[0045] Figure 3 for Figure 2A schematic flow chart of the sub-steps included in step S220;

[0046] Figure 4 A second flowchart of the autonomous driving behavior decision-making method provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the structure of the adaptive dynamic programming structure provided in the embodiment of the present application;

[0048] Figure 6 for Figure 4 Schematic diagram of the flow chart of the sub-steps included in step S250.

[0049] Icons: 10 - driving decision device; 11 - memory; 12 - processor; 13 - communication unit. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0052] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0053] In the description of the present application, it should be understood that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is conventionally placed when in use, or are the orientations or positional relationships conventionally understood by those skilled in the art. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0054] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0055] In the description of the present application, it is to be understood that the relational terms such as the term "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "includes", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "comprising one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood by specific circumstances.

[0056] Through painstaking research, the applicant discovered that the existing autonomous driving behavior decision-making methods are mainly implemented using three decision-making ideas: preset rules, search algorithms, and reinforcement learning; among them, the autonomous driving behavior decision-making method based on preset rules cannot effectively predict the behavior of other traffic participants in complex driving environments; although the autonomous driving behavior decision-making method based on the search algorithm can find a better decision path through global search, and has a certain degree of decision-making interpretability because the global search process is observable, this decision-making method often has the problems of high computational complexity and insufficient real-time performance, especially when facing large-scale state spaces, it is difficult to achieve fast and effective decision-making functions; although the autonomous driving behavior decision-making method based on reinforcement learning has a certain ability to adapt to the environment and can make fast decisions, this decision-making method is highly dependent on the quality and scale of training data, and it is difficult to reasonably explain the given decisions, and does not have strong decision-making interpretability.

[0057] In this case, in order to solve the above problems, the embodiments of the present application provide an autonomous driving behavior decision-making method, a driving decision-making device and a readable storage medium, which can realize a driving behavior decision-making function that is fast, robust and has strong decision-interpretation in a complex driving environment taking into account other traffic participants.

[0058] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0059] Please refer to Figure 1 , Figure 1 1 is a schematic diagram of the driving decision device 10 provided in an embodiment of the present application. In the embodiment of the present application, the driving decision device 10 can be connected to an autonomous driving vehicle equipped with an autonomous driving system to obtain real-time driving environment information of the autonomous driving vehicle during actual driving, and based on the acquired driving environment information, quickly and reliably decide the optimal driving action of the autonomous driving vehicle in a complex driving environment, so that the autonomous driving system of the autonomous driving vehicle can control the vehicle operation according to the optimal driving action decided, thereby meeting the preset requirements on the basis of ensuring driving safety. In this process, the driving environment condition information is used for the relative position relationship, relative posture relationship, relative speed change relationship, relative acceleration change relationship and other information between the autonomous driving vehicle and other static targets (for example, lane lines and road boundaries, green belts, etc.) and other traffic participants (for example, social vehicles, pedestrians, etc.) in the corresponding driving environment; the driving behavior decision (optimal driving action) given by the driving decision device 10 may involve the motion acceleration, steering angle, braking degree and other information of the corresponding autonomous driving vehicle to directly affect the vehicle's motion state; the driving decision device 10 can be integrated with the autonomous driving vehicle that requires driving behavior decision. At this time, the driving decision device 10 is the on-board device of the autonomous driving vehicle and is part of the autonomous driving system; the driving decision device 10 can also be a computer device independent of the autonomous driving vehicle. The computer device can be but is not limited to: a server, an external computer, etc.

[0060] In the embodiment of the present application, the driving decision device 10 may include a memory 11, a processor 12 and a communication unit 13. The memory 11, the processor 12 and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, the memory 11, the processor 12 and the communication unit 13 may be electrically connected to each other via one or more communication buses or signal lines.

[0061] In the embodiment of the present application, the memory 11 may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 11 is used to store a computer program, and the processor 12 may execute the computer program accordingly after receiving an execution instruction.

[0062] In the embodiment of the present application, the processor 12 may be an integrated circuit chip with signal processing capability. The processor 12 may be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., which may implement or execute the disclosed methods, steps, and logic block diagrams in the embodiment of the present application.

[0063] In an embodiment of the present application, the communication unit 13 is used to establish a communication connection between the driving decision device 10 and other electronic devices through a network, and to send and receive data through the network, wherein the network includes a wired communication network and a wireless communication network.

[0064] In an embodiment of the present application, the driving decision device 10 may pre-store a specific computer program related to the automatic driving behavior decision function in the memory 11, and by driving the processor 12 to execute the specific computer program stored in the memory 11, in the Monte Carlo tree search process based on the real-time vehicle driving environment, the adaptive dynamic programming technology is integrated into each decision stage of the Monte Carlo tree search algorithm, and the Monte Carlo tree search algorithm is guided to focus on the effective search for high-value decisions, reduce the workload of invalid searches, the burden of computing resource loss and time loss, and improve the decision efficiency and decision accuracy of the Monte Carlo tree search algorithm. In this way, through the organic combination of the Monte Carlo tree search algorithm with strong decision interpretability (that is, the "selection", "expansion", "preview" and "backtracking" of each decision stage of the Monte Carlo tree search algorithm can be observed) and the adaptive dynamic programming technology with strong environmental adaptability (that is, the adaptive dynamic programming technology can consider the real-time feedback mechanism of the vehicle driving environment information to perform model adaptive learning optimization to adapt to the dynamic traffic environment changes), a driving behavior decision function with fast, good robustness and strong decision interpretability is achieved in a complex driving environment considering other traffic participants.

[0065] Understandably, Figure 1 The block diagram shown is only a schematic diagram of a composition of the driving decision-making device 10. The driving decision-making device 10 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0066] In this application, in order to ensure that the driving decision device 10 can achieve a fast, robust and interpretable driving behavior decision function in a complex driving environment taking into account other traffic participants, the embodiment of this application achieves the above-mentioned purpose by providing an automatic driving behavior decision method. The automatic driving behavior decision method provided by this application is described in detail below.

[0067] Please refer to Figure 2 , Figure 2 It is one of the flow charts of the autonomous driving behavior decision method provided in the embodiment of the present application. In the embodiment of the present application, the autonomous driving behavior decision method may include steps S210 to S230.

[0068] Step S210, obtaining the actual driving environment state of the target vehicle at the current moment.

[0069] In this embodiment, the target vehicle is an autonomous driving vehicle that is communicatively connected to the driving decision-making device 10, and the actual driving environment state records information such as the relative position relationship, relative posture relationship, etc. between the target vehicle and other surrounding static targets and other traffic participants at the current moment.

[0070] Step S220, calling the driving behavior strategy network and the driving cost evaluation network to perform a Monte Carlo tree search based on the actual driving environment state, and determining the optimal child node of the root node in the Monte Carlo tree that meets the driving behavior cost minimization requirement when the search termination condition is met.

[0071] In this embodiment, the driving behavior strategy network and the driving cost evaluation network belong to the same adaptive dynamic programming structure; the driving behavior strategy network is used to estimate the driving action for the driving environment state at any control moment, and obtain the estimated driving behavior corresponding to the driving environment state (i.e., the driving behavior estimation result); the driving cost evaluation network is used to predict the driving behavior cost for the driving environment state and the estimated driving behavior at any control moment, and obtain the overall driving behavior cost generated by the vehicle in the future period when it operates in the driving environment state according to the corresponding estimated driving behavior (obtained by accumulating the comprehensive driving costs generated by all control moments in the future period, and the total number of control moments involved in the future period is fixed). In the process of training and optimizing the network models of the driving behavior strategy network and the driving cost evaluation network, a real-time feedback mechanism of the vehicle driving environment information can be considered, and the adaptive dynamic programming structure to which the driving behavior strategy network and the driving cost evaluation network belong is trained and learned with the model optimization purpose of "minimizing the driving behavior cost predicted by the driving cost evaluation network", so as to ensure that the driving behavior strategy network can adapt to various complex traffic environments, and at the same time, the driving behavior prediction results output by the driving behavior strategy network for any driving environment state have sufficiently high accuracy and reliability, and can efficiently reach the destination while ensuring the safety of the vehicle, and the driving cost evaluation network can also accurately evaluate the driving behavior cost caused by any driving environment state and any driving behavior prediction result in a fixed time period in the future.

[0072] In this process, for the driving cost evaluation network, the driving behavior cost evaluated by it can be expressed by the following function expression:

[0073]

[0074] Among them, s l It is used to represent the driving environment state at the control time l, u l It is used to represent the vehicle driving action (or estimated driving behavior) at the control time l, J(s t0,u t0 ) is used to represent the driving environment state s at the control time t0 t0 and vehicle driving action u t0 The driving behavior cost caused in a fixed time period in the future (the total number of control moments included is T+1), U(s l ,u l ) is used to indicate the driving environment state s l and vehicle driving action u l Matching comprehensive driving cost, J s (s l ) is used to indicate the driving environment state s l The security cost of J c (s l ) is used to indicate the driving environment state s l The comfort cost of J p (s l ) is used to indicate the driving environment state s l The passability cost, J u (s l ,u l ) is used to indicate the driving environment state s l and vehicle driving action u l The quadratic cost of matching, ω s ,ω c ,ω p and ω u All are comprehensive weights of costs.

[0075] Among them, the safety cost is inversely correlated with the distance from the corresponding vehicle to the surrounding objects (including static targets and traffic participants). For example, when the distance from the corresponding vehicle to a surrounding object exceeds a certain distance threshold, the safety cost caused by the object alone to the corresponding vehicle is zero; when the distance from the corresponding vehicle to a surrounding object is greater than 0 but less than or equal to the distance threshold, the safety cost caused by the object alone to the corresponding vehicle can be represented by the inverse of the distance; and when the distance from the corresponding vehicle to a surrounding object is 0, the safety cost caused by the object alone to the corresponding vehicle can be expressed as infinity. The comfort cost is positively correlated with the acceleration change rate of the vehicle, which represents the smoothness of the vehicle's movement, wherein the greater the acceleration change rate, the greater the corresponding comfort cost. The passability cost is related to the distance from the vehicle to the target location, which represents the goal orientation of the vehicle's driving action. The quadratic cost can be expressed as Wherein Q and R are both custom weights. In one implementation of this embodiment, if the driving environment state of a vehicle indicates that there are multiple objects around the vehicle, the safety cost corresponding to the driving environment state is obtained by adding the safety costs caused by the multiple objects to the vehicle respectively.

[0076] In this embodiment, after obtaining the actual driving environment state of the target vehicle at the current moment, the driving decision-making device 10 will call the trained driving behavior strategy network and the driving cost evaluation network. In the Monte Carlo tree search process based on the actual driving environment state, the output value of the driving behavior strategy network (i.e., the driving behavior prediction result output for the driving environment state represented by each node in the Monte Carlo tree) is used as guidance information, and the output value of the driving cost evaluation network (i.e., the driving behavior cost output for the driving environment state and the driving behavior prediction result of each node in the Monte Carlo tree) is used to characterize the value function result in the Monte Carlo tree algorithm, so as to realize the organic combination between the adaptive dynamic programming technology and each decision stage of the Monte Carlo tree search algorithm, so that the corresponding Monte Carlo tree search algorithm focuses on the effective search for high-value decisions, reduces the workload of invalid search, the burden of computing resource loss and the time loss, and improves the decision efficiency and accuracy of the Monte Carlo tree search algorithm.

[0077] Therefore, in the Monte Carlo tree search process based on the actual driving environment state, all nodes in the corresponding constructed Monte Carlo tree each represent a driving environment state, the root node in the Monte Carlo tree represents the actual driving environment state, and the initial driving behavior cost of all nodes in the Monte Carlo tree when they are first visited (that is, the actual number of visits is 1) can be directly obtained by the driving cost evaluation network based on the driving behavior prediction result of the node and the driving environment state evaluation, wherein the driving behavior prediction result of all nodes in the Monte Carlo tree is obtained by the driving behavior strategy network based on the driving environment state estimation of the node.

[0078] Meanwhile, during the implementation of the Monte Carlo tree search, the driving decision device 10 will confirm whether the Monte Carlo tree search operation is the last Monte Carlo tree search operation by detecting whether the Monte Carlo tree search operation satisfies the search termination condition (for example, the total number of searches is equal to the preset number of searches, or the total search time reaches the preset time) when completing a Monte Carlo tree search operation. When the Monte Carlo tree search operation does not satisfy the search termination condition, it indicates that the Monte Carlo tree search operation is not the last Monte Carlo tree search operation. At this time, the driving decision device 10 will continue to perform the next Monte Carlo tree search operation based on the Monte Carlo tree obtained by the Monte Carlo tree search operation until the search termination condition is finally satisfied.

[0079] When the Monte Carlo tree search operation satisfies the search termination condition, it indicates that the Monte Carlo tree search operation is the last Monte Carlo tree search operation. The driving decision device 10 will select child nodes for the root node of the target Monte Carlo tree obtained after the Monte Carlo tree search operation is completed, with the purpose of "ensuring the minimization effect of the corresponding driving behavior cost", and obtain the optimal child node of the root node (that is, the child node of the root node with the smallest actual driving behavior cost in the target Monte Carlo tree). At this time, the driving environment state represented by the optimal child node of the root node is the ideal driving environment state that the target vehicle expects to obtain at the next control moment at the current moment.

[0080] Optionally, see Figure 3 , Figure 3 yes Figure 2 Schematic diagram of the flow chart of the sub-steps included in step S220. In an embodiment of the present application, the step of "calling the driving behavior strategy network and the driving cost evaluation network to perform a Monte Carlo tree search based on the actual driving environment state" in step S220 may include sub-steps S221 to S224, so as to integrate the adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm to effectively search for high-value decisions, reduce the workload of invalid searches, the burden of computing resource loss and time loss, and improve the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm.

[0081] Sub-step S221, in each Monte Carlo tree search process, according to the actual driving behavior cost and the actual visit number of each node in the Monte Carlo tree, the optimal child node is selected layer by layer starting from the root node to determine the optimal node path in this Monte Carlo tree search process.

[0082] In this embodiment, in the Monte Carlo tree constructed based on the actual driving environment state, the actual driving behavior cost of each node in the Monte Carlo tree when it is first visited (i.e., the actual number of visits is 1) is the initial driving behavior cost of the corresponding node; for the "selection" stage in each Monte Carlo tree search process, the driving decision device 10 will start from the root node in the current Monte Carlo tree, layer by layer according to the actual driving behavior cost and the actual number of visits of each existing node, with the benchmark driving action of each node (i.e., the driving behavior prediction result of the corresponding node) as the selection guide, and with "ensuring that the cost and value between the actual driving behavior costs of all nodes on the corresponding node path are minimized" as the selection purpose, and adopt the UCB (Upper Confidence Bound, confidence interval upper bound) algorithm to select the optimal child node, so as to form the optimal node path in this Monte Carlo tree search process. Among them, two adjacent nodes on the optimal node path satisfy the parent-child node relationship, and the driving behavior action between each non-root node on the optimal node path and its parent node is obtained by adding a set action noise (for example, Gaussian noise) to the benchmark driving action of the corresponding parent node. By superimposing the action noise on the benchmark driving action, the strategy selection relationship between "utilization" and "exploration" is balanced, and the path diversity of the optimal node path is enhanced.

[0083] Sub-step S222, performing sub-node expansion on the end node on the optimal node path, and using the driving behavior estimation result of the end node as the reference driving action from the end node to the expanded target sub-node.

[0084] In this embodiment, for the "expansion" stage in each Monte Carlo tree search process, the driving decision device 10 can use the driving behavior estimation result of the end node on the optimal node path as the benchmark driving action between the end node and the new expansion node, and predict the future environment state based on the driving environment state of the end node and the benchmark driving action, thereby expanding the target child node on the basis of the current Monte Carlo tree, so as to provide diversified path selection for subsequent "preview" and "backtracking" stages, thereby improving the effectiveness of the search, reducing the search workload, computing resource loss burden and time loss caused by invalid search, and improving the decision efficiency and accuracy of the Monte Carlo tree search algorithm.

[0085] Sub-step S223, performing a driving rehearsal according to the driving environment state and the driving behavior estimation result of the target sub-node, and setting the actual number of visits to the target sub-node to 1.

[0086] In this embodiment, for the "preview" stage in each Monte Carlo tree search process, the driving decision device 10 starts from the target sub-node and uses the driving behavior prediction result and driving environment state of the target sub-node to perform a driving preview simulation until the simulation termination condition is met. At this time, the actual number of visits to the target sub-node can be set to 1, and the driving cost evaluation network is called to evaluate the driving behavior cost that matches the driving behavior prediction result and driving environment state of the target sub-node as the initial driving behavior cost of the target sub-node.

[0087] Sub-step S224 , based on the initial driving behavior cost of the target child node, backtracking and updating the actual number of visits and the actual driving behavior cost of each node on the optimal node path.

[0088] In this embodiment, for the "backtracking" stage in each Monte Carlo tree search process, the driving decision device 10 will, based on the initial driving behavior cost of the newly expanded target child node, reverse the node arrangement order of the optimal node path, and sequentially add one to the actual number of visits to each node on the optimal node path to obtain the actual number of visits to each node on the optimal node path after the Monte Carlo tree search is completed. At the same time, for each node on the optimal node path, the formula "q=q′+(q L -q′) / N” to update the actual driving behavior cost of each node, so as to ensure that the subsequent search operation can more effectively select the low-cost (high-value) node path, so that the Monte Carlo tree search algorithm can focus more on the effective search for high-value decisions. Among them, q is used to represent the actual driving behavior cost of the corresponding node after the update, q′ is used to represent the actual driving behavior cost of the corresponding node before the update, N is used to represent the actual number of visits to the corresponding node after the update, and q L Used to represent the initial driving behavior cost of the newly expanded target child node.

[0089] Therefore, the present application can integrate adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm by executing the above-mentioned sub-steps S221 to S224 to conduct effective search for high-value decisions, reduce the workload of invalid searches, the burden of computing resource loss and time loss, and improve the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm.

[0090] Step S230: taking the driving behavior action between the optimal child node and the root node as the optimal driving action of the target vehicle at the current moment.

[0091] In this embodiment, after determining the optimal child node of the root node from the target Monte Carlo tree that meets the search termination condition, the driving decision device 10 can extract the driving behavior action between the root node and the corresponding optimal child node from the target Monte Carlo tree as the optimal driving action of the target vehicle at the current moment in the current complex traffic environment, thereby realizing a driving behavior decision function that is fast, robust and has strong decision-making interpretability.

[0092] Therefore, the present application can integrate the adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm in the Monte Carlo tree search process based on the real-time vehicle driving environment conditions by executing the above steps S210 to S230, guide the Monte Carlo tree search algorithm to focus on the effective search for high-value decisions, reduce the workload of invalid searches, the burden of computing resource loss and time loss, and improve the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm. In this way, through the organic combination of the Monte Carlo tree search algorithm with strong decision-making interpretability and the adaptive dynamic programming technology with strong environmental adaptability, a driving behavior decision-making function that is fast, robust and has strong decision-making interpretability can be achieved in a complex driving environment taking into account other traffic participants.

[0093] Optionally, see Figure 4 , Figure 4 This is the second flow chart of the autonomous driving behavior decision method provided in the embodiment of the present application. Figure 2 Compared with the autonomous driving behavior decision method shown in Figure 4 The autonomous driving behavior decision method shown may also include steps S240 to S250 to ensure that the driving behavior strategy network and the driving cost evaluation network can perform model adaptive learning optimization based on the real-time feedback mechanism of the vehicle driving environment information to adapt to dynamic traffic environment changes and improve the network reliability and network robustness of the driving behavior strategy network and the driving cost evaluation network in complex driving environments.

[0094] Step S240, obtaining a plurality of driving network training samples, wherein each driving network training sample includes a historical driving environment state, a historical driving action and a comprehensive attenuated driving cost of a sample vehicle at a corresponding historical moment, and a historical driving environment state of the sample vehicle at a target moment associated with the historical moment.

[0095] In this embodiment, the reference Figure 5 The adaptive dynamic programming structure adopted in this application includes an execution network, a dynamic system and two evaluation networks, wherein the execution network corresponds to the driving behavior strategy network, the evaluation network corresponds to the driving cost evaluation network, and the dynamic system is used to control the input state s of the execution network at the control time t.t and output action u t Deducing the output action u at the target control time t+n+1 t+n+1 , so that one of the two evaluation networks is solely responsible for the input state s t and output action u t The driving behavior cost evaluation task, another evaluation network is responsible for the input state s t+n+1 and output action u t+n+1 The driving behavior cost evaluation operation is then performed through the externally introduced comprehensive attenuation driving cost from control time t to control time t+n The network optimization adjustment is performed to achieve the model training optimization effect of the adaptive dynamic programming structure. Wherein, n is used to represent the first preset number (its value can be 0, or 2, 3, or 5), and γ is used to represent the discount factor of the evaluation network.

[0096] On this basis, each driving network training sample acquired by the driving decision device 10 for the driving behavior strategy network and the driving cost evaluation network shall include the historical driving environment state, historical driving action and comprehensive attenuated driving cost of the corresponding sample vehicle at the corresponding historical moment, as well as the historical driving environment state of the sample vehicle at the target moment associated with the historical moment. The sample vehicle may be the above-mentioned target vehicle or other autonomous driving vehicles; different driving network training samples correspond to different historical time points (moments); the target moment of a single driving network training sample is separated from the corresponding historical moment by a first preset number of control moments (for example, for historical moment t, the corresponding target moment is t+n+1); the comprehensive attenuated driving cost included in a single driving network training sample is the comprehensive driving cost attenuation value of the sample vehicle for the second preset number of control moments (including the historical moment) starting from the corresponding historical moment (for example, for historical moment t, the comprehensive driving cost attenuation value of the control moment l after the historical moment is γ l-t U(s l ,u l )) (For example, for the historical moment t, the corresponding comprehensive attenuation driving cost is ), the second preset number is obtained by adding one to the first preset number.

[0097] It can be understood that the driving decision-making device 10 can construct a data pool with a fixed capacity separately for the target vehicle. The driving decision-making device 10 will collect the real-time driving environment status and real-time driving actions of the target vehicle to construct driving network training samples, and then add the constructed driving network training samples to the data pool, and delete the oldest driving network training samples stored in the data pool at the same time, so that each driving network training sample saved in the data pool can effectively reflect the dynamic traffic environment changes of the target vehicle, so as to batch extract samples at the data pool (for the driving behavior strategy network and the driving cost evaluation network) for model adaptive learning optimization.

[0098] Step S250, iteratively training the adaptive dynamic programming structure including the execution network and the evaluation network according to the plurality of driving network training samples, to obtain a driving behavior strategy network corresponding to the execution network and a driving cost evaluation network corresponding to the evaluation network.

[0099] In this embodiment, for the adaptive dynamic programming structure, the network function of the execution network can be described as Where σ(·) is used to represent the hidden layer activation function of the execution network (e.g., tanh activation function), W a It is used to represent the network weights between the hidden layer and the output layer of the execution network. In order to consider the output saturation characteristics of the actuator during the actual task execution process, an activation function (for example, tanh activation function) can also be used in the output layer of the execution network for model training; the network function of the evaluation network can be described as Where σ(·) is used to represent the hidden layer activation function of the evaluation network (e.g., tanh activation function), W c It is used to represent the network weight between the hidden layer and the output layer of the evaluation network, ξ is used to represent the learning coefficient of the evaluation network, and in order to ensure that the network function and the above-mentioned driving behavior cost function expression maintain positive definite effects, the ReLu activation function can be used in the output layer of the evaluation network for model training. Among them, multiple iterative optimization operations of the evaluation network are nested in one iterative training operation of the execution network.

[0100] For details, please refer to Figure 6 , Figure 6 yes Figure 4In the embodiment of the present application, the step S250 may include sub-steps S251 to S254 to ensure that the finally optimized driving behavior strategy network and driving cost evaluation network can effectively adapt to the dynamic traffic environment changes, and ensure that the driving behavior prediction results output by the corresponding driving behavior strategy network for any driving environment state have sufficiently high accuracy and reliability, and at the same time, the corresponding driving cost evaluation network can also accurately evaluate the driving behavior cost caused by any driving environment state and any driving behavior prediction results in a fixed time period in the future.

[0101] Sub-step S251, for each iterative training operation of the execution network, determine the initial network weights of the execution network in this iterative training operation, and iteratively optimize the evaluation network based on the initial network weights and multiple driving network training samples to obtain an evaluation network that meets the iterative optimization termination conditions of this iterative training operation.

[0102] In this embodiment, the initial network weight of the execution network in the first iteration training operation can be formed by a random seed algorithm; the initial network weight of the execution network in the non-first iteration training operation is the target network weight finally determined by the last iteration training operation of the non-first iteration training operation. In each iteration training operation of the execution network, the network iteration optimization process of the evaluation network can be expressed by the following equation:

[0103]

[0104] Among them, γ is used to represent the discount factor of the evaluation network, s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, Ju (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u are all comprehensive weights of costs, J j+1 (s t ,u t ) is used to represent the evaluation network after the j+1th iteration optimization for the historical driving environment state s t and historical driving actions t Predicted driving behavior cost, u(s t+n+1 ) is used to represent the execution network for the historical driving environment state s t+n+1 Predicted estimated driving behavior, J j (s t+n+1 ,u(s t+n+1 )) is used to represent the evaluation network after the j-th iteration optimization for the historical driving environment state s t+n+1 and estimated driving behavior u(s t+n+1 ) predicted driving behavior cost.

[0105] At the same time, during each iterative training operation of the execution network, the evaluation network adopts the same iterative optimization termination condition (for example, the corresponding iterative optimization number is equal to the preset optimization number, or the actual loss function value of the evaluation network after optimization is less than the preset value), so that multiple iterative optimization operations of the evaluation network are nested in one iterative training operation of the execution network. It can be understood that the actual iterative optimization number of the evaluation network in different iterative training operations of the execution network can be different.

[0106] Thus, during an iterative training operation of the execution network, the step of “iteratively optimizing the evaluation network based on the initial network weights and the plurality of driving network training samples to obtain an evaluation network that satisfies the iterative optimization termination condition of this iterative training operation” may include:

[0107] For each iterative optimization operation of the evaluation network during the iterative training operation of the execution network, determining a reference network weight of the evaluation network in the iterative optimization operation;

[0108] Based on the reference network weight, the initial network weight and the plurality of driving network training samples, the network weight of the evaluation network is updated to obtain the expected network weight of the evaluation network in this iterative optimization operation;

[0109] Detecting whether the expected network weights meet the iterative optimization termination condition;

[0110] If it is detected that the expected network weight meets the iterative optimization termination condition, the next iterative optimization operation is stopped and the evaluation network with the expected network weight is output. Otherwise, the next iterative optimization operation is performed on the evaluation network, wherein the reference network weight of the evaluation network in the next iterative optimization operation is the expected network weight of this iterative optimization operation.

[0111] Among them, the reference network weights used in the first iterative optimization operation of the evaluation network during the first iterative training operation of the execution network can be set to a zero matrix; the reference network weights used in the first iterative optimization operation of the evaluation network during the non-first iterative training operation of the execution network can also be set to a zero matrix. In order to improve the iterative optimization efficiency of the evaluation network, the expected network weights of the evaluation network that meet the iterative optimization termination conditions in any iterative training operation process (corresponding to the execution network) can be assigned to the reference network weights of the first iterative optimization operation (corresponding to the evaluation network) in the next iterative training operation process.

[0112] During the i-th iterative training operation of the execution network, the network weight update process of the j+1-th iterative optimization operation of the evaluation network is expressed by the following equation:

[0113]

[0114] Among them, W c,j+1 is used to represent the expected network weight of the evaluation network in the j+1th iteration optimization operation during the i-th iteration training operation of the execution network, W c,j is used to represent the expected network weight of the evaluation network during the jth iteration optimization operation of the execution network during the i-th iteration training operation, α c It is used to represent the learning rate of the evaluation network, W c is used to represent the network weight of the evaluation network, γ is used to represent the discount factor of the evaluation network, and s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, u i (s l |W a ) is used to indicate that the execution network uses the initial network weight W during the i-th iteration training operation.a For historical driving environment status s l Predicted estimated driving behavior, J(s t ,u i (s t |W a )|W c ) is used to represent the evaluation network for the historical driving environment state s t and estimated driving behavior u i (s t |W a ) predicts the network weight W c The driving behavior cost, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u All are comprehensive weights of costs.

[0115] Therefore, the present application can nest multiple iterative optimization operations of the evaluation network during one iterative training operation of the execution network by executing the specific step flow of the above sub-step S251.

[0116] Sub-step S252, based on multiple driving network training samples and the evaluation network that meets the iterative optimization termination condition, the network weight of the execution network is optimized to obtain the target network weight of the execution network in this iterative training operation.

[0117] In this embodiment, the network weight optimization process of the execution network in the i-th iteration training operation is expressed by the following equation:

[0118]

[0119] Among them, γ is used to represent the discount factor of the evaluation network, s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, u i (s t ) is used to represent the execution network for the historical driving environment state s when completing the i-th iteration training operation t The predicted driving behavior, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u are all comprehensive weights of costs, u i (s t+n+1 ) is used to represent the execution network for the historical driving environment state s when completing the i-th iteration training operation t+n+1 Predicted estimated driving behavior, It is used to indicate that the evaluation network meets the iterative optimization termination condition during the i-th iterative training operation for the historical driving environment state s t+n+1 and estimated driving behavior u(s t+n+1 ) predicted driving behavior cost, W a is used to represent the network weight to be optimized of the execution network, and σ(·) is used to represent the tanh activation function.

[0120] Sub-step S253, detecting whether the target network weights meet the conditions for ending iterative training.

[0121] In this embodiment, the execution network uses the same iterative training end condition in different iterative training operations (for example, the corresponding iterative training number is equal to the preset training number, or the evaluation network optimized by each of two adjacent iterative training operations is based on the difference in driving behavior costs estimated by the trained execution network under the same driving environment state (i.e. ) is less than the preset cost threshold).

[0122] After completing an iterative training operation for the execution network, the driving decision device 10 will confirm whether the iterative training operation belongs to the last iterative training operation based on whether the target network weight determined by the iterative training operation meets the iterative training end condition. Wherein, when the target network weight determined by the iterative training operation (i.e., this iterative training operation) does not meet the iterative training end condition, it indicates that this iterative training operation does not belong to the last iterative training operation, and the driving decision device 10 will start to execute the next iterative training operation for the execution network by jumping to sub-step S251 to continue execution. At this time, the initial network weight of the execution network in the next iterative training operation is the target network weight of this iterative training operation.

[0123] When the target network weight determined by the iterative training operation (i.e., this iterative training operation) meets the iterative training end condition, it indicates that this iterative training operation is the last iterative training operation, and the driving decision-making device 10 will execute sub-step S254 accordingly, so as to use the execution network with the target network weight determined by this iterative training operation as the driving behavior strategy network, and at the same time use the evaluation network that meets the iterative optimization termination condition determined during this iterative training operation as the driving cost evaluation network.

[0124] In sub-step S254, the execution network having the target network weight is directly used as the driving behavior strategy network, and the evaluation network that satisfies the iterative optimization termination condition of this iterative training operation is used as the driving cost evaluation network.

[0125] Therefore, the present application can ensure that the finally optimized driving behavior strategy network and driving cost evaluation network can effectively adapt to the changes in the dynamic traffic environment by cyclically executing the above sub-steps S251 to S254, and ensure that the driving behavior prediction results output by the corresponding driving behavior strategy network for any driving environment state have sufficiently high accuracy and reliability. At the same time, the corresponding driving cost evaluation network can also accurately evaluate the driving behavior cost caused by any driving environment state and any driving behavior prediction results in a fixed time period in the future.

[0126] In addition, the present application can ensure that the driving behavior strategy network and the driving cost evaluation network can perform model adaptive learning optimization based on the real-time feedback mechanism of the vehicle driving environment information by executing the above steps S240 to S250 to adapt to the dynamic traffic environment changes and improve the network reliability and network robustness of the driving behavior strategy network and the driving cost evaluation network in complex driving environments.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the various functions provided by the present application are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (for example, a server, a vehicle-mounted terminal, a personal computer, etc.) as the above-mentioned driving decision device 10 to perform all or part of the steps of the method recorded in each embodiment of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0129] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An automatic driving behavior decision-making method, characterized in that: The method comprises: Obtain the actual driving environment state of the target vehicle at the current moment; The driving behavior strategy network and the driving cost evaluation network are called to perform a Monte Carlo tree search based on the actual driving environment state, and when the search termination condition is met, the optimal child node of the root node in the corresponding Monte Carlo tree that meets the driving behavior cost minimization requirement is determined; wherein all nodes in the Monte Carlo tree each represent a driving environment state, the root node represents the actual driving environment state, the initial driving behavior costs of all nodes are obtained by the driving cost evaluation network based on the driving behavior prediction result of the node and the driving environment state evaluation, and the driving behavior prediction result of all nodes is obtained by the driving behavior strategy network based on the driving environment state estimation of the node; the driving behavior strategy network and the driving cost evaluation network are obtained based on adaptive dynamic programming structure training; The driving behavior action between the optimal child node and the root node is used as the optimal driving action of the target vehicle at the current moment.

2. The method according to claim 1, characterized in that The step of calling the driving behavior strategy network and the driving cost evaluation network to perform a Monte Carlo tree search based on the actual driving environment state includes: In each Monte Carlo tree search process, according to the actual driving behavior cost and the actual number of visits of each node in the Monte Carlo tree, the optimal child node is selected layer by layer starting from the root node to determine the optimal node path in this Monte Carlo tree search process, wherein the benchmark driving action of each node in the Monte Carlo tree is the estimated driving behavior result of the node, and the driving behavior action between each non-root node and its parent node on the optimal node path is obtained by adding action noise to the benchmark driving action of the corresponding parent node; Expanding the end node on the optimal node path to a sub-node, and using the estimated driving behavior result of the end node as a benchmark driving action from the end node to the expanded target sub-node; Perform a driving preview according to the driving environment state and driving behavior estimation result of the target subnode, and set the actual number of visits to the target subnode to 1; Based on the initial driving behavior cost of the target child node, the actual number of visits and the actual driving behavior cost of each node on the optimal node path are backtracked and updated.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Acquire multiple driving network training samples, wherein each driving network training sample includes a historical driving environment state, a historical driving action, and a comprehensive attenuated driving cost of a sample vehicle at a corresponding historical moment, and a historical driving environment state of the sample vehicle at a target moment associated with the historical moment; the target moment is separated from the historical moment by a first preset number of control moments, the comprehensive attenuated driving cost is the sum of comprehensive driving cost attenuation values ​​of the sample vehicle at a second preset number of consecutive control moments starting from the historical moment, and the second preset number is obtained by adding one to the first preset number; According to the multiple driving network training samples, an adaptive dynamic programming structure including an execution network and an evaluation network is iteratively trained to obtain the driving behavior strategy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network.

4. The method according to claim 3, characterized in that The step of iteratively training the adaptive dynamic programming structure including the execution network and the evaluation network according to the multiple driving network training samples to obtain the driving behavior strategy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network includes: For each iterative training operation of the execution network, determine the initial network weight of the execution network in this iterative training operation, and iteratively optimize the evaluation network based on the initial network weight and the multiple driving network training samples to obtain an evaluation network that meets the iterative optimization termination condition of this iterative training operation; Based on the multiple driving network training samples and the evaluation network that meets the iterative optimization termination condition, the network weight of the execution network is optimized to obtain the target network weight of the execution network in this iterative training operation; Detecting whether the target network weights meet the iterative training end condition; If it is detected that the target network weight meets the iterative training termination condition, the execution network with the target network weight is directly used as the driving behavior strategy network, and the evaluation network that meets the iterative optimization termination condition of this iterative training operation is used as the driving cost evaluation network, otherwise the next iterative training operation is performed on the execution network, wherein the initial network weight of the execution network in the next iterative training operation is the target network weight of this iterative training operation.

5. The method according to claim 4, characterized in that During each iterative training operation of the execution network, the network iterative optimization process of the evaluation network is expressed by the following equation: Among them, γ is used to represent the discount factor of the evaluation network, s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u are all comprehensive weights of costs, J j+1 (s t ,u t ) is used to represent the evaluation network after the j+1th iteration optimization for the historical driving environment state s t and historical driving actions t Predicted driving behavior cost, u(s t+n+1 ) is used to represent the execution network for the historical driving environment state s t+n+1 Predicted estimated driving behavior, J j (s t+n+1 ,u(s t+n+1 )) is used to represent the evaluation network after the jth iteration optimization for the historical driving environment state s t+n+1 and estimated driving behavior u(s t+n+1 ) predicted driving behavior cost.

6. The method according to claim 4, characterized in that The step of iteratively optimizing the evaluation network based on the initial network weights and the multiple driving network training samples to obtain an evaluation network that satisfies the iterative optimization termination condition of this iterative training operation includes: For each iterative optimization operation of the evaluation network during the iterative training operation of the execution network, determining a reference network weight of the evaluation network in the iterative optimization operation; Based on the reference network weight, the initial network weight and the plurality of driving network training samples, the network weight of the evaluation network is updated to obtain the expected network weight of the evaluation network in this iterative optimization operation; Detecting whether the expected network weights meet the iterative optimization termination condition; If it is detected that the expected network weight meets the iterative optimization termination condition, the next iterative optimization operation is stopped and the evaluation network with the expected network weight is output. Otherwise, the next iterative optimization operation is performed on the evaluation network, wherein the reference network weight of the evaluation network in the next iterative optimization operation is the expected network weight of this iterative optimization operation.

7. The method according to claim 6, characterized in that The network weight update process of the evaluation network in the j+1th iterative optimization operation during the i-th iterative training operation of the execution network is expressed by the following equation: Among them, W c,j+1 is used to represent the expected network weight of the evaluation network in the j+1th iteration optimization operation during the i-th iteration training operation of the execution network, W c,j is used to represent the expected network weight of the evaluation network during the jth iteration optimization operation of the execution network during the i-th iteration training operation, α c It is used to represent the learning rate of the evaluation network, W c is used to represent the network weight of the evaluation network, γ is used to represent the discount factor of the evaluation network, and s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, u i (s l |W a ) is used to indicate that the execution network uses the initial network weight W during the i-th iteration training operation. a For historical driving environment status s l Predicted estimated driving behavior, J(s t ,u i (s t |W a )|W c ) is used to represent the evaluation network for the historical driving environment state s t and estimated driving behavior u i (s t |W a ) predicts the network weight W c The driving behavior cost, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u All are comprehensive weights of costs.

8. The method according to claim 4, characterized in that The network weight optimization process of the execution network at the i-th iteration training operation is expressed by the following equation: Among them, γ is used to represent the discount factor of the evaluation network, s t It is used to represent the historical driving environment state at the historical time t, u t is used to represent the historical driving action at the historical time t, s t+n+1 is used to represent the historical driving environment state at the target moment associated with the historical moment t, n is used to represent the first preset number, u i (s t ) is used to represent the execution network for the historical driving environment state s when completing the i-th iteration training operation t The predicted driving behavior, U(s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t Matching comprehensive driving cost, J s (s t ) is used to represent the historical driving environment state s t The security cost of J c (s t ) is used to represent the historical driving environment state s t The comfort cost of J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions t The quadratic cost of matching, ω s ,ω c ,ω p and ω u are all comprehensive weights of costs, u i (s t+n+1 ) is used to represent the execution network for the historical driving environment state s when completing the i-th iteration training operation t+n+1 Predicted driving behavior, It is used to indicate that the evaluation network meets the iterative optimization termination condition for the historical driving environment state s during the i-th iterative training operation. t+n+1 and estimated driving behavior u(s t+n+1 ) predicted driving behavior cost, W a is used to represent the network weight to be optimized of the execution network, and σ(·) is used to represent the tanh activation function.

9. A driving decision-making device, characterized in that: It includes a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the automatic driving behavior decision method described in any one of claims 1-8.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer device, the automatic driving behavior decision method described in any one of claims 1 to 8 is implemented.

Images