Autonomous driving behavior decision-making methods, driving decision-making devices and readable storage media

Abstract

This application provides an autonomous driving behavior decision-making method, driving decision-making device, and readable storage medium, relating to the field of autonomous driving technology. This application utilizes a driving behavior policy network and a driving cost evaluation network trained using adaptive dynamic programming techniques. Based on the real-time driving environment state of the target vehicle, a Monte Carlo tree search is performed. This integrates adaptive dynamic programming into each decision-making stage of the Monte Carlo tree search algorithm, guiding it to focus on effective searches for high-value decisions. This reduces the workload, computational resource consumption, and time consumption of ineffective searches, improving the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm. Consequently, it quickly and reliably determines the optimal driving action of the target vehicle in a complex driving environment at the current moment, enabling fast, robust, and interpretable driving behavior decision-making in complex driving environments that consider other traffic participants.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and more specifically, to an autonomous driving behavior decision-making method, a driving decision-making device, and a readable storage medium. Background Technology

[0002] With the development of automotive intelligence and driver assistance technologies, autonomous driving, as an advanced stage of driver assistance, has become a crucial means to solve people's future travel problems and has become a new focus and hot topic of research globally. Especially in recent years, autonomous driving technology has flourished, achieving a milestone in the history of human transportation development.

[0003] Currently, the architecture of autonomous driving technology is mainly divided into three modules: environmental perception, driving decision planning, and control execution. The driving decision planning module, based on the external environmental conditions and the vehicle's internal state perceived by the environmental perception module, determines the optimal driving behavior for the autonomous vehicle in dynamic and complex environments. This decision is then executed by the control execution module to meet preset requirements while ensuring driving safety. Therefore, the performance of driving decision planning directly affects the final autonomous driving effect. It is worth noting that existing autonomous driving behavior decision-making methods struggle to simultaneously balance decision efficiency, environmental adaptability, and decision interpretability in complex and dynamic environments. They cannot quickly provide autonomous vehicles with driving behavior decision results that are adapted to the vehicle's driving environment and have strong interpretability. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an autonomous driving behavior decision-making method, driving decision-making device, and readable storage medium, which can integrate adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm in the Monte Carlo tree search process based on vehicle driving environment conditions, so as to effectively reduce the amount of invalid search work, improve the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm, and thus achieve fast, robust and interpretable driving behavior decision-making function in complex driving environments that take into account other traffic participants.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0006] In a first aspect, this application provides an autonomous driving behavior decision-making method, the method comprising:

[0007] Obtain the actual driving environment status of the target vehicle at the current moment;

[0008] The driving behavior policy network and the driving cost evaluation network are invoked to perform a Monte Carlo tree search based on the actual driving environment state. When the search termination condition is met, the optimal child node of the root node in the corresponding Monte Carlo tree that minimizes the driving behavior cost is determined. Each node in the Monte Carlo tree represents a driving environment state, the root node represents the actual driving environment state, and the initial driving behavior cost of each node is obtained by the driving cost evaluation network based on the node's driving behavior prediction result and the driving environment state evaluation. The driving behavior prediction result of each node is obtained by the driving behavior policy network based on the node's driving environment state prediction. The driving behavior policy network and the driving cost evaluation network are trained based on an adaptive dynamic programming structure.

[0009] The driving behavior between the optimal child node and the root node is taken as the optimal driving action of the target vehicle at the current moment.

[0010] In an optional implementation, the step of invoking the driving behavior policy network and the driving cost evaluation network to perform Monte Carlo tree search based on the actual driving environment state includes:

[0011] In each Monte Carlo tree search, based on the actual driving behavior cost and actual number of visits of each node in the Monte Carlo tree, the optimal child node is selected layer by layer starting from the root node to determine the optimal node path in this Monte Carlo tree search process. The baseline driving action of each node in the Monte Carlo tree is the driving behavior prediction result of that node. The driving behavior action between each non-root node and its parent node on the optimal node path is obtained by adding action noise to the baseline driving action of the corresponding parent node.

[0012] The terminal node on the optimal node path is expanded into a sub-node, and the driving behavior prediction result of the terminal node is used as the baseline driving action from the terminal node to the expanded target sub-node.

[0013] A driving simulation is performed based on the driving environment status and driving behavior prediction results of the target sub-node, and the actual number of visits to the target sub-node is set to 1.

[0014] Based on the initial driving behavior cost of the target sub-node, the actual number of visits and actual driving behavior cost of each node on the optimal node path are updated backtracking.

[0015] In an optional implementation, the method further includes:

[0016] Multiple driving network training samples are acquired, wherein each driving network training sample includes the historical driving environment state, historical driving actions, and comprehensive attenuation driving cost of the sample vehicle at the corresponding historical moment, as well as the historical driving environment state of the sample vehicle at a target moment associated with the historical moment; the target moment is separated from the historical moment by a first preset number of control moments, and the comprehensive attenuation driving cost is the sum of the comprehensive driving cost attenuation values ​​of the sample vehicle for each of the second preset number of consecutive control moments starting from the historical moment, the second preset number being obtained by adding one to the first preset number;

[0017] Based on the multiple driving network training samples, an adaptive dynamic programming structure including an execution network and an evaluation network is iteratively trained to obtain the driving behavior policy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network.

[0018] In an optional implementation, the step of iteratively training an adaptive dynamic programming structure including an execution network and an evaluation network based on the plurality of driving network training samples to obtain the driving behavior policy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network includes:

[0019] For each iteration training operation of the execution network, the initial network weights of the execution network in this iteration training operation are determined, and the evaluation network is iteratively optimized based on the initial network weights and the multiple driving network training samples to obtain the evaluation network that satisfies the iteration optimization termination condition for this iteration training operation.

[0020] Based on the multiple driving network training samples and the evaluation network that meets the iteration optimization termination condition, the execution network is optimized to obtain the target network weights of the execution network in this iteration training operation.

[0021] Detect whether the target network weights meet the iterative training termination condition;

[0022] If the target network weights are detected to meet the iteration training termination condition, the execution network with the target network weights is directly used as the driving behavior policy network, and the evaluation network that meets the iteration optimization termination condition in this iteration training operation is used as the driving cost evaluation network. Otherwise, the next iteration training operation is performed on the execution network, wherein the initial network weights of the execution network in the next iteration training operation are the target network weights of this iteration training operation.

[0023] In an optional implementation, during each iteration of the training operation of the execution network, the network iterative optimization process of the evaluation network is expressed by the following equation:

[0024]

[0025] Where γ represents the discount factor of the evaluation network, s t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 U(s) is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number. t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The cost of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are cost-comprehensive weights, J j+1 (s t ,u t ) is used to represent the evaluation network after the (j+1)th iteration optimization for the historical driving environment state s. t and historical driving actions u t Predicted driving behavior cost, u(s) t+n+1 ) is used to represent the execution network in response to the historical driving environment state s. t+n+1 Predicted driving behavior, J j (s t+n+1 ,u(s t+n+1 )) is used to represent the evaluation network after the j-th iteration optimization for the historical driving environment state s. t+n+1 and predicted driving behavior u(s) t+n+1 The predicted cost of driving behavior.

[0026] In an optional implementation, the step of iteratively optimizing the evaluation network based on the initial network weights and the plurality of driving network training samples to obtain an evaluation network that satisfies the iteration optimization termination condition for this iteration training operation includes:

[0027] For each iteration optimization operation of the evaluation network during the current iteration training operation of the execution network, determine the reference network weights of the evaluation network in the current iteration optimization operation;

[0028] The evaluation network weights are updated based on the reference network weights, the initial network weights, and the multiple driving network training samples to obtain the expected network weights of the evaluation network in this iteration optimization operation.

[0029] Detect whether the desired network weights satisfy the iterative optimization termination condition;

[0030] If the expected network weights are detected to meet the iteration optimization termination condition, the next iteration optimization operation is stopped, and the evaluation network with the expected network weights is output. Otherwise, the next iteration optimization operation is performed on the evaluation network, wherein the reference network weights of the evaluation network in the next iteration optimization operation are the expected network weights of the current iteration optimization operation.

[0031] In an optional implementation, the network weight update process of the evaluation network in the (j+1)th iteration optimization operation during the i-th iteration training operation of the execution network is expressed by the following equation:

[0032]

[0033] Among them, W c,j+1 W is used to represent the expected network weights of the evaluation network in the (j+1)th iteration optimization operation during the i-th iteration training operation of the execution network. c,j α is used to represent the expected network weights of the evaluation network in the j-th iteration optimization operation during the i-th iteration training operation of the execution network. c W is used to represent the learning rate of the evaluation network. c The network weights of the evaluation network are used to represent the network weights, γ is used to represent the discount factor of the evaluation network, and s is used to represent the network weights of the evaluation network. t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 This is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number, and u... i (s l |W a) is used to indicate that the execution network utilizes the initial network weights W during the i-th iteration of training. a Based on historical driving environment conditions s l Predicted driving behavior, J(s) t ,u i (s t |W a )|W c ) is used to represent the evaluation network for historical driving environment state s. t and predict driving behavior i (s t |W a The predicted network weight W c The cost of driving behavior, U(s) t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The cost of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are comprehensive cost weights.

[0034] In an optional implementation, the network weight optimization process for the i-th iteration of the training operation is expressed by the following equation:

[0035]

[0036] Where γ represents the discount factor of the evaluation network, s t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1This is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number, and u... i (s t ) is used to represent the execution network's response to the historical driving environment state s when completing the i-th iteration of training. t Predicted driving behavior, U(s) t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The cost of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are cost-comprehensive weights, u i (s t+n+1 ) is used to represent the execution network's response to the historical driving environment state s when completing the i-th iteration of training. t+n+1 Predicted driving behavior This is used to indicate that the evaluation network satisfies the iteration optimization termination condition during the i-th iteration training operation for the historical driving environment state s. t+n+1 and predicted driving behavior u(s) t+n+1 The predicted cost of driving behavior, W a The weights to be optimized in the execution network are represented by σ(·), and the tanh activation function is represented by σ(·).

[0037] Secondly, this application provides a driving decision-making device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the autonomous driving behavior decision-making method described in any of the foregoing embodiments.

[0038] Thirdly, this application provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a computer device, it implements the autonomous driving behavior decision-making method described in any of the foregoing embodiments.

[0039] In this case, the beneficial effects of the embodiments of this application may include the following:

[0040] This application utilizes a driving behavior strategy network and a driving cost evaluation network trained using adaptive dynamic programming techniques. Based on the actual driving environment state of the target vehicle at the current moment, a Monte Carlo tree search is performed. The initial driving behavior cost of each node in the Monte Carlo tree is obtained by the driving cost evaluation network based on the node's driving behavior prediction and driving environment state evaluation. Similarly, the driving behavior prediction result of each node is obtained by the driving behavior strategy network based on the node's driving environment state prediction. This integrates adaptive dynamic programming into each decision-making stage of the Monte Carlo tree search algorithm, guiding it to focus on effective searches for high-value decisions. This reduces the workload, computational resource consumption, and time consumption of ineffective searches, improving the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm. Through the organic combination of the highly interpretable Monte Carlo tree search algorithm and the environmentally adaptable adaptive dynamic programming technique, the optimal driving action of the target vehicle in a complex driving environment at the current moment can be quickly and reliably determined. This enables a fast, robust, and highly interpretable driving behavior decision-making function in complex driving environments that consider other traffic participants.

[0041] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the composition of the driving decision-making device provided in the embodiments of this application;

[0044] Figure 2 One of the flowcharts of the autonomous driving behavior decision-making method provided in the embodiments of this application;

[0045] Figure 3 for Figure 2A flowchart illustrating the sub-steps included in step S220;

[0046] Figure 4 A second schematic flowchart illustrating the autonomous driving behavior decision-making method provided in this application embodiment;

[0047] Figure 5 This is a schematic diagram of the adaptive dynamic programming structure provided in the embodiments of this application;

[0048] Figure 6 for Figure 4 The flowchart of the sub-steps included in step S250 is shown below.

[0049] Icons: 10-Driving decision-making equipment; 11-Memory; 12-Processor; 13-Communication unit. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0053] In the description of this application, it should be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are used only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the equipment or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0054] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0055] In the description of this application, it should be understood that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0056] Through diligent research, the applicant discovered that existing autonomous driving behavior decision-making methods primarily employ three approaches: pre-defined rules, search algorithms, and reinforcement learning. Among these, rule-based methods fail to effectively predict the behavior of other road users in complex driving environments. While search algorithms can find optimal decision paths through global search and offer some interpretability due to the observable nature of the search process, they often suffer from high computational complexity and insufficient real-time performance, particularly in large-scale state spaces, hindering rapid and effective decision-making. Reinforcement learning-based methods, despite possessing some environmental adaptability and the ability to make rapid decisions, are highly dependent on the quality and scale of training data and struggle to provide reasonable explanations for their decisions, lacking strong interpretability.

[0057] In this context, to address the aforementioned issues, embodiments of this application provide an autonomous driving behavior decision-making method, a driving decision-making device, and a readable storage medium, which enable fast, robust, and interpretable driving behavior decision-making functions in complex driving environments that take into account other traffic participants.

[0058] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0059] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the driving decision device 10 provided in this application embodiment. In this application embodiment, the driving decision device 10 can communicate with an autonomous vehicle equipped with an autonomous driving system to obtain real-time driving environment information of the autonomous vehicle during actual driving, and quickly and reliably decide on the optimal driving action of the autonomous vehicle in complex driving environments based on the obtained driving environment information. This facilitates the autonomous driving system of the autonomous vehicle to control the vehicle operation according to the determined optimal driving action, thereby meeting preset requirements while ensuring driving safety. In this process, the driving environment information is used to determine the relative positional relationship, relative attitude relationship, relative speed change relationship, and relative acceleration change relationship between the autonomous vehicle and other static targets (e.g., lane lines and road boundaries, green belts, etc.) and other traffic participants (e.g., social vehicles, pedestrians, etc.) in the corresponding driving environment. The driving behavior decision (optimal driving action) given by the driving decision device 10 may involve information such as the motion acceleration, steering angle, and braking degree of the corresponding autonomous vehicle to directly affect the vehicle's motion state. The driving decision device 10 can be integrated with the autonomous vehicle that requires driving behavior decision-making. In this case, the driving decision device 10 is the on-board equipment of the autonomous vehicle and is part of the autonomous driving system. The driving decision device 10 can also be a computer device independent of the autonomous vehicle. The computer device may be, but is not limited to, a server, an external computer, etc.

[0060] In this embodiment, the driving decision-making device 10 may include a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0061] In this embodiment, the memory 11 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 11 is used to store computer programs, and the processor 12 can execute the computer programs accordingly after receiving execution instructions.

[0062] In this embodiment, the processor 12 can be an integrated circuit chip with signal processing capabilities. The processor 12 can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment.

[0063] In this embodiment of the application, the communication unit 13 is used to establish a communication connection between the driving decision device 10 and other electronic devices through a network, and to send and receive data through the network, wherein the network includes a wired communication network and a wireless communication network.

[0064] In this embodiment, the driving decision-making device 10 can pre-store specific computer programs related to autonomous driving behavior decision-making functions in the memory 11. By driving the processor 12 to execute the specific computer programs stored in the memory 11, adaptive dynamic programming technology is integrated into each decision stage of the Monte Carlo tree search algorithm based on real-time vehicle driving environment conditions. This guides the Monte Carlo tree search algorithm to focus on effective searches for high-value decisions, reducing the workload, computational resource consumption, and time consumption of ineffective searches, and improving the decision efficiency and accuracy of the Monte Carlo tree search algorithm. Thus, through the organic combination of the Monte Carlo tree search algorithm with strong decision interpretability (i.e., all decision stages of the Monte Carlo tree search algorithm, such as "selection," "expansion," "pre-exercise," and "backtracking," are observable) and the adaptive dynamic programming technology with strong environmental adaptability (i.e., the adaptive dynamic programming technology can consider the real-time feedback mechanism of vehicle driving environment information to perform model adaptive learning optimization to adapt to dynamic traffic environment changes), a fast, robust, and highly interpretable driving behavior decision-making function is achieved in complex driving environments that consider other traffic participants.

[0065] Understandable Figure 1 The block diagram shown is only a schematic diagram of one composition of the driving decision device 10. The driving decision device 10 may also include components such as... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0066] In this application, to ensure that the driving decision-making device 10 can achieve fast, robust, and interpretable driving behavior decision-making functions in complex driving environments that take into account other traffic participants, this application embodiment achieves the aforementioned objective by providing an autonomous driving behavior decision-making method. The autonomous driving behavior decision-making method provided in this application will be described in detail below.

[0067] Please refer to Figure 2 , Figure 2 This is one of the flowcharts illustrating the autonomous driving behavior decision-making method provided in this application embodiment. In this application embodiment, the autonomous driving behavior decision-making method may include steps S210 to S230.

[0068] Step S210: Obtain the actual driving environment status of the target vehicle at the current moment.

[0069] In this embodiment, the target vehicle is an autonomous driving vehicle that is communicatively connected to the driving decision device 10. The actual driving environment state records information such as the relative positional relationship and relative attitude relationship between the target vehicle and other static targets and other traffic participants at the current moment.

[0070] Step S220: Invoke the driving behavior policy network and the driving cost evaluation network to perform Monte Carlo tree search based on the actual driving environment state, and determine the optimal child node of the root node in the corresponding Monte Carlo tree that meets the requirement of minimizing driving behavior cost when the search termination condition is met.

[0071] In this embodiment, the driving behavior strategy network and the driving cost evaluation network belong to the same adaptive dynamic programming structure. The driving behavior strategy network is used to predict driving actions based on the driving environment state at any control time, and obtain the predicted driving behavior (i.e., the driving behavior prediction result) corresponding to the driving environment state. The driving cost evaluation network is used to predict the driving behavior cost based on the driving environment state and the predicted driving behavior at any control time, and obtain the total driving behavior cost generated by the vehicle in the future period when it runs according to the corresponding predicted driving behavior in the driving environment state (obtained by accumulating the comprehensive driving costs generated by all control times in the future period, wherein the total number of control times involved in the future period is fixed). During the training and optimization of the network models of the driving behavior strategy network and the driving cost evaluation network, a real-time feedback mechanism of vehicle driving environment information can be considered. With the goal of "minimizing the driving behavior cost predicted by the driving cost evaluation network", the adaptive dynamic programming structure to which the driving behavior strategy network and the driving cost evaluation network belong is trained and learned. This ensures that the driving behavior strategy network can adapt to various complex traffic environments. At the same time, the driving behavior prediction results output by the driving behavior strategy network for any driving environment state have sufficiently high accuracy and reliability, and can efficiently reach the destination while ensuring vehicle safety. The driving cost evaluation network can also accurately evaluate the driving behavior cost caused by any driving environment state and any driving behavior prediction result in a fixed time period in the future.

[0072] In this process, the driving cost evaluation network can express the driving behavior cost it evaluates using the following functional expression:

[0073]

[0074] Among them, s l Used to represent the driving environment state at control time l, u l J(s) is used to represent the vehicle driving action (or predicted driving behavior) at control time l. t0,u t0 ) is used to represent the driving environment state s at control time t0. t0 and vehicle driving actions u t0 The cost of driving behavior incurred within a fixed future time period (which includes a total of control moments of T+1), U(s) l ,u l ) is used to represent the driving environment state s l and vehicle driving actions u l Matching overall driving costs, J s (s l ) is used to represent the driving environment state s l The security cost, J c (s l ) is used to represent the driving environment state s l The cost of comfort, J p (s l ) is used to represent the driving environment state s l The passability cost, J u (s l ,u l ) is used to represent the driving environment state s l and vehicle driving actions u l The cost of matching quadratic forms, ω s ω c ω p and ω u All are comprehensive cost weights.

[0075] The safety cost is inversely correlated with the distance from the vehicle to surrounding objects (including static targets and traffic participants). For example, when the distance from the vehicle to a certain object exceeds a certain distance threshold, the safety cost borne by that object alone is zero; when the distance is greater than 0 but less than or equal to the distance threshold, the safety cost borne by that object alone can be represented by the reciprocal of the distance; and when the distance is 0, the safety cost borne by that object alone can be represented as infinity. The comfort cost is positively correlated with the vehicle's rate of acceleration change, representing the smoothness of vehicle movement; the greater the rate of acceleration change, the greater the corresponding comfort cost. The accessibility cost is related to the distance from the vehicle to the destination, representing the goal orientation of the vehicle's driving actions. The quadratic cost can be expressed as... Where Q and R are both custom weights. In one embodiment of this example, if the driving environment state of a vehicle indicates that there are multiple objects around the vehicle, then the safety cost corresponding to the driving environment state is obtained by adding the safety costs caused to the vehicle by each of the multiple objects.

[0076] In this embodiment, after obtaining the actual driving environment state of the target vehicle at the current moment, the driving decision device 10 calls the trained driving behavior strategy network and the driving cost evaluation network. During the Monte Carlo tree search process based on the actual driving environment state, the output value of the driving behavior strategy network (i.e., the driving behavior prediction result output for the driving environment state represented by each node in the Monte Carlo tree) is used as guidance information. At the same time, the output value of the driving cost evaluation network (i.e., the driving behavior cost output for the driving environment state and driving behavior prediction result of each node in the Monte Carlo tree) is used to characterize the value function result in the Monte Carlo tree algorithm. This realizes the organic combination between adaptive dynamic programming technology and the various decision stages of the Monte Carlo tree search algorithm, so that the corresponding Monte Carlo tree search algorithm focuses on the effective search for high-value decisions, reduces the workload, computational resource consumption burden and time consumption of invalid searches, and improves the decision efficiency and decision accuracy of the Monte Carlo tree search algorithm.

[0077] Therefore, in the Monte Carlo tree search process based on the actual driving environment state, each node in the constructed Monte Carlo tree represents a driving environment state, and the root node in the Monte Carlo tree represents the actual driving environment state. The initial driving behavior cost of each node in the Monte Carlo tree at the first visit (i.e., the actual number of visits is 1) can be directly obtained by the driving cost evaluation network based on the driving behavior prediction result of the node and the driving environment state evaluation. The driving behavior prediction result of each node in the Monte Carlo tree is obtained by the driving behavior policy network based on the driving environment state of the node.

[0078] Simultaneously, during the Monte Carlo tree search process, the driving decision-making device 10, upon completing a Monte Carlo tree search operation, confirms whether the search operation is the final Monte Carlo tree search operation by detecting whether the search termination condition is met (e.g., the total number of searches equals a preset number, or the total search time reaches a preset time). If the search termination condition is not met, it indicates that the search operation is not the final Monte Carlo tree search operation. In this case, the driving decision-making device 10 will continue to execute the next Monte Carlo tree search operation based on the Monte Carlo tree obtained from the previous search, until the search termination condition is finally met.

[0079] When the Monte Carlo tree search operation meets the search termination condition, it indicates that the Monte Carlo tree search operation is the last Monte Carlo tree search operation. The driving decision device 10 will select child nodes for the root node of the target Monte Carlo tree obtained after the Monte Carlo tree search operation is completed, with the aim of "ensuring the minimization of the corresponding driving behavior cost". The optimal child node of the root node (that is, the child node of the root node with the minimum actual driving behavior cost in the target Monte Carlo tree) is obtained. At this time, the driving environment state represented by the optimal child node of the root node is the ideal driving environment state that the target vehicle expects to obtain in the next control moment.

[0080] Alternatively, please refer to Figure 3 , Figure 3 yes Figure 2 The flowchart of step S220 is shown below. In this embodiment, the step "calling the driving behavior strategy network and the driving cost evaluation network to perform Monte Carlo tree search based on the actual driving environment state" in step S220 may include sub-steps S221 to S224, so as to integrate adaptive dynamic programming technology into each decision stage of the Monte Carlo tree search algorithm to perform effective search for high-value decisions, reduce the workload, computational resource consumption burden and time consumption of invalid search, and improve the decision efficiency and decision accuracy of the Monte Carlo tree search algorithm.

[0081] Sub-step S221: In each Monte Carlo tree search process, based on the actual driving behavior cost and actual number of visits of each node in the Monte Carlo tree, the optimal child node is selected layer by layer starting from the root node to determine the optimal node path in this Monte Carlo tree search process.

[0082] In this embodiment, in the Monte Carlo tree constructed based on the actual driving environment state, the actual driving behavior cost of each node in the Monte Carlo tree at the first visit (i.e., the actual number of visits is 1) is the initial driving behavior cost of the corresponding node. For the "selection" stage in each Monte Carlo tree search process, the driving decision device 10 starts from the root node in the current Monte Carlo tree and, layer by layer, selects the optimal child node based on the actual driving behavior cost and actual number of visits of each existing node, guided by the baseline driving action of each node (i.e., the driving behavior prediction result of the corresponding node), and with the selection objective of "minimizing the sum of costs among the actual driving behavior costs of all nodes on the corresponding node path". The UCB (Upper Confidence Bound) algorithm is used to select the optimal child node to form the optimal node path in this Monte Carlo tree search process. In this context, adjacent nodes on the optimal node path satisfy a parent-child relationship, and the driving behavior action between each non-root node on the optimal node path and its parent node is obtained by adding a set action noise (e.g., Gaussian noise) to the baseline driving action of the corresponding parent node. This balances the strategy selection relationship between "utilization" and "exploration" by superimposing action noise on the baseline driving action, while enhancing the path diversity of the optimal node path.

[0083] Sub-step S222: Expand the end node on the optimal node path into a sub-node, and use the driving behavior prediction result of the end node as the baseline driving action from the end node to the expanded target sub-node.

[0084] In this embodiment, for the "expansion" phase of each Monte Carlo tree search process, the driving decision device 10 can use the estimated driving behavior of the last node on the optimal node path as the benchmark driving action between the last node and the newly expanded node, and predict the future environment state based on the driving environment state of the last node and the benchmark driving action, thereby expanding the target child node on the current Monte Carlo tree to provide diversified path selection for the subsequent "preview" and "backtrack" phases, thereby improving the effectiveness of the search, reducing the search workload, computational resource consumption burden and time consumption caused by invalid searches, and improving the decision efficiency and decision accuracy of the Monte Carlo tree search algorithm.

[0085] Sub-step S223: Perform a driving simulation based on the driving environment status and driving behavior prediction results of the target sub-node, and set the actual number of visits to the target sub-node to 1.

[0086] In this embodiment, for the "pre-rehearsal" phase of each Monte Carlo tree search process, the driving decision device 10 starts from the target child node and uses the driving behavior prediction result and driving environment state of the target child node to perform driving pre-rehearsal simulation until the simulation termination condition is met. At this time, the actual access count of the target child node can be set to 1, and the driving cost evaluation network is called to evaluate the driving behavior cost that matches the driving behavior prediction result and driving environment state of the target child node, which is used as the initial driving behavior cost of the target child node.

[0087] Sub-step S224: Based on the initial driving behavior cost of the target sub-node, backtrack and update the actual number of visits and actual driving behavior cost of each node on the optimal node path.

[0088] In this embodiment, during the "backtracking" phase of each Monte Carlo tree search, the driving decision-making device 10, based on the initial driving behavior cost of the newly expanded target child node, reverses the node arrangement order of the optimal node path and increments the actual visit count of each node on the optimal node path by one, thus obtaining the actual visit count of each node on the optimal node path after the completion of this Monte Carlo tree search. Simultaneously, for each node on the optimal node path, the formula "q=q′+(q L The `-q′) / N` parameter updates the actual driving cost of each node, ensuring that subsequent search operations can more effectively select low-cost (high-value) node paths, making the Monte Carlo tree search algorithm more focused on effective searches for high-value decisions. Here, `q` represents the actual driving cost of the corresponding node after the update, `q′` represents the actual driving cost of the corresponding node before the update, and `N` represents the actual number of visits to the corresponding node after the update. L Used to represent the initial driving behavior cost of the newly expanded target child node.

[0089] Therefore, by executing the above sub-steps S221 to S224, this application can integrate adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm to perform effective searches for high-value decisions, reduce the workload, computational resource consumption burden and time consumption of invalid searches, and improve the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm.

[0090] Step S230: The driving behavior action between the optimal child node and the root node is taken as the optimal driving action of the target vehicle at the current moment.

[0091] In this embodiment, after determining the optimal child node of the root node from the target Monte Carlo tree that meets the search termination condition, the driving decision device 10 can extract the driving behavior action between the root node and the corresponding optimal child node from the target Monte Carlo tree as the optimal driving action of the target vehicle in the current complex traffic environment at the current moment, thereby realizing a fast, robust and interpretable driving behavior decision function.

[0092] Therefore, by executing steps S210 to S230 above, this application integrates adaptive dynamic programming technology into each decision-making stage of the Monte Carlo tree search algorithm during the Monte Carlo tree search process based on real-time vehicle driving environment conditions. This guides the Monte Carlo tree search algorithm to focus on effective searches for high-value decisions, reducing the workload, computational resource consumption, and time consumption of ineffective searches, and improving the decision-making efficiency and accuracy of the Monte Carlo tree search algorithm. Thus, through the organic combination of the Monte Carlo tree search algorithm with strong decision interpretability and the adaptive dynamic programming technology with strong environmental adaptability, a fast, robust, and highly interpretable driving behavior decision-making function can be achieved in complex driving environments that consider other traffic participants.

[0093] Alternatively, please refer to Figure 4 , Figure 4 This is a second schematic flowchart of the autonomous driving behavior decision-making method provided in this application embodiment. In this application embodiment, [the method is]... Figure 2 Compared to the autonomous driving behavior decision-making methods shown, Figure 4 The autonomous driving behavior decision-making method shown may further include steps S240 to S250 to ensure that the driving behavior strategy network and the driving cost evaluation network can perform model adaptive learning optimization based on the real-time feedback mechanism of vehicle driving environment information to adapt to dynamic traffic environment changes and improve the network reliability and network robustness of the driving behavior strategy network and the driving cost evaluation network in complex driving environments.

[0094] Step S240: Obtain multiple driving network training samples, wherein each driving network training sample includes the historical driving environment state, historical driving actions and comprehensive attenuation driving cost of the sample vehicle at the corresponding historical time, as well as the historical driving environment state of the sample vehicle at the target time associated with the historical time.

[0095] In this embodiment, reference can be made to Figure 5 The adaptive dynamic programming structure adopted in this application includes an execution network, a dynamic system, and two evaluation networks. The execution network corresponds to the driving behavior policy network, and the evaluation networks correspond to the driving cost assessment network. The dynamic system is used to determine the input state s of the execution network at control time t.t and output action u t Deduce the output action u at target control time t+n+1. t+n+1 So that one of the two evaluation networks is solely responsible for evaluating the input state s. t and output action u t In the driving behavior cost assessment task, another evaluation network is solely responsible for evaluating the input state s. t+n+1 and output action u t+n+1 The driving behavior cost assessment task is then performed, and the driving cost is further reduced by an externally introduced comprehensive factor from control time t to control time t+n. Network optimization is performed to improve the training effect of the adaptive dynamic programming structure model. Here, n represents a first preset number (its value can be 0, 2, 3, or 5), and γ represents the discount factor of the evaluation network.

[0096] Based on this, each driving network training sample obtained by the driving decision device 10 for the driving behavior strategy network and the driving cost evaluation network must include the historical driving environment state, historical driving actions, and comprehensive attenuation driving cost of the corresponding sample vehicle at the corresponding historical time, as well as the historical driving environment state of the sample vehicle at the target time associated with that historical time. The sample vehicle can be the aforementioned target vehicle or other autonomous driving vehicles; different driving network training samples correspond to different historical time points; the target time of a single driving network training sample is spaced apart from the corresponding historical time by a first preset number of control times (e.g., for historical time t, the corresponding target time is t+n+1); the comprehensive attenuation driving cost included in a single driving network training sample is the comprehensive driving cost attenuation value of the sample vehicle for each of the second preset number of consecutive control times (including that historical time) starting from the corresponding historical time (e.g., for historical time t, the comprehensive driving cost attenuation value of control time l after that historical time is γ). l-t U(s l ,u l The sum of (e.g., for historical time t, the corresponding comprehensive decay driving cost is) The second preset number is obtained by adding one to the first preset number.

[0097] It is understood that the driving decision device 10 can construct a data pool with a fixed capacity for the target vehicle. The driving decision device 10 collects the real-time driving environment status and real-time driving actions of the target vehicle to construct driving network training samples, and then adds the constructed driving network training samples to the data pool. At the same time, it deletes the oldest driving network training samples stored in the data pool, so that the driving network training samples stored in the data pool can effectively reflect the dynamic traffic environment changes of the target vehicle, so as to batch extract samples (for the driving behavior policy network and the driving cost evaluation network) in the data pool for model adaptive learning optimization.

[0098] Step S250: Based on multiple driving network training samples, iteratively train the adaptive dynamic programming structure, which includes an execution network and an evaluation network, to obtain the driving behavior policy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network.

[0099] In this embodiment, for the adaptive dynamic programming structure, the network function of the execution network can be described as follows: Where σ(·) represents the hidden layer activation function (e.g., tanh activation function) of the execution network, W a The network weights used to represent the hidden and output layers of the execution network, and to account for the saturation characteristics of the executor output during actual task execution, an activation function (e.g., tanh activation function) can also be used for model training in the output layer of the execution network; the network function of the evaluation network can be described as follows: Where σ(·) represents the hidden layer activation function (e.g., tanh activation function) of the evaluation network, W c The network weights ξ represent the network weights between the hidden and output layers of the evaluation network, and ξ represents the learning coefficients of the evaluation network. To ensure that the network function and the driving behavior cost function expression remain positive definite, a ReLU activation function can be used for model training in the output layer of the evaluation network. Furthermore, each iteration of the execution network's training operation contains nested iterations of the evaluation network's optimization operations.

[0100] Specifically, please refer to Figure 6 , Figure 6 yes Figure 4The flowchart of step S250 is shown below. In this embodiment, step S250 may include sub-steps S251 to S254 to ensure that the finally optimized driving behavior strategy network and driving cost evaluation network can effectively adapt to changes in the dynamic traffic environment, and to ensure that the driving behavior prediction results output by the corresponding driving behavior strategy network for any driving environment state have sufficiently high accuracy and reliability. At the same time, the corresponding driving cost evaluation network can also accurately evaluate the driving behavior cost caused by any driving environment state and any driving behavior prediction result in a fixed period of time in the future.

[0101] Sub-step S251: For each iteration training operation of the execution network, determine the initial network weights of the execution network in this iteration training operation, and iteratively optimize the evaluation network based on the initial network weights and multiple driving network training samples to obtain the evaluation network that satisfies the iteration optimization termination condition for this iteration training operation.

[0102] In this embodiment, the initial network weights of the execution network in the first iteration of training can be formed using a random seed algorithm; the initial network weights of the execution network in subsequent iterations of training are the target network weights finally determined in the previous iteration of training. During each iteration of training of the execution network, the network iterative optimization process of the evaluation network can be expressed by the following equation:

[0103]

[0104] Where γ represents the discount factor of the evaluation network, s t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 U(s) is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number. t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The cost of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, Ju (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are cost-comprehensive weights, J j+1 (s t ,u t ) is used to represent the evaluation network after the (j+1)th iteration optimization for the historical driving environment state s. t and historical driving actions u t The predicted cost of driving behavior, u(s) t+n+1 ) is used to represent the execution network in response to the historical driving environment state s. t+n+1 Predicted driving behavior, J j (s t+n+1 ,u(s t+n+1 )) is used to represent the evaluation network after the j-th iteration optimization for the historical driving environment state s. t+n+1 and predicted driving behavior u(s) t+n+1 The predicted cost of driving behavior.

[0105] Simultaneously, during each iteration of the execution network's training operation, the evaluation network employs the same iteration optimization termination condition (e.g., the corresponding iteration optimization number equals a preset optimization number, or the actual loss function value of the evaluation network after optimization is less than a preset value), thereby nesting multiple iteration optimization operations of the evaluation network within a single iteration training operation of the execution network. It is understood that the actual number of iteration optimizations performed by the evaluation network in different iteration training operations of the execution network may differ.

[0106] Therefore, during one iteration training operation of the execution network, the step "iteratively optimizing the evaluation network based on the initial network weights and the multiple driving network training samples to obtain the evaluation network that satisfies the iteration optimization termination condition for this iteration training operation" may include:

[0107] For each iteration optimization operation of the evaluation network during the current iteration training operation of the execution network, determine the reference network weights of the evaluation network in the current iteration optimization operation;

[0108] The evaluation network weights are updated based on the reference network weights, the initial network weights, and the multiple driving network training samples to obtain the expected network weights of the evaluation network in this iteration optimization operation.

[0109] Detect whether the desired network weights satisfy the iterative optimization termination condition;

[0110] If the expected network weights are detected to meet the iteration optimization termination condition, the next iteration optimization operation is stopped, and the evaluation network with the expected network weights is output. Otherwise, the next iteration optimization operation is performed on the evaluation network, wherein the reference network weights of the evaluation network in the next iteration optimization operation are the expected network weights of the current iteration optimization operation.

[0111] The reference network weights used by the evaluation network in the first iteration optimization operation during the first iteration training operation of the execution network can be set to a zero matrix. The reference network weights used by the evaluation network in the first iteration optimization operation during the non-first iteration training operation of the execution network can also be set to a zero matrix. In order to improve the iteration optimization efficiency of the evaluation network, the expected network weights of the evaluation network that meet the iteration optimization termination condition in any iteration training operation (corresponding to the execution network) can be assigned to the reference network weights of the first iteration optimization operation (corresponding to the evaluation network) in the next iteration training operation.

[0112] During the i-th iteration training operation of the execution network, the network weight update process of the (j+1)-th iteration optimization operation of the evaluation network is expressed by the following equation:

[0113]

[0114] Among them, W c,j+1 W is used to represent the expected network weights of the evaluation network in the (j+1)th iteration optimization operation during the i-th iteration training operation of the execution network. c,j α is used to represent the expected network weights of the evaluation network in the j-th iteration optimization operation during the i-th iteration training operation of the execution network. c W is used to represent the learning rate of the evaluation network. c The network weights of the evaluation network are used to represent the network weights, γ is used to represent the discount factor of the evaluation network, and s is used to represent the network weights of the evaluation network. t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 This is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number, and u... i (s l |W a ) is used to indicate that the execution network utilizes the initial network weights W during the i-th iteration of training.a Based on historical driving environment conditions s l Predicted driving behavior, J(s) t ,u i (s t |W a )|W c ) is used to represent the evaluation network for historical driving environment state s. t and predict driving behavior u i (s t |W a The predicted network weight W c The cost of driving behavior, U(s) t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The cost of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are comprehensive cost weights.

[0115] Therefore, this application can nest multiple iterative optimization operations of the evaluation network during the one-time iterative training operation of the execution network by executing the specific steps of the above sub-step S251.

[0116] Sub-step S252: Based on multiple driving network training samples and the evaluation network that meets the iteration optimization termination condition, optimize the network weights of the execution network to obtain the target network weights of the execution network in this iteration training operation.

[0117] In this embodiment, the network weight optimization process of the network in the i-th iteration of training is expressed by the following equation:

[0118]

[0119] Where γ represents the discount factor of the evaluation network, s t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 This is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number, and u... i (s t ) is used to represent the execution network's response to the historical driving environment state s when completing the i-th iteration of training. t Predicted driving behavior, U(s) t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The cost of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are cost-comprehensive weights, u i (s t+n+1 ) is used to represent the execution network's response to the historical driving environment state s when completing the i-th iteration of training. t+n+1 Predicted driving behavior This is used to indicate that the evaluation network satisfies the iteration optimization termination condition during the i-th iteration training operation for the historical driving environment state s. t+n+1 and predicted driving behavior u(s) t+n+1 The predicted cost of driving behavior, W a The weights to be optimized in the execution network are represented by σ(·), and the tanh activation function is represented by σ(·).

[0120] Sub-step S253: Check whether the target network weights meet the conditions for ending iterative training.

[0121] In this embodiment, the execution network uses the same iteration training termination condition in different iteration training operations (for example, the corresponding iteration training number is equal to the preset training number, or the difference between the driving behavior costs estimated by the evaluation network optimized by each of two adjacent iteration training operations under the same driving environment condition based on the trained execution network). (less than the preset cost threshold).

[0122] After completing one iterative training operation for the execution network, the driving decision-making device 10 determines whether the iterative training operation belongs to the last iterative training operation based on whether the target network weights determined in the iterative training operation meet the iterative training termination condition. Specifically, if the target network weights determined in the current iterative training operation do not meet the iterative training termination condition, it indicates that the current iterative training operation does not belong to the last iterative training operation. The driving decision-making device 10 will then jump to sub-step S251 to continue execution and begin the next iterative training operation for the execution network. In this case, the initial network weights of the execution network in the next iterative training operation are the target network weights of the current iterative training operation.

[0123] When the target network weights determined in this iteration training operation (i.e., this iteration training operation) meet the iteration training termination condition, it indicates that this iteration training operation is the last iteration training operation. The driving decision device 10 will execute sub-step S254 to use the execution network with the target network weights determined in this iteration training operation as the driving behavior strategy network, and at the same time use the evaluation network that meets the iteration optimization termination condition determined in this iteration training operation as the driving cost evaluation network.

[0124] In sub-step S254, the execution network with the target network weights is directly used as the driving behavior policy network, and the evaluation network that meets the iteration optimization termination condition in this iteration training operation is used as the driving cost evaluation network.

[0125] Therefore, this application can ensure that the final optimized driving behavior strategy network and driving cost evaluation network can effectively adapt to dynamic traffic environment changes by repeatedly executing the above sub-steps S251 to S254, and ensure that the driving behavior prediction results output by the corresponding driving behavior strategy network for any driving environment state have sufficiently high accuracy and reliability. At the same time, the corresponding driving cost evaluation network can also accurately evaluate the driving behavior cost caused by any driving environment state and any driving behavior prediction result in a fixed period of time in the future.

[0126] Furthermore, by executing the above steps S240 to S250, this application ensures that the driving behavior strategy network and the driving cost evaluation network can perform model adaptive learning optimization based on the real-time feedback mechanism of vehicle driving environment information, so as to adapt to dynamic traffic environment changes and improve the network reliability and robustness of the driving behavior strategy network and the driving cost evaluation network in complex driving environments.

[0127] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0128] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the various functions provided in this application are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (e.g., a server, vehicle terminal, personal computer, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application as the aforementioned driving decision-making device 10. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0129] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An autonomous driving behavior decision-making method, characterized in that, The method includes: Obtain the actual driving environment status of the target vehicle at the current moment; The driving behavior policy network and the driving cost evaluation network are invoked to perform a Monte Carlo tree search based on the actual driving environment state. When the search termination condition is met, the optimal child node of the root node in the corresponding Monte Carlo tree that minimizes the driving behavior cost is determined. Each node in the Monte Carlo tree represents a driving environment state, the root node represents the actual driving environment state, and the initial driving behavior cost of each node is obtained by the driving cost evaluation network based on the node's driving behavior prediction result and the driving environment state evaluation. The driving behavior prediction result of each node is obtained by the driving behavior policy network based on the node's driving environment state prediction. The driving behavior policy network and the driving cost evaluation network are trained based on an adaptive dynamic programming structure. The driving behavior between the optimal child node and the root node is taken as the optimal driving action of the target vehicle at the current moment.

2. The method according to claim 1, characterized in that, The step of invoking the driving behavior strategy network and the driving cost evaluation network to perform Monte Carlo tree search based on the actual driving environment state includes: In each Monte Carlo tree search, based on the actual driving behavior cost and actual number of visits of each node in the Monte Carlo tree, the optimal child node is selected layer by layer starting from the root node to determine the optimal node path in this Monte Carlo tree search process. The baseline driving action of each node in the Monte Carlo tree is the driving behavior prediction result of that node. The driving behavior action between each non-root node and its parent node on the optimal node path is obtained by adding action noise to the baseline driving action of the corresponding parent node. The terminal node on the optimal node path is expanded into a sub-node, and the driving behavior prediction result of the terminal node is used as the baseline driving action from the terminal node to the expanded target sub-node. A driving simulation is performed based on the driving environment status and driving behavior prediction results of the target sub-node, and the actual number of visits to the target sub-node is set to 1. Based on the initial driving behavior cost of the target sub-node, the actual number of visits and actual driving behavior cost of each node on the optimal node path are updated backtracking.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Multiple driving network training samples are acquired, wherein each driving network training sample includes the historical driving environment state, historical driving actions, and comprehensive attenuation driving cost of the sample vehicle at the corresponding historical moment, as well as the historical driving environment state of the sample vehicle at a target moment associated with the historical moment; the target moment is separated from the historical moment by a first preset number of control moments, and the comprehensive attenuation driving cost is the sum of the comprehensive driving cost attenuation values ​​of the sample vehicle for each of the second preset number of consecutive control moments starting from the historical moment, the second preset number being obtained by adding one to the first preset number; Based on the multiple driving network training samples, an adaptive dynamic programming structure including an execution network and an evaluation network is iteratively trained to obtain the driving behavior policy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network.

4. The method according to claim 3, characterized in that, The step of iteratively training an adaptive dynamic programming structure, including an execution network and an evaluation network, based on the multiple driving network training samples to obtain the driving behavior policy network corresponding to the execution network and the driving cost evaluation network corresponding to the evaluation network includes: For each iteration training operation of the execution network, the initial network weights of the execution network in this iteration training operation are determined, and the evaluation network is iteratively optimized based on the initial network weights and the multiple driving network training samples to obtain the evaluation network that satisfies the iteration optimization termination condition for this iteration training operation. Based on the multiple driving network training samples and the evaluation network that meets the iteration optimization termination condition, the execution network is optimized to obtain the target network weights of the execution network in this iteration training operation. Detect whether the target network weights meet the iterative training termination condition; If the target network weights are detected to meet the iteration training termination condition, the execution network with the target network weights is directly used as the driving behavior policy network, and the evaluation network that meets the iteration optimization termination condition in this iteration training operation is used as the driving cost evaluation network. Otherwise, the next iteration training operation is performed on the execution network, wherein the initial network weights of the execution network in the next iteration training operation are the target network weights of this iteration training operation.

5. The method according to claim 4, characterized in that, During each iteration of the training operation of the execution network, the network iterative optimization process of the evaluation network is expressed by the following equation: Where γ represents the discount factor of the evaluation network, s t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 U(s) is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number. t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The price of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are cost-comprehensive weights, J j+1 (s t ,u t ) is used to represent the evaluation network after the (j+1)th iteration optimization for the historical driving environment state s. t and historical driving actions u t The predicted cost of driving behavior, u(s) t+n+1 ) is used to represent the execution network in response to the historical driving environment state s. t+n+1 Predicted driving behavior, J j (s t+n+1 ,u(s t+n+1 )) is used to represent the evaluation network after the j-th iteration optimization for the historical driving environment state s. t+n+1 and predicted driving behavior u(s) t+n+1 The predicted cost of driving behavior.

6. The method according to claim 4, characterized in that, The step of iteratively optimizing the evaluation network based on the initial network weights and the multiple driving network training samples to obtain an evaluation network that satisfies the iteration optimization termination condition for this iteration training operation includes: For each iteration optimization operation of the evaluation network during the current iteration training operation of the execution network, determine the reference network weights of the evaluation network in the current iteration optimization operation; The evaluation network weights are updated based on the reference network weights, the initial network weights, and the multiple driving network training samples to obtain the expected network weights of the evaluation network in this iteration optimization operation. Detect whether the desired network weights satisfy the iterative optimization termination condition; If the expected network weights are detected to meet the iteration optimization termination condition, the next iteration optimization operation is stopped, and the evaluation network with the expected network weights is output. Otherwise, the next iteration optimization operation is performed on the evaluation network, wherein the reference network weights of the evaluation network in the next iteration optimization operation are the expected network weights of the current iteration optimization operation.

7. The method according to claim 6, characterized in that, The network weight update process of the evaluation network in the (j+1)th iteration optimization operation during the i-th iteration training operation of the execution network is expressed by the following equation: Among them, W c,j+1 W is used to represent the expected network weights of the evaluation network in the (j+1)th iteration optimization operation during the i-th iteration training operation of the execution network. c,j α is used to represent the expected network weights of the evaluation network in the j-th iteration optimization operation during the i-th iteration training operation of the execution network. c W is used to represent the learning rate of the evaluation network. c The network weights of the evaluation network are used to represent the network weights, γ is used to represent the discount factor of the evaluation network, and s is used to represent the network weights of the evaluation network. t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 This is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number, and u... i (s l |W a ) is used to indicate that the execution network utilizes the initial network weights W during the i-th iteration of training. a Based on historical driving environment conditions s l Predicted driving behavior, J(s) t ,u i (s t |W a )|W c ) is used to represent the evaluation network for historical driving environment state s. t and predict driving behavior u i (s t |W a The predicted network weight W c The cost of driving behavior, U(s) t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The price of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are cost-based weights.

8. The method according to claim 4, characterized in that, The network weight optimization process of the execution network in the i-th iteration training operation is expressed by the following equation: Where γ represents the discount factor of the evaluation network, s t u is used to represent the historical driving environment state at historical time t. t s is used to represent historical driving actions at historical moment t. t+n+1 This is used to represent the historical driving environment state at a target time associated with historical time t, where n represents the first preset number, and u... i (s t ) is used to represent the execution network's response to the historical driving environment state s when completing the i-th iteration of training. t Predicted driving behavior, U(s) t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t Matching overall driving costs, J s (s t ) is used to represent the historical driving environment state s t The security cost, J c (s t ) is used to represent the historical driving environment state s t The price of comfort, J p (s t ) is used to represent the historical driving environment state s t The passability cost, J u (s t ,u t ) is used to represent the historical driving environment state s t and historical driving actions u t The cost of matching quadratic forms, ω s ω c ω p and ω u All are cost-comprehensive weights, u i (s t+n+1 ) is used to represent the execution network's response to the historical driving environment state s when completing the i-th iteration of training. t+n+1 Predicted driving behavior This is used to indicate that the evaluation network satisfies the iteration optimization termination condition during the i-th iteration training operation for the historical driving environment state s. t+n+1 and predicted driving behavior u(s) t+n+1 The predicted cost of driving behavior, W a The weights to be optimized in the execution network are represented by σ(·), and the tanh activation function is represented by σ(·).

9. A driving decision-making device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor, the processor being able to execute the computer program to implement the autonomous driving behavior decision method according to any one of claims 1-8.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a computer device, it implements the autonomous driving behavior decision-making method according to any one of claims 1-8.

Images