Decision model training method and device and driving decision method and device

Abstract

The invention discloses a decision model training method and device and a driving decision method and device, and the method comprises the steps: inputting each first candidate decision behavior corresponding to a first traffic scene into a to-be-trained decision model, and obtaining the decision probability of each first candidate decision behavior; based on the decision probability of each first candidate decision behavior, performing inverse reinforcement learning on each first candidate decision behavior to obtain an inverse reinforcement learning loss value of the first traffic scene; performing imitation learning on the first optimal decision behavior with the maximum decision probability in the first candidate decision behaviors to obtain an imitation learning loss value of the first traffic scene; based on the inverse reinforcement weight and the imitation weight, performing weighted fusion on the inverse reinforcement learning loss value and the imitation learning loss value to obtain a weighted loss value of the first traffic scene; and based on the weighted loss value, optimizing model parameters of the decision model to be trained to obtain a trained decision model. The reliability of the decision model can be improved.

Description

Technical Field

[0001] This application belongs to the field of intelligent driving technology, and in particular relates to a training method and device for a decision-making model, as well as a driving decision-making method and device. Background Technology

[0002] In recent years, intelligent driving technology has developed rapidly, attracting much attention in both theoretical research and engineering applications. With continuous technological advancements, people expect vehicles to possess more intelligent driving capabilities, assisting or even replacing humans in completing some driving tasks to improve travel safety and efficiency. Against this backdrop, driver assistance functions have become a key research direction.

[0003] Currently, vehicles equipped with driver assistance functions rely on pre-set decision-making models to operate. These models analyze the vehicle's environment and derive driving decisions appropriate to that environment. When training these models, cost or reward functions are typically constructed using manually defined rules, which guide the model to learn optimal decision-making behavior.

[0004] However, in the training method of this decision-making model, the design of the cost function or reward function depends on human experience, which makes it impossible for the decision-making model to make accurate and reliable decisions when faced with complex and ever-changing real driving scenarios, resulting in low reliability of the decision-making model. Summary of the Invention

[0005] This application provides a method and apparatus for training a decision model, as well as a method and apparatus for making driving decisions, which can significantly improve the reliability of the decision model.

[0006] A first aspect of this application provides a method for training a decision model, the method comprising:

[0007] Input each first candidate decision behavior corresponding to the first traffic scenario into the decision model to be trained to obtain the decision probability of each first candidate decision behavior;

[0008] Based on the decision probability of each first candidate decision behavior, inverse reinforcement learning is performed on each first candidate decision behavior to obtain the inverse reinforcement learning loss value of the first traffic scenario; imitation learning is performed on the first best decision behavior with the highest decision probability among the first candidate decision behaviors to obtain the imitation learning loss value of the first traffic scenario.

[0009] Based on inverse reinforcement weights and imitation weights, the inverse reinforcement learning loss value and the imitation learning loss value are weighted and fused to obtain the weighted loss value for the first traffic scenario.

[0010] Based on the weighted loss value, the model parameters of the decision model to be trained are optimized to obtain a trained decision model; the decision model is used to predict the best decision behavior of the vehicle based on the traffic scenario in which the vehicle is located.

[0011] A second aspect of this application provides a driving decision-making method, which is applied to a vehicle. The vehicle is modeled with a decision model, and the decision model is trained based on the training method for the decision model provided in any of the above aspects. The method includes:

[0012] Obtain the second scene features of the second traffic scene in which the vehicle is currently located;

[0013] Based on the features of the second scenario, predict at least one second candidate decision behavior of the vehicle;

[0014] Each second candidate decision behavior is input into the decision model to obtain the decision probability of each second candidate decision behavior;

[0015] The vehicle's movement is controlled based on the second-best decision action with the highest probability among all second-candidate decision actions.

[0016] A third aspect of this application provides a training apparatus for a decision model, the apparatus comprising:

[0017] The probability determination module is used to input each first candidate decision behavior corresponding to the first traffic scenario into the decision model to be trained, and obtain the decision probability of each first candidate decision behavior;

[0018] The behavior learning module is used to perform inverse reinforcement learning on each first candidate decision behavior based on the decision probability of each first candidate decision behavior to obtain the inverse reinforcement learning loss value of the first traffic scenario; and to perform imitation learning on the first best decision behavior with the highest decision probability among the first candidate decision behaviors to obtain the imitation learning loss value of the first traffic scenario.

[0019] The loss fusion module is used to perform weighted fusion of the inverse reinforcement learning loss value and the imitation learning loss value based on the inverse reinforcement weight and the imitation weight to obtain the weighted loss value of the first traffic scenario.

[0020] The model optimization module is used to optimize the model parameters of the decision model to be trained based on the weighted loss value, so as to obtain a trained decision model; the decision model is used to predict the best decision behavior of the vehicle based on the traffic scenario in which the vehicle is located.

[0021] A fourth aspect of this application provides a driving decision-making device applied to a vehicle, wherein the vehicle is modeled with a decision model, and the decision model is trained based on the training method for the decision model provided in any of the above aspects; the device includes:

[0022] The feature acquisition module is used to acquire the second scene features of the second traffic scene in which the vehicle is currently located;

[0023] The behavior prediction module is used to predict at least one second candidate decision behavior of the vehicle based on the features of the second scene.

[0024] The probability acquisition module is used to input each second candidate decision behavior into the decision model to obtain the decision probability of each second candidate decision behavior;

[0025] The vehicle control module is used to control the vehicle's movement based on the second best decision behavior with the highest probability among all second candidate decision behaviors.

[0026] A fifth aspect of the present application provides an electronic device, the device comprising: a memory and a program or instructions stored in the memory and executable on a processor, wherein when the program or instructions are executed by the processor, a training method for a decision model as provided in any of the above-described embodiments of the present application is implemented.

[0027] A sixth aspect of the embodiments of this application provides a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, a training method for a decision model as provided in any of the embodiments of this application described above is implemented.

[0028] A seventh aspect of the embodiments of this application provides a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a training method for a decision model as provided in any of the embodiments of this application described above.

[0029] The training method for the decision-making model provided in this application first calculates the inverse reinforcement learning loss value for the first traffic scenario based on the decision probabilities of each first candidate decision behavior obtained from the decision-making model to be trained, thereby deeply exploring the potential rules hidden behind the decision behavior. Simultaneously, imitation learning is performed on the first optimal decision behavior with the highest decision probability to obtain the imitation learning loss value for the first traffic scenario, further providing explicit supervision and constraints for inverse reinforcement learning, ensuring that the decision-making model conforms to actual traffic rules, physical limitations, and driving regulations. Next, based on the inverse reinforcement weights and imitation weights, the two are weighted and fused to obtain the weighted loss value for the first traffic scenario, combining the advantages of different learning methods to more comprehensively and accurately reflect the gap between the model output and the expected decision. Finally, the model parameters are optimized based on the weighted loss value, allowing the model to be continuously adjusted and improved, thereby more accurately establishing the mapping relationship between candidate decision behaviors and decision probabilities. Thus, the final trained decision-making model can provide more accurate and reliable decisions when facing complex and ever-changing real-world driving scenarios, effectively improving the reliability of the decision-making model. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating a training method for a decision model provided in one embodiment of this application;

[0032] Figure 2 This is a flowchart illustrating the training process of a decision model provided in one embodiment of this application;

[0033] Figure 3 This is a flowchart illustrating a driving decision-making method according to an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the structure of a training device for a decision model provided in one embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of a driving decision device provided in one embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the structure of a training device for a decision model provided in one embodiment of this application. Detailed Implementation

[0037] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0039] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0040] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0041] In the training process of traditional intelligent driving decision-making models, manually defined cost or reward functions are difficult to cover complex and ever-changing real-world driving scenarios, resulting in insufficient reliability of the model's decisions in dynamic environments. In particular, reward mechanisms constructed based on human experience cannot fully represent the correlation between multi-dimensional driving behaviors and lack adaptive adjustment capabilities in long-tail scenarios, making the model prone to making suboptimal decisions when faced with sudden traffic events or unstructured road environments.

[0042] For example, in urban intersections with multiple types of traffic participants, the fixed reward function pre-set by traditional methods struggles to balance the dynamic relationship between trajectory smoothness, collision risk, and traffic efficiency. When a pedestrian suddenly enters a vehicle's blind spot or a vehicle in an adjacent lane forcibly changes lanes, the fixed reward mechanism cannot adjust the weight distribution of each decision dimension in real time, causing the model output to deviate from the safety threshold. In such cases, the decision model may incorrectly choose to accelerate through instead of braking urgently, or prematurely execute an avoidance maneuver, triggering a chain reaction.

[0043] To address the aforementioned challenges, this application first considers how to overcome the limitations of manually preset reward functions. It explores a training mechanism capable of adaptively adjusting learning objectives, considering the complex interrelationships of multi-dimensional decision-making elements in dynamic traffic scenarios. Traditional methods with fixed weight allocation struggle to handle unforeseen situations; therefore, a fusion framework capable of capturing the core features of decision-making behavior in different scenarios is needed. By analyzing the advantages of inverse reinforcement learning in reward function derivation and the role of imitation learning in aligning behavioral norms, this application attempts to combine the two to create a complementary effect. However, simply using the two methods in parallel may lead to conflicting learning objectives. Therefore, a dynamic weight adjustment mechanism needs to be further designed to enable the model to automatically balance the importance of different learning objectives based on scenario characteristics.

[0044] In this regard, such as Figure 1 As shown, this application provides a method for training a decision model. This method can be applied to devices with high-performance computing capabilities, large-capacity data storage, and dedicated machine learning software environments, such as high-performance computers, servers, or cloud computing platforms.

[0045] The training method for this decision model may include the following steps S101 to S104:

[0046] S101, input each first candidate decision behavior corresponding to the first traffic scenario into the decision model to be trained, and obtain the decision probability of each first candidate decision behavior.

[0047] In this step, the first traffic scenario refers to a specific traffic environment condition, which encompasses various traffic-related factors, such as road type (highway, urban road, etc.), traffic flow (congestion, smooth flow), traffic light status, and the location and behavior of surrounding vehicles and pedestrians.

[0048] The first candidate decision behavior refers to a series of decision behavior options that a vehicle may take in a given first traffic scenario. For example, when approaching an intersection, candidate decision behaviors may include going straight, turning left, turning right, stopping and waiting, etc.

[0049] A decision model is a machine learning model used to make driving decisions in traffic scenarios. This model is trained to predict the optimal decision-making behavior of a vehicle based on the traffic situation it is in.

[0050] Decision probability refers to the predicted probability of a decision behavior by a decision model for each candidate decision behavior, representing the degree to which the decision model believes the candidate decision behavior is suitable for a given traffic scenario. Specifically, the decision probability can be obtained by normalizing the raw scores output by the decision model using the softmax function.

[0051] Specifically, the first step is to define the primary traffic scenario, which includes traffic-related factors such as road type, traffic flow, traffic light status, and the surrounding vehicles and pedestrians. Based on this scenario, first candidate decision behaviors are determined. For example, when a vehicle approaches an intersection, options include going straight, turning left, turning right, or stopping and waiting. Next, the determined first candidate decision behaviors are input into the decision model to be trained. Upon receiving the input information, the model generates raw scores for each candidate decision behavior. These raw scores are then normalized using a softmax function, ultimately yielding the decision probability for each candidate decision behavior. The decision probability reflects the degree to which the decision model considers each candidate decision behavior to be suitable for the given traffic scenario.

[0052] S102, based on the decision probability of each first candidate decision behavior, perform inverse reinforcement learning on each first candidate decision behavior to obtain the inverse reinforcement learning loss value of the first traffic scenario; perform imitation learning on the first best decision behavior with the highest decision probability among each first candidate decision behavior to obtain the imitation learning loss value of the first traffic scenario.

[0053] In this step, inverse reinforcement learning refers to inferring the underlying reward function by observing a preset standard decision behavior. That is, it infers the potential reward or cost in a given traffic scenario based on the decision probability of candidate decisions, thereby calculating the inverse reinforcement learning loss value. The inverse reinforcement learning loss value is the loss value that measures the difference between candidate decisions and preset standard decisions through the inverse reinforcement learning algorithm. Specifically, it can be calculated by using a binary cross-entropy function combined with binary classification labels through the inverse reinforcement learning algorithm. It is used to deduce the implicit reward function from the preset standard decisions to guide the direction of model optimization.

[0054] Imitation learning learns by having the decision-making model directly imitate the behavior of experts who meet preset standard driving rules. Specifically, imitation learning of the first-best decision behavior means teaching the decision-making model how to make decisions that meet preset standard driving rules in real-world situations; the imitation learning loss value is used to measure the degree to which the first-best decision behavior predicted by the decision-making model conforms to the preset standard driving rules.

[0055] Specifically, inverse reinforcement learning is first performed based on the decision probabilities of each first candidate decision behavior obtained from the above steps. The inverse reinforcement learning process observes the preset standard decision behavior, infers the potential reward or cost of a given traffic scenario based on the decision probability, and uses the binary cross-entropy function combined with binary classification labels to calculate and measure the difference between the candidate decision behavior and the preset standard decision behavior, thereby obtaining the inverse reinforcement learning loss value, which is used to back-infer the implicit reward function to guide model optimization.

[0056] Furthermore, the optimal decision behavior with the highest probability is identified from each candidate decision behavior and then subjected to imitation learning. This allows the decision model to mimic the behavior of experts who meet the preset standard driving specifications. By measuring the degree of conformity between the model's predicted optimal decision behavior and the preset standard driving specifications, the imitation learning loss value is obtained, thereby improving the model's ability to make decisions that conform to the specifications.

[0057] S103, based on inverse reinforcement weights and imitation weights, the inverse reinforcement learning loss value and the imitation learning loss value are weighted and fused to obtain the weighted loss value of the first traffic scenario.

[0058] In this step, the inverse reinforcement weight and the imitation weight are parameters used to adjust the importance of the inverse reinforcement learning loss value and the imitation learning loss value in the weighted fusion process. The inverse reinforcement weight reflects the contribution of the inverse reinforcement learning loss value to the final weighted loss value, while the imitation weight reflects the contribution of the imitation learning loss value to the final weighted loss value. By appropriately setting these two weights, the roles of inverse reinforcement learning and imitation learning in the training of the decision model can be balanced, enabling the model to comprehensively utilize the advantages of both learning methods.

[0059] For example, the inverse reinforcement weights and imitation weights can be fixed weights pre-set based on human experience, or dynamic weights adjusted in real time according to the actual traffic scenario. For instance, the inverse reinforcement weights and imitation weights can be determined using convolutional neural networks or graph neural networks. Convolutional neural networks extract scene spatial features through convolution operations to generate weights; graph neural networks learn weight allocation based on the interaction graph structure of traffic participants.

[0060] Weighted fusion refers to the linear combination of inverse reinforcement learning loss value and imitation learning loss value according to the dynamic importance of different learning objectives, in order to balance the policy optimization ability of inverse reinforcement learning and the rule constraint ability of imitation learning.

[0061] Specifically, the inverse reinforcement weight determines the contribution of the inverse reinforcement learning loss value to the final weighted loss value, while the imitation weight determines the contribution of the imitation learning loss value. By appropriately setting these two weights, the roles of inverse reinforcement learning and imitation learning in the training of the decision model can be balanced, allowing the model to comprehensively leverage the advantages of both learning methods.

[0062] During weighted fusion, the inverse reinforcement learning loss value and the imitation learning loss value are linearly combined based on the dynamic importance of different learning objectives. This operation can both optimize the decision-making strategy with the help of inverse reinforcement learning and provide rule constraints with the help of imitation learning, ultimately obtaining the weighted loss value of the first traffic scenario, which provides a basis for subsequent optimization of the decision model parameters.

[0063] S104. Based on the weighted loss value, the model parameters of the decision model to be trained are optimized to obtain a trained decision model. The decision model is used to predict the best decision behavior of the vehicle based on the traffic scenario in which the vehicle is located.

[0064] In this step, model parameter optimization refers to adjusting the internal parameters of the decision model through the backpropagation algorithm to minimize the weighted loss value. Specifically, it can be implemented using stochastic gradient descent or its variants to gradually improve the decision accuracy and reliability of the decision model in complex traffic scenarios.

[0065] Specifically, based on the weighted loss value calculated above, the backpropagation algorithm is used to optimize the internal parameters of the decision model to be trained. The backpropagation algorithm starts from the output layer and propagates the gradient of the weighted loss value with respect to the parameters of each layer back to each layer of the network, thereby determining the direction and magnitude of parameter adjustment. In practice, stochastic gradient descent or its variants can be used to gradually update the model parameters along the reverse direction of the gradient, continuously reducing the weighted loss value. Through this iterative optimization process, the decision model can gradually learn the mapping relationship between features in complex traffic scenarios and optimal decision behaviors, improving the accuracy and reliability of decision-making. Finally, after multiple parameter adjustments, when the weighted loss value converges to a small value, a well-trained decision model is obtained. This well-trained decision model can accurately predict the optimal decision behavior based on the traffic scenario in which the vehicle is located.

[0066] This application combines inverse reinforcement learning with imitation learning, and uses dynamic weights to fuse the loss values ​​generated by the two learning paradigms. This enables the decision-making model to learn implicit strategies from expert demonstrations and follow explicit driving rules, thereby improving the model's adaptability and decision reliability in complex and ever-changing scenarios.

[0067] As an example, we first construct a city intersection scenario with multiple traffic participants as the first traffic scenario. This scenario can include multiple vehicles, pedestrians, and non-motorized vehicles to simulate complex traffic flows. For the decision model to be trained, a deep neural network structure is used, including convolutional layers for extracting scene features and fully connected layers for decision reasoning. The input layer receives candidate decision behaviors corresponding to the traffic scenario, and the output layer provides the probability distribution of each candidate decision behavior.

[0068] Then, multiple first candidate decision behaviors corresponding to the first traffic scenario are generated, including going straight, turning left, turning right, slowing down, and stopping. These first candidate decision behaviors are input into the decision model to obtain the decision probability of each first candidate decision behavior. Specifically, each first candidate decision behavior corresponding to the first traffic scenario input into the decision model to be trained contains information in five dimensions (B, M, N, T, F). Among them, B is used to represent the number of training batches (i.e., the number of traffic scenarios in the decision model to be trained), which can be set according to the actual situation during offline training, and is generally 1 during online training; M is used to represent the number of first candidate decision behaviors; N is used to represent the number of agents, which are entities with autonomous behavior capabilities in the first traffic scenario (such as vehicles, pedestrians, cyclists, etc.); T is the time step for the future prediction of the first candidate decision behavior; and F is the feature dimension of the future trajectory of the agent first candidate decision behavior.

[0069] Then, during inverse reinforcement learning, binary labels are constructed based on the decision probability distribution to mark the first optimal decision behavior with the highest decision probability. The cross-entropy loss function is used to calculate the inverse reinforcement learning loss value, reflecting the difference between the model prediction and the preset standard decision behavior. For imitation learning, the first optimal decision behavior with the highest decision probability is selected, and the trajectory features, speed features, etc. of the first optimal decision behavior are extracted and compared with the preset standard driving specifications to calculate the imitation learning loss value.

[0070] Then, a dynamic weight allocation network is designed, taking the scene features of the traffic scenario and candidate decision behaviors as input, and outputting inverse reinforcement weights and imitation weights respectively. The inverse reinforcement learning loss value and the imitation learning loss value are weighted and summed according to these two weights to obtain the final weighted loss value.

[0071] Finally, optimization algorithms such as stochastic gradient descent are used to iteratively update the parameters of the decision model based on the weighted loss value. This process is repeated until the model converges or reaches the preset number of training epochs. The resulting trained decision model can output the optimal decision behavior and its probability for the traffic scenario in which the vehicle is located.

[0072] As another example, such as Figure 2The diagram illustrates a training flowchart for a decision-making model. First, decision model 100 outputs the decision probability of the first candidate decision action in step S210, and the weighted parsing network 200 outputs inverse reinforcement weights and imitation weights in step S220. Then, based on the decision probability of the first candidate decision action, and the inverse reinforcement weights and imitation weights, loss learning is performed in step S230. Specifically, loss learning includes inverse reinforcement learning in step S231 and imitation learning in step S232. Finally, in step S240, the inverse reinforcement learning loss value and the imitation learning loss value are weighted and fused to obtain a weighted loss value. Based on the weighted loss value, the model parameters of decision model 100 can be optimized, resulting in a decision model 100 with higher prediction accuracy.

[0073] In this embodiment, the decision probabilities of each first candidate decision behavior obtained from the decision model to be trained are first calculated using inverse reinforcement learning to determine the inverse reinforcement learning loss value for the first traffic scenario. This allows for a deeper understanding of the underlying rules hidden behind the decision behaviors. Simultaneously, imitation learning is performed on the first optimal decision behavior with the highest decision probability, yielding the imitation learning loss value for the first traffic scenario. This provides explicit supervision and constraints for inverse reinforcement learning, ensuring that the decision model conforms to actual traffic rules, physical limitations, and driving regulations. Next, based on the inverse reinforcement weights and imitation weights, the two are weighted and fused to obtain the weighted loss value for the first traffic scenario. This combines the advantages of different learning methods, providing a more comprehensive and accurate reflection of the gap between the model output and the expected decision. Finally, the model parameters are optimized based on the weighted loss value, continuously adjusting and improving the model to more accurately establish the mapping relationship between candidate decision behaviors and decision probabilities. Thus, the final trained decision model can provide more accurate and reliable decisions when facing complex and ever-changing real-world driving scenarios, effectively improving the reliability of the decision model.

[0074] In some of the solutions described above in this application, when performing inverse reinforcement learning based on the decision probability of candidate decision behaviors, there is a problem that the best decision behavior cannot be effectively distinguished from ordinary decision behavior, which makes it difficult for the model to accurately capture the core features of the preset standard decision behavior during the optimization process.

[0075] In this regard, this application further proposes that S102 may include:

[0076] Based on the decision probability of each first candidate decision behavior, obtain the binary label corresponding to each first candidate decision behavior; the binary label corresponding to each first candidate decision behavior is used to characterize whether each first candidate decision behavior is the first best decision behavior;

[0077] The decision probability of each first candidate decision behavior and its corresponding binary classification label are input into the inverse reinforcement learning loss function to obtain the inverse reinforcement learning loss value. The inverse reinforcement learning loss function is used to determine the degree of difference between the best decision behavior predicted by the decision model and the preset standard decision behavior.

[0078] In this embodiment, the binary classification label is a marker used to identify whether a candidate decision action is the optimal decision action. It is a binary value (e.g., 0 or 1) used to indicate whether a first candidate decision action is the first optimal decision action. Specifically, when the decision probability of the first candidate decision action is the highest, it is marked as 1; when the decision probability of the first candidate decision action is not the highest, it is marked as 0.

[0079] The inverse reinforcement learning loss function is a function used to measure the difference between the optimal decision predicted by a decision model and a preset standard decision. It takes the decision probability of each candidate decision and its corresponding binary label as input, and outputs an inverse reinforcement learning loss value representing the degree of difference between the optimal decision predicted by the decision model and the preset standard decision.

[0080] Preset standard decision-making behavior refers to the decision-making behavior considered optimal or most reasonable in a specific traffic scenario. Preset standard decision-making behavior can usually be pre-set based on expert knowledge, traffic rules, or historical data.

[0081] The binary classification labels are generated through a relative ranking mechanism of decision probabilities, used to clearly identify whether the first candidate decision action belongs to the first optimal decision action. The inverse reinforcement learning loss function adopts the form of a binary cross-entropy function. Its input layer receives the decision probabilities and binary classification labels, and the output layer generates a scalar loss value through log-likelihood calculation.

[0082] Specifically, during model training, after the first candidate decision behavior is probabilistically predicted, a corresponding binary label is generated through a relative ranking mechanism. This binary label, along with the decision probability of the original first candidate decision behavior, is input into a cross-entropy calculation layer containing a sigmoid activation function. The global loss value is obtained through element-wise logarithmic operations and mean calculation. This design allows the decision model to precisely adjust the parameter update direction by quantifying the matching degree between the prediction results and the standard labels, effectively improving the alignment accuracy between the decision behavior and the preset standard.

[0083] As an example, firstly, based on the decision probability of each first candidate decision action, a binary classification label is obtained for each first candidate decision action. The binary classification label for each first candidate decision action is used to characterize whether each first candidate decision action is the best first decision action. Specifically, when the decision probability of a certain first candidate decision action is the highest, its corresponding binary classification label is set to 1, indicating that the decision action is the best first decision action; otherwise, the binary classification label is set to 0, indicating that it is not the best first decision action.

[0084] Furthermore, the decision probability of each first candidate decision action and its corresponding binary label are input into the inverse reinforcement learning loss function to obtain the inverse reinforcement learning loss value. The inverse reinforcement learning loss function is used to determine the degree of difference between the optimal decision action predicted by the decision model and the preset standard decision action. Specifically, the cross-entropy loss function can be used as the inverse reinforcement learning loss function. Thus, the inverse reinforcement learning loss value is obtained by calculating the cross-entropy between the decision probability output by the decision model and the binary label.

[0085] As another example, the maximum entropy loss function can also be used as the inverse reinforcement learning loss function. Its core assumption is that the standard decision behavior has the maximum entropy property under the reward function, and the reward function is regularized by maximizing the entropy value of the trajectory distribution.

[0086] This embodiment achieves inverse reinforcement learning for each first candidate decision behavior, obtaining the inverse reinforcement learning loss value for the first traffic scenario. This avoids manually defining rules to construct cost or reward functions, improving the decision model's ability to provide accurate and reliable decisions in complex and ever-changing real-world driving scenarios. Furthermore, by introducing binary classification labels, the first optimal decision behavior and non-first optimal decision behaviors are clearly distinguished, helping the decision model to better learn the optimal decision strategy.

[0087] In some of the solutions described above in this application, when calculating the inverse reinforcement learning loss value based on the decision probability of candidate decision behaviors and binary classification labels, the influence of different training batches and the number of candidate decision behaviors on the loss function is not considered, which may lead to problems such as training instability or convergence difficulties during the training process.

[0088] To this end, this application further proposes that before inputting the decision probability of each first candidate decision behavior and its corresponding binary classification label into the inverse reinforcement learning loss function to obtain the inverse reinforcement learning loss value, the method further includes:

[0089] The scope of inverse reinforcement learning is determined based on the number of training batches for each traffic scenario and the number of candidate decision behaviors corresponding to each traffic scenario.

[0090] Based on the decision probability of each candidate decision behavior and the corresponding binary classification label, the individual inverse reinforcement loss value of each candidate decision behavior is determined by the binary cross-entropy function.

[0091] Based on the individual inverse reinforcement loss value of each candidate decision behavior within the inverse reinforcement learning range, an inverse reinforcement learning loss function is constructed.

[0092] In this embodiment, the inverse reinforcement learning range is determined by the product of the number of training batches and the number of candidate decision behaviors. For example, when there are 5 training batches and each batch contains 10 candidate decision behaviors, the inverse reinforcement learning range is 50. The binary cross-entropy function calculates the difference between the decision probability of each candidate decision behavior and the binary classification label, generating an individual inverse reinforcement loss value. Specifically, the decision probability value is substituted into the logarithmic loss function. The individual inverse reinforcement loss values ​​are averaged or summed within the inverse reinforcement learning range to form the final inverse reinforcement learning loss function.

[0093] Specifically, during training, the number of training batches in the current training phase is first counted, for example, set to 5 training batches, with each batch corresponding to one traffic scene instance. Each traffic scene instance corresponds to multiple candidate decision behaviors, for example, each scene contains 8 candidate decision behaviors. By multiplying the number of training batches by the number of candidate decision behaviors, the inverse reinforcement learning scope is determined to cover 40 candidate decision behavior samples. For each candidate decision behavior, its decision probability and binary classification label are input into a binary cross-entropy function to calculate the individual inverse reinforcement learning loss value. The individual inverse reinforcement learning losses of all candidate decision behaviors are averaged or summed within the inverse reinforcement learning scope to generate the global inverse reinforcement learning loss value. This method dynamically adjusts the inverse reinforcement learning scope to ensure the consistency of data scale across different training phases, avoiding differences in the loss function dimensions due to fluctuations in the number of samples, thereby improving the stability of model training.

[0094] As an example, the inverse reinforcement learning loss function is shown in the following formula (1):

[0095]

[0096] Among them, L IRL The inverse reinforcement learning loss value, p, is used to characterize the inverse reinforcement learning loss function. b,m y is used to characterize the decision probability of the m-th candidate decision behavior in the b-th training batch. b,m The binary label value used to characterize the m-th candidate decision behavior in the b-th training batch, where B represents the number of training batches and M represents the number of candidate decision behaviors in each training batch.

[0097] This embodiment effectively improves the accuracy and stability of inverse reinforcement learning. As a result, the decision-making model can learn the optimal decision-making behavior more accurately, thereby enhancing its decision-making ability in complex traffic scenarios. Furthermore, by introducing the scope of inverse reinforcement learning, overfitting can be avoided, enhancing the model's generalization ability. In addition, using the binary cross-entropy function to calculate the individual inverse reinforcement loss value can better capture the difference between the decision probability and the actual label, thus optimizing the direction and magnitude of model parameter adjustments.

[0098] In some of the solutions described above in this application, when imitation learning is performed based on the first best decision behavior with the highest decision probability, it is difficult to comprehensively measure the degree of conformity between the best decision behavior and the preset standard driving specifications through evaluation of only a single dimension, which limits the optimization effect of the model.

[0099] In this regard, this application further proposes that S102 may specifically include:

[0100] To obtain the learning dimensions of the first optimal decision-making behavior imitation learning, and the characteristics of decision-making behavior in the learning dimensions;

[0101] Based on the characteristics of decision-making behavior, the imitation learning loss value is obtained through the imitation learning loss function corresponding to the learning dimension; the imitation learning loss function is used to determine the degree of conformity between the best decision-making behavior predicted by the decision model and the preset standard driving specifications in the learning dimension.

[0102] In this embodiment, the learning dimension refers to the specific aspects or feature categories focused on during the imitation learning process. It defines the angles from which the first optimal decision-making behavior is analyzed and learned. For example, learning dimensions may include road consistency learning dimensions, path smoothness learning dimensions, and inter-frame continuity learning dimensions. The road consistency learning dimension is used to constrain the driving trajectory to be consistent with the lane centerline and its direction; the path smoothness learning dimension is used to penalize dangerous actions such as rapid acceleration / sharp turns; and the inter-frame continuity learning dimension is used to ensure the physical continuity of the driving trajectory. Different learning dimensions correspond to different aspects of driving behavior. By comprehensively considering multiple learning dimensions, the optimal decision-making behavior can be evaluated and imitated more comprehensively. The imitation learning loss function corresponding to the learning dimension can adopt the mean squared error function or the cross-entropy function, and independent loss calculation logic is designed for different dimensions.

[0103] Decision-making behavior characteristics refer to specific information that describes and quantifies the first-best decision-making behavior under a particular learning dimension. For example, when the learning dimension is speed control, decision-making behavior characteristics could be vehicle speed, acceleration, etc.; when the learning dimension is lane keeping, decision-making behavior characteristics could be the distance between the vehicle and the lane centerline, the lateral position of the vehicle within the lane, etc. These characteristics can specifically reflect the performance of the first-best decision-making behavior in the learning dimension.

[0104] Specifically, when the learning dimension of imitation learning is trajectory learning, the trajectory coordinates of the first optimal decision behavior at multiple time points are collected and compared point-by-point with the corresponding trajectory coordinates of the preset standard decision behavior. The squared position deviation at each time point is calculated and averaged to form the imitation learning loss value for the trajectory learning dimension. When the learning dimension of imitation learning is collision learning, the distance between the agent and the nearest obstacle is monitored in real time. When this distance is less than a preset safe distance threshold, the difference between the distance and the safe distance threshold is amplified by an exponential function to generate the imitation learning loss value for the collision dimension. By weighted summing of multi-dimensional loss values, the performance of the decision behavior in terms of trajectory compliance and collision safety can be constrained simultaneously, ensuring that the optimal decision behavior output by the model not only meets the path planning requirements but also effectively avoids collision risks.

[0105] As an example, we first obtain the learning dimensions of the first optimal decision-making behavior imitation learning, as well as the decision-making behavior characteristics within those learning dimensions. These learning dimensions can include a trajectory learning dimension and a collision learning dimension. The trajectory learning dimension measures the distance deviation between the driving trajectory of the optimal decision-making behavior and the driving trajectory of the preset standard decision-making behavior. The collision learning dimension measures the probability of collision between the agent and the nearest obstacle in the optimal decision-making behavior.

[0106] Furthermore, based on the characteristics of the decision-making behavior, the imitation learning loss value is obtained through the imitation learning loss function corresponding to the learning dimension. The imitation learning loss function is used to determine the degree to which the optimal decision-making behavior predicted by the decision model conforms to the preset standard driving norms in the learning dimension. For example, for the trajectory learning dimension, the imitation learning loss value of the first optimal decision-making behavior can be determined based on the positional deviation between the first optimal decision-making behavior and the preset standard decision-making behavior at each time step in the first traffic scenario. For the collision learning dimension, the imitation learning loss value of the first optimal decision-making behavior can be determined based on the difference between the distance between the agent and the nearest obstacle and the safe distance threshold at each time step in the first optimal decision-making behavior in the first traffic scenario.

[0107] Therefore, by introducing multiple learning dimensions for imitation learning, the degree to which the optimal decision-making behavior predicted by the decision-making model conforms to the preset standard driving norms can be evaluated from multiple perspectives, thereby improving the comprehensiveness and accuracy of imitation learning.

[0108] This embodiment enables multi-dimensional imitation learning evaluation of the optimal decision-making behavior predicted by the decision-making model. By introducing multiple learning dimensions, the optimal decision-making behavior can be comprehensively evaluated. This multi-dimensional imitation learning method can more accurately measure the performance of the decision-making model and improve its adaptability and reliability in complex traffic scenarios.

[0109] In some of the solutions mentioned above in this application, the temporal continuity of training data is not considered during the imitation learning process. As a result, the evaluation of positional deviation in the trajectory learning dimension is based only on the instantaneous state of a single scene, which makes it difficult to accurately reflect the overall trajectory matching degree of the agent in a dynamic environment and affects the optimization effect of the decision model on the driving trajectory.

[0110] In response, this application further proposes a learning dimension including a trajectory learning dimension, which is used to measure the distance deviation between the driving trajectory of the optimal decision-making behavior and the driving trajectory of the preset standard decision-making behavior.

[0111] Based on the characteristics of decision-making behavior, before obtaining the imitation learning loss value through the imitation learning loss function corresponding to the learning dimension, this method also includes:

[0112] The scope of imitation learning is determined based on the number of training batches in the traffic scenario.

[0113] Based on the positional deviation between the best decision-making behavior and the preset standard decision-making behavior of each traffic scenario at each time point, the individual imitation loss value of each best decision-making behavior is determined.

[0114] Based on the individual imitation loss values ​​of the best decision behaviors within the imitation learning range, an imitation learning loss function for the trajectory learning dimension is constructed.

[0115] In this embodiment, the trajectory learning dimension is specifically used to measure the distance deviation between the driving trajectory of the optimal decision-making behavior and the driving trajectory of the preset standard decision-making behavior. The driving trajectory reflects the movement path of the agent in a traffic scenario. By comparing the difference between the actual decision trajectory and the standard trajectory, the rationality and standardization of the decision-making behavior in terms of trajectory planning can be evaluated. For example, in an autonomous driving scenario, the preset standard decision-making behavior might be to drive smoothly along the center line of the road, while if the driving trajectory of the optimal decision-making behavior deviates significantly from the center line, it indicates a deviation in the trajectory learning dimension.

[0116] The scope of imitation learning is determined based on the number of training batches in the traffic scenario. It defines the range of data considered when constructing the imitation learning loss function for the trajectory learning dimension. For example, if the number of training batches is large, the imitation learning scope may cover more traffic scenarios and decision-making behavior data, thus making the loss function more reflective of the model's performance under different conditions; conversely, if the number of training batches is small, the imitation learning scope may be relatively narrow.

[0117] Positional deviation refers to the difference in position between the agent corresponding to the optimal decision-making behavior and the agent corresponding to the preset standard decision-making behavior at any given time. By calculating the positional deviation, the degree of deviation between the two decision-making behaviors in spatial location can be quantified. For example, at a certain moment, if there is a certain distance difference between the vehicle position of the optimal decision-making behavior and the vehicle position of the preset standard decision-making behavior in the lateral or longitudinal direction, this distance difference is the positional deviation.

[0118] Specifically, when constructing the imitation learning loss function for the trajectory learning dimension, the optimal decision behaviors of N consecutive training batches are first obtained, with the value of N dynamically adjusted according to the current training stage. For the agent trajectory data in the optimal decision behavior of each training batch, the set of coordinate points between the agent and the standard trajectory at times t0 to tn is extracted, and the spatial distance between each time point is calculated through matrix operations. Subsequently, the deviation values ​​at each time point are input into an exponential smoothing function to generate a time-weighted individual imitation loss value. Finally, all individual loss values ​​within the N training batches are normalized, and the root mean square error algorithm is used to generate the imitation learning loss function for the trajectory learning dimension. This process enables the decision model to capture the distribution pattern of trajectory deviations in multiple training batches, improving trajectory fitting accuracy by strengthening temporal continuity and batch correlation.

[0119] As an example, the imitation learning loss function for the trajectory learning dimension is shown in the following formula (2):

[0120]

[0121] In formula (2), L dist The imitation learning loss value in the imitation learning loss function is used to characterize the trajectory learning dimension. The coordinates of the nth agent at time t are used to represent the best decision-making behavior in the bth training batch. B is used to represent the coordinates of the nth agent in the preset standard decision-making behavior at time t, N is used to represent the number of training batches, T is used to represent the number of agents in the optimal decision-making behavior, and T is used to represent the driving time of the agent.

[0122] This embodiment enables precise evaluation and optimization of the decision-making model in the trajectory learning dimension. Introducing the trajectory learning dimension effectively measures the deviation between the predicted trajectory and the standard trajectory. Constructing an imitation learning range based on multiple training batches improves the stability and robustness of the evaluation. Calculating the agent's positional deviation to obtain the individual imitation loss value allows for fine-grained quantification of trajectory deviation. The final constructed imitation learning loss function comprehensively reflects the decision-making model's performance in trajectory prediction, providing effective guidance for model optimization.

[0123] In some of the solutions mentioned above in this application, the imitation learning process mainly focuses on the trajectory learning dimension. The decision model is optimized by measuring the deviation between the driving trajectory and the preset standard trajectory. However, the optimization of trajectory deviation cannot directly reflect the collision risk between the agent and the obstacle, which may lead to the decision model generating decision behaviors with potential collision risks in complex traffic scenarios.

[0124] In this regard, this application further proposes a learning dimension including a collision learning dimension, which is used to measure the probability of collision between the agent and the nearest obstacle in the best decision-making behavior;

[0125] Based on the characteristics of decision-making behavior, before obtaining the imitation learning loss value through the imitation learning loss function corresponding to the learning dimension, this method also includes:

[0126] The scope of imitation learning is determined based on the number of training batches in the traffic scenario.

[0127] Based on the difference between the target distance and the safe distance threshold of the agent at each time step for the best decision-making behavior in each traffic scenario, the individual imitation loss value of each best decision-making behavior is determined; the target distance is used to characterize the distance between the agent and the nearest obstacle.

[0128] Based on the individual imitation loss values ​​of the best decision behaviors within the imitation learning range, an imitation learning loss function for the collision learning dimension is constructed.

[0129] In this embodiment, the collision learning dimension is specifically used to measure the probability of a collision between the agent (e.g., an autonomous vehicle) and the nearest obstacle during optimal decision-making behavior, i.e., the collision probability. By focusing on the collision learning dimension, it can be ensured that the decision-making behavior generated by the decision model can effectively avoid collisions, ensuring the safety of the agent and its surrounding environment. For example, in autonomous driving scenarios, the collision learning dimension assesses the risk of the vehicle colliding with vehicles, pedestrians, obstacles, etc., ahead when performing decision-making behavior.

[0130] The target distance is used to characterize the distance between the agent and the nearest obstacle. At each moment, the decision model calculates the actual distance between the agent and the nearest surrounding obstacle; this distance is the target distance. For example, in autonomous driving, the distance between the vehicle and the vehicle in front is the target distance.

[0131] A safe distance threshold is a pre-defined standard for maintaining a safe distance from obstacles when an agent makes decisions, thus preventing collisions. The setting of the safe distance threshold typically considers factors such as the agent's speed, braking performance, and reaction time. For example, the safe distance threshold is higher when driving on a highway than when driving on city roads.

[0132] The difference between the target distance and the safe distance threshold refers to the numerical difference between the target distance and the safe distance threshold of the agent at each time point in each traffic scenario, representing the optimal decision-making behavior. If the target distance is less than the safe distance threshold, the difference is negative, indicating that the agent is too close to the obstacle and there is a risk of collision; if the target distance is greater than the safe distance threshold, the difference is positive, indicating that the agent maintains a safe distance from the obstacle.

[0133] Specifically, during the training of the decision-making model, for each training batch of traffic scenarios, the agent's motion sequence corresponding to the best decision behavior is extracted. For the agent's position at each moment, the real-time distance to the nearest obstacle is calculated as the target distance. This target distance is compared with a preset safe distance threshold. If the target distance is lower than the safe distance threshold, a loss value proportional to the corresponding difference is generated. The loss values ​​for all moments and traffic scenarios are averaged within a set training batch range to form the final imitation learning loss function for the collision learning dimension. The model parameters are optimized through backpropagation, causing the decision-making model to actively increase the distance to obstacles when generating the best decision, thereby reducing the probability of collision. For example, the safe distance threshold can be set to 2 meters. When the detected target distance is 1.5 meters, a loss value of 0.5 is generated; if the distance is further reduced to 1 meter, the loss value increases to 1.0. This non-linear loss calculation method forces the model to prioritize the safe distance during training, thereby improving the safety of decision-making behavior.

[0134] As an example, the imitation learning loss function for the collision learning dimension is shown in the following formula (3):

[0135]

[0136] In formula (3), L col The imitation learning loss value in the imitation learning loss function is used to characterize the collision learning dimension. This is used to characterize the distance between the nth agent and the nearest obstacle at time t in the best decision-making behavior of the bth training batch. B is used to characterize the safe distance threshold, N is used to characterize the number of training batches, T is used to characterize the number of agents in the best decision behavior, and T is used to characterize the driving time of the agents.

[0137] This embodiment effectively measures the collision risk of decision-making behavior, improving the decision-making model's ability to perceive and avoid obstacles. By introducing a collision learning dimension, the decision-making model can learn safer driving strategies, reducing the probability of collisions. Simultaneously, the dynamically adjusted safe distance threshold allows the model to adapt to different driving environments and speed conditions, enhancing the flexibility and adaptability of decision-making.

[0138] In some of the schemes mentioned above in this application, the inverse reinforcement weights and imitation weights use fixed values, which cannot adapt to the dynamic changes of different traffic scenarios. This results in the model training process being unable to dynamically adjust the loss weights according to scenario features and candidate decision behavior features, thus affecting the model optimization effect.

[0139] In this regard, prior to S103, the method further includes:

[0140] Obtain the first scene features of the first traffic scenario, as well as the first candidate decision-making behaviors of each first traffic scenario;

[0141] Global feature aggregation is performed on the decision behaviors of each first candidate to obtain aggregated behavioral features;

[0142] The aggregated behavioral features are fused with the first scene features to obtain joint features;

[0143] By encoding the joint features using a multilayer perceptron, inverse reinforcement weights and imitation weights are obtained.

[0144] In this embodiment, global feature aggregation can be achieved through an attention mechanism or average pooling operation to extract the overall distribution pattern of the first candidate decision behavior. The first scene features may include the location of traffic participants and road topology information, extracted through a convolutional neural network. Joint features can be fused with the first scene features and aggregated behavior features using channel concatenation or weighted summation. The multilayer perceptron contains two fully connected layers and a ReLU activation function; the output layer generates weight coefficients using a Softmax function.

[0145] Specifically, during model training, the first candidate decision behavior includes information in five dimensions: (B, M, N, T, F). B represents the number of training batches (i.e., the number of traffic scenarios input into the decision model to be trained), which can be set according to actual conditions during offline training, but is generally 1 during online training; M represents the number of first candidate decision behaviors; N represents the number of agents, which are entities with autonomous behavior capabilities in the first traffic scenario (such as vehicles, pedestrians, cyclists, etc.); T is the time step for predicting the future of the first candidate decision behavior; and F is the feature dimension of the future trajectory of the agent for the first candidate decision behavior. Obstacle density, lane curvature, etc., of the traffic scenario constitute the first scene features of the first traffic scenario.

[0146] Global feature aggregation employs spatiotemporal pooling to extract key features of the first candidate decision behavior, which are then concatenated with the first scene features to form a joint feature vector. This joint feature vector is input into a Multi-Layer Perceptron (MLP) network. The first layer compresses the dimension to half of the original feature size, the second layer restores the original dimension, and the final output layer generates normalized inverse reinforcement weights and imitation weights. For example, when the first scene features show high-density obstacles, the network outputs higher imitation weights to enhance trajectory tracking capabilities; when the first scene features show open roads, the network increases the inverse reinforcement weights to enhance behavior optimization capabilities. Through a dynamic weight allocation mechanism, the contribution of different loss terms can be adaptively adjusted according to real-time scene features, achieving more accurate gradient backpropagation and parameter updates.

[0147] As an example, the first candidate decision behavior is aggregated globally using spatiotemporal pooling with the following formula (4) to obtain the aggregated behavior features:

[0148] X scene =Pooling(X input )∈R B*M*F′ (4)

[0149] In formula (4), X scene Pooling is used to characterize aggregation behavior features, and X is used to characterize spatiotemporal pooling. input R is used to represent the first candidate decision behavior, B is used to represent the set of real numbers, M is used to represent the number of first candidate decision behaviors in the training batch, and F′ is used to represent the feature dimension after processing.

[0150] Then, the first scene features are linearly transformed using the following formula (5) to obtain the linearly transformed first scene features:

[0151] X' env =Linear(X) env )∈R B*F′ (5)

[0152] In formula (5), X' env X is used to represent the first scene features after linear transformation; Linear is used to represent the linear layer for dimensionality reduction of the first scene features; R is used to represent the set of real numbers; B is used to represent the number of training batches; and F′ is used to represent the feature dimension after processing. env Used to characterize the features of the first scene, wherein the features of the first scene include (B, N) scene F scene There are three dimensions: B represents the number of training batches, N... scene F is used to characterize the number of environmental elements (e.g., road topology, historical traffic participants, etc.) in the first traffic scenario.scene Environmental feature dimensions used to characterize environmental elements.

[0153] Then, the aggregated behavioral features are fused with the linearly transformed first scene features using the following formula (6) to obtain the joint features:

[0154] X joint =Concat(MeanPooling(X) scene ), X' env )∈R B*2F′ (6)

[0155] In formula (6), X joint Used to characterize joint features, X scene Used to characterize aggregation behavior features, X' env The first scene feature is represented by the linear transformation, MeanPooling is represented by averaging the first candidate decision behaviors of M, Concat is represented by concatenation, R is represented by the set of real numbers, B is represented by the number of training batches, and F′ is represented by the feature dimension after processing.

[0156] Finally, the joint features are encoded using the following formulas (7) to (9) through a multilayer perceptron to obtain the inverse reinforcement weights and imitation weights:

[0157] h1 = ReLU(W1X) joint +b1)∈R B*d (7)

[0158] In formula (7), h1 represents the output of the first MLP layer, R represents the set of real numbers, B represents the number of training batches, d represents the dimension of the hidden layer, ReLU represents the activation function and is used to introduce nonlinearity, W1 represents the weight matrix of the first MLP layer, b1 represents the bias vector of the first MLP layer, and X represents the weight matrix of the first MLP layer. joint Joint features used to characterize the input.

[0159] h2=ReLU(W2h1+b2)∈R B*d (8)

[0160] In formula (8), h2 is used to represent the output of the second MLP, R is used to represent the set of real numbers, B is used to represent the number of training batches, d is used to represent the dimension of the hidden layer, ReLU is used to represent the activation function and is used to introduce nonlinearity, W2 is used to represent the weight matrix of the second MLP, b2 is used to represent the bias vector of the second MLP, and h1 is used to represent the output of the first MLP.

[0161] a=Softmax(W3h2+b3)∈R B*K (9)

[0162] In formula (9), a represents the weight of each loss function, h2 represents the output of the second MLP, b3 represents the bias vector of the third MLP, W3 represents the weight matrix of the second MLP, B represents the number of training batches, R represents the set of real numbers, K represents the total number of loss functions, and Softmax represents the activation function used to convert the output into a probability distribution.

[0163] This embodiment achieves adaptive weighting of inverse reinforcement learning loss and imitation learning loss. By introducing first scene features, the differences between different scenes are considered, making the weighting more flexible and reasonable. Multilayer perceptron encoding is used to improve the non-linear expressive power of weight generation. Compared with fixed weights, this adaptive weighting method is more adaptable to complex and changing traffic scenarios, improving the generalization ability and robustness of the decision model.

[0164] In some of the solutions described above in this application, the decision-making model cannot provide accurate and reliable decisions in complex and ever-changing real-world driving scenarios, resulting in low vehicle reliability.

[0165] In response, this application further proposes a driving decision-making method for vehicles, wherein the vehicle is modeled with a decision-making model, and the decision-making model is trained based on the training method of the decision-making model provided in any of the above aspects.

[0166] like Figure 3 As shown, the driving decision-making method includes the following steps S301 to S304:

[0167] S301, Obtain the second scene features of the second traffic scene where the vehicle is currently located;

[0168] S302, based on the features of the second scene, predict at least one second candidate decision behavior of the vehicle;

[0169] S303, Input each second candidate decision behavior into the decision model to obtain the decision probability of each second candidate decision behavior;

[0170] S304, based on the second best decision behavior with the highest decision probability among all second candidate decision behaviors, control the vehicle's movement.

[0171] In this embodiment, the second scene features include the location of traffic participants, road topology, traffic signal status, and obstacle distribution information; the candidate decision behavior prediction is achieved through a preset behavior generation algorithm, which dynamically generates candidate behaviors such as lane changing, following other vehicles, or emergency braking based on scene features; the decision model is implemented through a multi-layer neural network architecture, where the input layer receives the candidate decision behavior feature vector, the hidden layer performs nonlinear transformation, and the output layer generates a probability distribution; the selection of the behavior with the highest probability is achieved by comparing the activation values ​​of the output layer, where the activation value corresponds to the confidence level of the candidate behavior.

[0172] Specifically, the vehicle collects dynamic data of the second traffic scenario in real time through the environmental perception module, including the speed of surrounding vehicles, pedestrian movement trajectories, and road boundary information. This data is then processed through feature extraction to form second scenario features. The candidate decision-making behavior generation module enumerates possible driving strategies based on the scenario features, such as generating candidate behaviors for lane keeping or lane changing in a congested scenario. The decision model performs a probability assessment on each candidate behavior, integrating historical driving data and real-time environmental parameters in the probability calculation process to ensure that the assessment results conform to actual driving patterns. Finally, the control module generates control commands based on the candidate behavior with the highest probability, driving the vehicle to perform acceleration, steering, or braking operations. For example, in an intersection scenario, it prioritizes the non-collision-free passage path with the highest probability to avoid collisions with vehicles approaching from the side.

[0173] Specifically, the trained decision-making model can employ a multilayer perceptron and softmax structure to predict the optimal decision behavior. First, the second candidate decision behavior is dimensionality-reduced and encoded to obtain an encoded feature vector. Then, the encoded feature vector is deeply mapped using a multilayer perceptron to learn the non-linear relationship between the decision behavior and the score. Finally, the score output by the multilayer perceptron is mapped to a probability distribution using a softmax function, thus obtaining the decision probability of the second candidate decision behavior.

[0174] Furthermore, decision-making models can also employ convolutional neural networks, graph neural networks, or Transformer architectures. Convolutional neural networks extract spatial features through rasterized scene feature maps, making them suitable for structured road scenarios; graph neural networks model traffic participants as graph nodes, learning dynamic features through interaction relationships, adapting to complex scenarios; and Transformers utilize attention mechanisms to capture long-term temporal dependencies in multimodal scenarios, enhancing dynamic modeling capabilities.

[0175] As an example, when a vehicle enters a merging zone on an urban expressway, the environmental perception system collects real-time data on the movement trajectories of surrounding vehicles, road boundary coordinates, and traffic sign information, forming a second scene feature that includes dynamic obstacle velocity vectors and static road topology. Candidate decision behaviors are predicted as three operating modes: lane keeping, acceleration for overtaking, and emergency braking. The decision probability for acceleration for overtaking reaches 0.82. Based on the probability distribution, the decision model selects acceleration for overtaking as the execution strategy, adjusting power output through the longitudinal controller while the lateral controller plans a smooth lane-changing trajectory, ultimately achieving a safe and efficient merging operation.

[0176] This embodiment effectively addresses the problem of insufficient adaptability of traditional decision-making models in complex traffic scenarios. By acquiring multi-dimensional scene features in real time and using a pre-trained model for probability evaluation, the vehicle can dynamically select the optimal driving strategy, avoiding the decision-making rigidity caused by manually preset rules. This solution demonstrates stronger environmental adaptability in scenarios such as sudden obstacle avoidance and multi-vehicle interactive game, significantly improving the decision-making reliability and execution stability of the autonomous driving system in real road environments.

[0177] The application provides a training method for the decision model. Accordingly, it also provides specific embodiments of a training apparatus for the decision model.

[0178] like Figure 4 As shown, the training device 400 for the decision model provided in this application embodiment includes a probability determination module 410, a behavior learning module 420, a loss fusion module 430, and a model optimization module 440.

[0179] The probability determination module 410 is used to input each first candidate decision behavior corresponding to the first traffic scenario into the decision model to be trained, and obtain the decision probability of each first candidate decision behavior.

[0180] The behavior learning module 420 is used to perform inverse reinforcement learning on each first candidate decision behavior based on the decision probability of each first candidate decision behavior to obtain the inverse reinforcement learning loss value of the first traffic scenario; and to perform imitation learning on the first best decision behavior with the highest decision probability among the first candidate decision behaviors to obtain the imitation learning loss value of the first traffic scenario.

[0181] The loss fusion module 430 is used to perform weighted fusion of the inverse reinforcement learning loss value and the imitation learning loss value based on the inverse reinforcement weight and the imitation weight to obtain the weighted loss value of the first traffic scenario.

[0182] The model optimization module 440 is used to optimize the model parameters of the decision model to be trained based on the weighted loss value, so as to obtain a trained decision model; the decision model is used to predict the best decision behavior of the vehicle based on the traffic scenario in which the vehicle is located.

[0183] In the training device for the decision-making model provided in this application embodiment, the decision probability of each first candidate decision behavior obtained from the decision-making model to be trained is first used to calculate the inverse reinforcement learning loss value of the first traffic scenario through inverse reinforcement learning, thereby deeply exploring the potential rules hidden behind the decision behavior. At the same time, imitation learning is carried out on the first best decision behavior with the highest decision probability to obtain the imitation learning loss value of the first traffic scenario, further providing explicit supervision and constraints for inverse reinforcement learning, ensuring that the decision-making model conforms to actual traffic rules, physical constraints, and driving regulations. Then, based on the inverse reinforcement weights and imitation weights, the two are weighted and fused to obtain the weighted loss value of the first traffic scenario, which combines the advantages of different learning methods and can more comprehensively and accurately reflect the gap between the model output and the expected decision. Finally, the model parameters are optimized based on the weighted loss value, so that the model is continuously adjusted and improved, thereby more accurately establishing the mapping relationship between candidate decision behaviors and decision probabilities. In this way, the final trained decision-making model can give more accurate and reliable decisions when facing complex and ever-changing actual driving scenarios, effectively improving the reliability of the decision-making model.

[0184] Furthermore, this application also proposes that the behavior learning module 420 includes the following units:

[0185] The label acquisition unit is used to acquire the binary label corresponding to each first candidate decision behavior based on the decision probability of each first candidate decision behavior; the binary label corresponding to each first candidate decision behavior is used to characterize whether each first candidate decision behavior is the first best decision behavior.

[0186] The inverse reinforcement learning unit is used to input the decision probability of each first candidate decision behavior and its corresponding binary classification label into the inverse reinforcement learning loss function to obtain the inverse reinforcement learning loss value; the inverse reinforcement learning loss function is used to determine the degree of difference between the best decision behavior predicted by the decision model and the preset standard decision behavior.

[0187] Furthermore, this application also proposes that, before inputting the decision probability of each first candidate decision behavior and its corresponding binary classification label into the inverse reinforcement learning loss function to obtain the inverse reinforcement learning loss value, the behavior learning module 420 further includes the following units:

[0188] The scope determination unit is used to determine the scope of inverse reinforcement learning based on the number of training batches in the traffic scenario and the number of candidate decision behaviors corresponding to the traffic scenario in each training batch.

[0189] The loss value determination unit is used to determine the individual inverse reinforcement loss value of each candidate decision behavior based on the decision probability of each candidate decision behavior and the corresponding binary classification label, through the binary cross-entropy function.

[0190] The function building unit is used to construct the inverse reinforcement learning loss function based on the individual inverse reinforcement loss value of each candidate decision behavior within the inverse reinforcement learning range.

[0191] Furthermore, this application also proposes that the behavior learning module 420 further includes the following units:

[0192] The feature acquisition unit is used to acquire the learning dimension of the first optimal decision behavior imitation learning, and the decision behavior features in the learning dimension;

[0193] The imitation learning unit is used to obtain the imitation learning loss value based on the characteristics of decision-making behavior through the imitation learning loss function corresponding to the learning dimension. The imitation learning loss function is used to determine the degree of conformity between the best decision-making behavior predicted by the decision model and the preset standard driving specifications in the learning dimension.

[0194] Furthermore, this application also proposes that the learning dimension includes a trajectory learning dimension, which is used to measure the distance deviation between the driving trajectory of the optimal decision-making behavior and the driving trajectory of the preset standard decision-making behavior.

[0195] Before obtaining the imitation learning loss value based on the decision-making behavior characteristics and through the imitation learning loss function corresponding to the learning dimension, the behavior learning module 420 also includes the following units:

[0196] The range determination unit is also used to determine the range of imitation learning based on the number of training batches in the traffic scenario;

[0197] The loss value determination unit is also used to determine the individual imitation loss value of each optimal decision behavior based on the positional deviation between the best decision behavior and the preset standard decision behavior at each time point in each traffic scenario.

[0198] The function building unit is also used to construct the imitation learning loss function for the trajectory learning dimension based on the individual imitation loss value of each best decision behavior within the imitation learning range.

[0199] Furthermore, this application also proposes that the learning dimension includes a collision learning dimension, which is used to measure the probability of collision between the agent and the nearest obstacle in the best decision-making behavior;

[0200] Before obtaining the imitation learning loss value based on the decision-making behavior characteristics and through the imitation learning loss function corresponding to the learning dimension, the behavior learning module 420 also includes the following units:

[0201] The range determination unit is also used to determine the range of imitation learning based on the number of training batches in the traffic scenario;

[0202] The loss value determination unit is also used to determine the individual imitation loss value of each optimal decision behavior based on the difference between the target distance and the safe distance threshold of the agent at each time step for the optimal decision behavior in each traffic scenario; the target distance is used to characterize the distance between the agent and the nearest obstacle.

[0203] The function building unit is also used to construct the imitation learning loss function for the collision learning dimension based on the individual imitation loss value of each best decision behavior within the imitation learning range.

[0204] Furthermore, this application also proposes that, before weighted fusion of the inverse reinforcement learning loss value and the imitation learning loss value based on inverse reinforcement weights and imitation weights to obtain the weighted loss value for the first traffic scenario, the training device 400 of the decision model further includes the following modules:

[0205] The feature acquisition module is used to acquire the first scene features of the first traffic scenario, as well as the first candidate decision behaviors of the first traffic scenario.

[0206] The feature aggregation module is used to perform global feature aggregation on each first candidate decision behavior to obtain aggregated behavior features;

[0207] The feature fusion module is used to fuse aggregated behavioral features with first scene features to obtain joint features;

[0208] The feature encoding module is used to encode the joint features through a multilayer perceptron to obtain inverse reinforcement weights and imitation weights.

[0209] Based on the driving decision provided in this application, specific embodiments of the driving decision device are also provided accordingly.

[0210] like Figure 5 As shown, the driving decision device 500 provided in this application embodiment is applied to a vehicle. The vehicle is modeled with a decision model, which is trained based on the training method of the decision model provided in any of the above aspects.

[0211] The driving decision-making device 500 includes a feature acquisition module 510, a behavior prediction module 520, a probability acquisition module 530, and a vehicle control module 540.

[0212] Feature acquisition module 510 is used to acquire the second scene features of the second traffic scene in which the vehicle is currently located;

[0213] The behavior prediction module 520 is used to predict at least one second candidate decision behavior of the vehicle based on the features of the second scene.

[0214] The probability acquisition module 530 is used to input each second candidate decision behavior into the decision model to obtain the decision probability of each second candidate decision behavior;

[0215] The vehicle control module 540 is used to control the vehicle's movement based on the second best decision behavior with the highest decision probability among the second candidate decision behaviors.

[0216] This application provides a training method for the decision model. Correspondingly, this application also provides a specific embodiment of a training device for the decision model.

[0217] Figure 6 A schematic diagram of the hardware structure of a training device for a decision model provided in an embodiment of this application is shown.

[0218] The training device for the decision model may include a processor 601 and a memory 602 storing computer program instructions.

[0219] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0220] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal to an integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0221] The processor 601 reads and executes computer program instructions stored in the memory 602 to implement any of the decision model training methods in the above embodiments.

[0222] In one example, the training device for the decision model may further include a communication interface 603 and a bus 610. Wherein, as Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0223] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0224] Bus 610 includes hardware, software, or both, that couples components of a training device for a decision model together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0225] Furthermore, in conjunction with the training methods of the decision models in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the training methods of the decision models in the above embodiments.

[0226] In addition, in conjunction with the decision model training method in the above embodiments, this application embodiment can provide a computer program product to implement it. When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device executes the decision model training method provided by any aspect of the above-described embodiments of this application.

[0227] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0228] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0229] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0230] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0231] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for training a decision model, characterized in that, The method includes: Input each first candidate decision behavior corresponding to the first traffic scenario into the decision model to be trained to obtain the decision probability of each first candidate decision behavior; Based on the decision probability of each first candidate decision behavior, inverse reinforcement learning is performed on each first candidate decision behavior to obtain the inverse reinforcement learning loss value of the first traffic scenario; imitation learning is performed on the first best decision behavior with the highest decision probability among each first candidate decision behavior to obtain the imitation learning loss value of the first traffic scenario. Based on the inverse reinforcement weight and the imitation weight, the inverse reinforcement learning loss value and the imitation learning loss value are weighted and fused to obtain the weighted loss value of the first traffic scenario; Based on the weighted loss value, the model parameters of the decision model to be trained are optimized to obtain a trained decision model; the decision model is used to predict the optimal decision behavior of the vehicle based on the traffic scenario in which the vehicle is located.

2. The method according to claim 1, characterized in that, The step of performing inverse reinforcement learning on each of the first candidate decision behaviors to obtain the inverse reinforcement learning loss value of the first traffic scenario includes: Based on the decision probability of each first candidate decision behavior, a binary label corresponding to each first candidate decision behavior is obtained; the binary label corresponding to each first candidate decision behavior is used to characterize whether each first candidate decision behavior is the first optimal decision behavior. The decision probability of each first candidate decision behavior and the corresponding binary classification label are input into the inverse reinforcement learning loss function to obtain the inverse reinforcement learning loss value; the inverse reinforcement learning loss function is used to determine the degree of difference between the best decision behavior predicted by the decision model and the preset standard decision behavior.

3. The method according to claim 2, characterized in that, Before inputting the decision probability of each first candidate decision behavior and the corresponding binary classification label into the inverse reinforcement learning loss function to obtain the inverse reinforcement learning loss value, the method further includes: The scope of inverse reinforcement learning is determined based on the number of training batches for each traffic scenario and the number of candidate decision behaviors corresponding to each traffic scenario in each training batch. Based on the decision probability of each candidate decision behavior and the corresponding binary classification label, the individual inverse reinforcement loss value of each candidate decision behavior is determined by the binary cross-entropy function. Based on the individual inverse reinforcement loss value of each candidate decision behavior within the inverse reinforcement learning range, an inverse reinforcement learning loss function is constructed.

4. The method according to claim 1, characterized in that, The imitation learning of the first optimal decision behavior with the highest decision probability among all the first candidate decision behaviors to obtain the imitation learning loss value of the first traffic scenario includes: Obtain the learning dimension of the first optimal decision behavior imitation learning, and the decision behavior characteristics of the learning dimension; Based on the decision-making behavior characteristics, the imitation learning loss value is obtained through the imitation learning loss function corresponding to the learning dimension; the imitation learning loss function is used to determine the degree to which the optimal decision-making behavior predicted by the decision model conforms to the preset standard driving specifications in the learning dimension.

5. The method according to claim 4, characterized in that, The learning dimension includes a trajectory learning dimension, which is used to measure the distance deviation between the driving trajectory of the optimal decision-making behavior and the driving trajectory of the preset standard decision-making behavior. Before obtaining the imitation learning loss value based on the decision behavior characteristics using the imitation learning loss function corresponding to the learning dimension, the method further includes: The scope of imitation learning is determined based on the number of training batches in the traffic scenario. Based on the positional deviation between the optimal decision-making behavior and the preset standard decision-making behavior of each traffic scenario at each time point, the individual imitation loss value of each optimal decision-making behavior is determined. Based on the individual imitation loss values ​​of the best decision behaviors within the imitation learning range, an imitation learning loss function for the trajectory learning dimension is constructed.

6. The method according to claim 4, characterized in that, The learning dimension includes a collision learning dimension, which is used to measure the probability of collision between the agent and the nearest obstacle in the optimal decision-making behavior. Before obtaining the imitation learning loss value based on the decision behavior characteristics using the imitation learning loss function corresponding to the learning dimension, the method further includes: The scope of imitation learning is determined based on the number of training batches in the traffic scenario; Based on the difference between the target distance and the safe distance threshold of the agent at each time step for the optimal decision-making behavior in each traffic scenario, the individual imitation loss value of each optimal decision-making behavior is determined; the target distance is used to characterize the distance between the agent and the nearest obstacle. Based on the individual imitation loss values ​​of the best decision behaviors within the imitation learning range, the imitation learning loss function of the collision learning dimension is constructed.

7. The method according to any one of claims 1-6, characterized in that, Before the step of weightedly fusing the inverse reinforcement learning loss value and the imitation learning loss value based on inverse reinforcement weights and imitation weights to obtain the weighted loss value for the first traffic scenario, the method further includes: Obtain the first scene features of the first traffic scenario, and the first candidate decision behaviors of each of the first traffic scenarios; Global feature aggregation is performed on each of the first candidate decision behaviors to obtain aggregated behavior features; The aggregated behavioral features are fused with the first scene features to obtain joint features; The joint features are encoded using a multilayer perceptron to obtain inverse reinforcement weights and imitation weights.

8. A driving decision-making method, characterized in that, The method is applied to a vehicle, the vehicle being modeled with a decision model, the decision model being trained based on the method described in any one of claims 1-7; the method includes: Obtain the second scene features of the second traffic scene in which the vehicle is currently located; Based on the second scenario features, predict at least one second candidate decision behavior of the vehicle; Each of the second candidate decision behaviors is input into the decision model to obtain the decision probability of each of the second candidate decision behaviors; The vehicle's movement is controlled based on the second optimal decision action, which has the highest decision probability among all the second candidate decision actions.

9. A training device for a decision-making model, characterized in that, The device includes: The probability determination module is used to input each first candidate decision behavior corresponding to the first traffic scenario into the decision model to be trained, and obtain the decision probability of each first candidate decision behavior. The behavior learning module is used to perform inverse reinforcement learning on each of the first candidate decision behaviors based on the decision probability of each first candidate decision behavior to obtain the inverse reinforcement learning loss value of the first traffic scenario; and to perform imitation learning on the first best decision behavior with the highest decision probability among the first candidate decision behaviors to obtain the imitation learning loss value of the first traffic scenario. The loss fusion module is used to perform weighted fusion of the inverse reinforcement learning loss value and the imitation learning loss value based on the inverse reinforcement weight and the imitation weight to obtain the weighted loss value of the first traffic scenario. The model optimization module is used to optimize the model parameters of the decision model to be trained based on the weighted loss value, so as to obtain a trained decision model; the decision model is used to predict the optimal decision behavior of the vehicle based on the traffic scenario in which the vehicle is located.

10. A driving decision-making device, characterized in that, The device is applied to a vehicle, the vehicle being modeled with a decision model, the decision model being trained based on the method described in any one of claims 1-7; the method includes: The feature acquisition module is used to acquire the second scene features of the second traffic scene in which the vehicle is currently located; The behavior prediction module is used to predict at least one second candidate decision behavior of the vehicle based on the second scene features; The probability acquisition module is used to input each of the second candidate decision behaviors into the decision model to obtain the decision probability of each of the second candidate decision behaviors; The vehicle control module is used to control the driving of the vehicle based on the second optimal decision behavior with the highest decision probability among each of the second candidate decision behaviors.

Images