A large language model driving action arbitration method based on road traffic safety law imitation learning

Abstract

This invention discloses a driving action arbitration method based on a large language model using imitation learning of road traffic safety laws. The method first uses a camera to identify targets such as vehicles, pedestrians, and cyclists, and organizes them into a structured scene. Then, it transforms road traffic safety law clauses into executable rules, uses these rules to generate training samples and a dual-head neural network, and outputs the acceptance probability and the probability of alternative actions. Simple safety arbitration can only eliminate actions deemed dangerous; it cannot proactively propose the next action based on the current road conditions. For example, after an action is prohibited, the vehicle still needs to decide whether to decelerate, stop, or turn; even when multiple actions are legal, the most suitable action for the current scenario must be selected. Therefore, this method first uses a large language model to understand the scene and propose candidate high-level actions such as cruising, decelerating, stopping, accelerating, turning left, or turning right. Then, a dual-head neural network determines whether to retain the action; if safe, it is retained; if unsafe, it is replaced with a safe action, and finally, the action is handed over to the trajectory module and PID controller for execution.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving decision-making technology, specifically relating to a driving action arbitration method based on a large language model of road traffic safety law imitation learning. Background Technology

[0002] Large language models can read road conditions organized according to fixed fields and propose the vehicle's overall driving intention for the next step from a finite set of actions. For example, when there is a slow vehicle ahead and the left lane is available, a large language model can suggest slowing down or turning left. However, large language models are only suitable for proposing candidate high-level actions and should not directly control the steering wheel, accelerator, and brakes, as their output may not comply with the relevant provisions of the "Road Traffic Safety Law of the People's Republic of China".

[0003] Simple safety arbitration typically only determines whether a given action is prohibited or permitted, which still presents two problems: first, after an action is prohibited, the vehicle still needs to decide whether to slow down, stop, or turn; second, when multiple actions comply with regulations, the vehicle still needs to choose the more appropriate action based on road conditions. Therefore, using only hard constraint rules or threshold filtering cannot simultaneously achieve scenario understanding, candidate high-level action generation, and selection of safe alternative actions.

[0004] To address the aforementioned issues, this invention employs a method of first proposing suggestions and then conducting safety arbitration: First, a large language model proposes candidate high-level actions based on the current road conditions; then, a dual-head neural network trained according to the Road Traffic Safety Law determines whether the action should be retained; if a replacement is needed, the same network provides a safe alternative action. The large language model does not directly control the vehicle; ultimately, the high-level actions must be arbitrated. Summary of the Invention

[0005] This invention provides a driving action arbitration method based on a large language model of road traffic safety law imitation learning. The method first converts legal provisions into computer-executable rules, then automatically generates training labels using these rules, and trains a dual-head neural network with an acceptance probability head and an action probability head. During runtime, the acceptance probability head determines whether to retain the large language model's suggestion, and the action probability head selects a safe action when replacement is needed.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A driving action arbitration method based on a large language model of road traffic safety law imitation learning includes the following steps:

[0008] Step S1: Acquire an image of the road ahead and use a real-time object detector to identify each target in the road ahead. The targets are vehicles, pedestrians, and cyclists identified in the image. A priority score is calculated for each target using the following formula, and the top K priority targets with the highest scores are selected, where K is the preset number of priority targets. These top K priority targets are then combined with the vehicle's speed, lane, and traffic signal status to form a structured scene. The structured scenario refers to current road condition data organized according to fixed fields:

[0009]

[0010] Indicate target Priority score, Indicate target Basic risk classification, Indicate target The distance levels that participate in the distance bonus determination include close range and medium range. Indicates close proximity. Indicates medium distance; Indicate target The directions involved in the direction bonus judgment include directly in front, left front, and right front. Indicates directly in front. Indicates the left front side. Indicates the right front side; Indicate target The width of the detection frame; Indicate target The height of the detection frame, This indicates taking the smaller of the two values; Indicate target Category; This represents a set of vulnerable traffic participant categories, consisting of pedestrian and cyclist categories. , This is the distance bonus factor. , This is the direction addition factor. Addition factor for vulnerable traffic participants; This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise.

[0011] Step S2: Convert Articles 22, 35, 38, 42, 43, 47, and 67 of the "Road Traffic Safety Law of the People's Republic of China" into a set of safety rules that can be executed by a computer, denoted as... , to These correspond to the seven safety rules formed by Articles 22, 35, 38, 42, 43, 47, and 67 of the Road Traffic Safety Law of the People's Republic of China. The set of safety rules is used to determine whether candidate high-level actions comply with regulations in the current scenario and to provide safe alternative actions when they do not. Candidate high-level actions represent the vehicle's overall driving intention for the next step, not steering wheel angle, throttle opening, or brake pressure. Six sets of optional high-level actions for the vehicle are defined: Among them, the action set , , , , , , Indicates cruising. Indicates deceleration. Indicates stopping the car. Indicates acceleration. Indicates a left turn. Indicates a right turn; for each structured scenario and the actions of each candidate for senior management Through regulatory expert functions This refers to a program that automatically makes judgments based on a set of security rules, generating label pairs for training the model. in This indicates acceptance of the label. This indicates that the actions of the candidate high-level executives do not violate the set of security rules. The rules in This indicates that the actions of the candidate high-level executives violated a set of security rules. The rules in; This indicates that the actions of the candidate high-level executives violated a set of security rules. Time by set of security rules The derived safe alternative actions, and .

[0012] Step S3: Based on the structured scenario and the actions of the candidates for senior management Constructing eigenvectors:

[0013]

[0014] The feature vector is then input into a dual-head neural network. , This represents all trainable parameters in a dual-headed neural network; a dual-headed neural network is a decision model with two outputs: one output determines whether to retain a candidate high-level action, and the other output provides a safe action when a candidate high-level action needs to be replaced.

[0015]

[0016]

[0017]

[0018] in, This represents the hidden features of the shared backbone output; This represents a function that converts the calculation results of the alternative action output into an action probability distribution; This is the Sigmoid function, used to convert the judgment result into an acceptance probability between 0 and 1; This represents the feature matrix of the first K key targets selected in step S1, where K is the preset number of key targets in step S1. This represents a summary feature of the scene, consisting of the vehicle's speed, the lane it is in, and the traffic signal status. This means expanding the matrix into vectors one by one. This indicates the set of actions. The actions in the code are converted into one-hot codes that are easy for the model to process; This represents the basic computational component shared by both outputs, i.e., the weight matrix that shares the main structure. Representing the eigenvector ; This indicates that the weights and biases at the output end are accepted for judgment. Indicates transpose; This represents the weights and biases at the output of the alternative action; This represents the layer normalization operator used to stabilize the data range; Represents the linear rectification activation function. Represents the random deactivation operator. This represents a probability distribution consisting of the probabilities of six high-level actions, where the six probabilities correspond to... , , , , and All six probabilities are not less than zero, and the sum of the six probabilities is one; use regulatory expert functions. Automatically generated training samples Jointly train the dual-head neural network according to the following formula. :

[0019] in, This represents the joint training loss of a dual-head neural network; and This represents the loss weights at the two outputs, and and All are greater than zero; This indicates the binary cross-entropy loss function used for the acceptance judgment; This represents the multi-class cross-entropy loss function used for alternative action selection.

[0020] Step S4: During vehicle operation, the large language model is based on the same structured scene. From the action set The process proposes a candidate high-level action, denoted as... The large language model is only responsible for making suggestions and does not directly control the vehicle; it will select high-level candidates for actions. After being concatenated with the current scene features, it is fed into the dual-head neural network trained in step S3. Perform a forward calculation, which completes a quick judgment, to obtain the final high-level action after arbitration:

[0021]

[0022] in, To accept the threshold, and The acceptance threshold is a probability threshold used to determine whether a candidate high-level action should be retained. When a candidate high-level action passes arbitration, the action proposed by the large language model is retained. When a candidate high-level action fails arbitration, the action with the highest probability in the alternative action output is selected as the safe alternative action. The action with the highest probability refers to the action corresponding to the highest probability value in the action probability distribution, i.e. ; The final high-level actions are then fed into the low-level trajectory generation module and PID controller. The low-level trajectory generation module generates the vehicle's driving trajectory based on the final high-level actions, and the PID controller adjusts the vehicle's steering, throttle, and braking according to the driving trajectory. As the vehicle's only source of high-level actions. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the overall process of the driving action arbitration method based on a large language model using road traffic safety law imitation learning, as described in this invention.

[0024] Figure 2 This is a comparison chart of the arbitration performance of the method of the present invention and that without an arbitration baseline on the CARLA simulation platform test set. Detailed Implementation

[0025] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0026] See Figure 1 The method of this invention is divided into a training phase and an operation phase. The purpose of the training phase is to allow the computer to learn in advance whether driving actions can be retained in a certain road scenario; if not, what safe actions should be replaced; the purpose of the operation phase is to quickly arbitrate the driving actions proposed by the large language model when the vehicle is actually driving or in simulation.

[0027] like Figure 1 As shown, the method of the present invention includes a training phase and an operation phase. Figure 1 The left side represents the training phase, which is used to pre-train the safety arbitration model. Specifically, firstly, camera footage and vehicle status are processed to obtain road scene data containing information such as vehicles, pedestrians, cyclists, vehicle speed, lanes, and traffic signals. Then, relevant clauses of the Road Traffic Safety Law are converted into computer-executable safety rules. Next, optional high-level actions for the vehicle are defined, including cruising, deceleration, stopping, acceleration, left turn, and right turn. Then, training samples are automatically generated according to the safety rules, marking whether candidate high-level actions can be retained and what safe action should be replaced if they cannot be retained. Subsequently, the road scene and candidate high-level actions are processed into digital features that the model can read. Finally, these training samples are used to train a dual-head neural network, enabling it to output the acceptance probability of candidate high-level actions and the probability of safe alternative actions. Figure 1 The right side represents the execution phase, which arbitrates the candidate high-level actions proposed in real time by the large language model. Specifically, during vehicle operation, the camera first identifies road conditions in real time, forming a structured scene with the same format as in the training phase. The large language model proposes a candidate high-level action from the action set based on the structured scenario. The trained dual-head neural network serves as a safety arbitration model to check whether the candidate high-level action complies with regulatory requirements and the safety requirements of the current scenario. If the candidate high-level action is safe, the action proposed by the large language model is retained. If the candidate high-level action is unsafe or non-compliant, it is replaced with a safe action. Finally, the final high-level action of the vehicle is output and sent to the underlying trajectory generation module and PID controller for execution.

[0028] The large language model in this invention does not directly control the vehicle's steering wheel, accelerator, or brakes. Instead, it proposes a candidate high-level action based solely on the current road conditions, such as cruising, decelerating, stopping, accelerating, turning left, or turning right. This candidate high-level action must be arbitrated by a dual-head neural network before being output as the vehicle's final high-level action to the underlying trajectory generation module and PID controller.

[0029] The specific implementation process of step S1 is as follows: The image in front of the vehicle is acquired by the vehicle-mounted forward-looking camera, and the image is input into a real-time target detector, such as YOLOv7, YOLOv8, or other detectors capable of detecting road targets in real time. The detector is used to identify the position, category, and detection box size of each road participant in the image; road participants include vehicles, pedestrians, cyclists, etc.

[0030] For each detected target Extract its basic risk score Distance level ,direction Detection frame width Detection frame height and categories The distance levels involved in determining the distance bonus include close range and medium range. Indicates close proximity. Indicates medium distance; the directions involved in the directional bonus judgment include directly in front, left front, and right front. Indicates directly in front. Indicates the left front side. Indicates the right front side; detection box width and detection frame height Used to reflect the area occupied by the target in the image; category The output of the detector, such as vehicles, pedestrians, or cyclists.

[0031] Calculate the priority score for each objective using the following formula. :

[0032]

[0033] in, Indicate target Priority score, Indicate target Basic risk classification, Indicate target The distance levels that participate in the distance bonus determination include close range and medium range. Indicates close proximity. Indicates medium distance; Indicate target The directions involved in the direction bonus judgment include directly in front, left front, and right front. Indicates directly in front. Indicates the left front side. Indicates the right front side; Indicate target The width of the detection frame; Indicate target The height of the detection frame, This indicates taking the smaller of the two values; Indicate target Category; This represents a set of vulnerable traffic participant categories, consisting of pedestrian and cyclist categories. , This is the distance bonus factor. , This is the direction addition factor. Addition factor for vulnerable traffic participants; This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise.

[0034] The above formula means that if a target is close to the vehicle, its priority score is increased; if the target is directly in front of the vehicle or to the left or right front, its priority score is increased; if the target's detection box in the image is large, it means that the target may be close to the vehicle or have a significant impact, so its priority score is also increased; if the target is a pedestrian, cyclist or other vulnerable road user, its priority score is further increased.

[0035] After calculating the priority scores of all objectives Then, the scores are sorted from highest to lowest, and the top K key targets with the highest scores are selected. K is the preset number of candidates, which can be set according to the vehicle's perception range, computing power, and scene complexity.

[0036] The first K key targets, along with information such as vehicle speed, lane location, lane markings, and traffic signal status, are organized into a structured scenario. The structured scenario It's current road condition data organized according to fixed fields. In other words, it's a structured scenario. It is not the original image, but the image and vehicle status converted into standardized data that can be read by computers, large language models and dual-head neural networks.

[0037] The specific implementation process of step S2 is as follows: Articles 22, 35, 38, 42, 43, 47, and 67 of the "Road Traffic Safety Law of the People's Republic of China" (a total of seven articles) are converted one by one into a set of safety rules that can be executed by a computer, denoted as: ,in, to These correspond to the seven safety rules formed by Articles 22, 35, 38, 42, 43, 47, and 67 of the "Road Traffic Safety Law of the People's Republic of China"; each rule can be understood as a Boolean decision function, with the input being a structured scenario. and the actions of the candidates for senior management The output is the action. In the scene The result of the judgment on whether the corresponding clauses are violated.

[0038] Define a finite set containing six high-level actions. ,in Indicates cruising. Indicates deceleration. Indicates stopping the car. Indicates acceleration. Indicates a left turn. This indicates a right turn. The aforementioned higher-level actions indicate the vehicle's overall driving intention for the next step, not lower-level control parameters such as steering wheel angle, throttle opening, or brake pressure. For example, It's a high-level action, but the specific amount of braking is determined by the underlying PID controller.

[0039] For each structured scenario Actions with each candidate for senior management Through regulatory expert functions Automatically generate imitation learning label pairs:

[0040]

[0041] in, This represents a regulatory expert function, i.e., a function based on a set of safety rules. A program that performs automatic judgment; This indicates acceptance of the label. This represents the action label, which is the safe alternative action derived from the set of safety rules R when the candidate high-level action is not accepted.

[0042] When the candidate for high-level position moves Not violating the set of rules Any rule, This indicates that the candidate's action is acceptable; when the candidate's action... set of rules violated When at least one of the rules is in This indicates that the candidate high-level action cannot be executed directly. At this point, the regulatory expert function... It will be based on the current structured scenario From the action set A safe alternative action is derived from the derivation. .when hour, The value can be the original candidate high-level action. ;when hour, Choose a safer alternative action, such as or .

[0043] For example, in the current structured scenario In the middle, if there are pedestrians or vehicles nearby, and the candidate high-level actions... for Then the regulatory expert function It can be determined that the candidate's actions do not comply with regulatory requirements and safety principles, and the output is... and will Set as or For example, given that the roads are currently clear, lanes are functioning normally, and traffic signals permit passage, if the candidate high-level officials take action... for Then the regulatory expert function It can output This indicates that the action can be retained.

[0044] Using the above method, a large number of imitation learning training samples can be automatically generated. Each sample includes a structured scene. Actions of candidates for senior management Accepting tags and action tags In this way, the subsequent dual-head neural network does not directly memorize the legal provisions, but learns the judgment results of the legal rules in different road scenarios through a large number of samples.

[0045] The specific implementation process of step S3 is as follows: for each structured scenario Actions with the candidate for senior management This is converted into a feature vector composed of numbers, enabling the computer model to process it. The feature vector is constructed using the following formula. :

[0046]

[0047] in, Represents a structured scene Actions with the candidate for senior management The input feature vector formed; The feature matrix of the first K key targets selected in step S1, where K is the number of key targets preset in step S1. Each target feature includes basic risk score, distance level, direction, detection box width, detection box height and category. It represents a summary of features related to the vehicle's status, lane, and traffic signals, such as the vehicle's speed, lane information, and traffic signal information. This represents a vectorization operator used to expand a matrix into vectors sequentially. Represents a set of actions One-hot encoding operator on the top is used to encode candidate high-level actions. Mapped to Unique heat vector.

[0048] eigenvectors Input dual-head neural network Dual-head neural network It has a shared backbone and two outputs. The shared backbone is used to extract common features of the scene and actions; the first output is an acceptance probability header, used to determine whether candidate high-level actions can be retained; the second output is an action probability header, used to provide safe alternative actions when candidate high-level actions need to be replaced. The shared backbone outputs the hidden representation as follows: :

[0049]

[0050] in, This represents the hidden features of the shared backbone output. , This indicates the basic computational component shared by both outputs. Representing the eigenvector That is, the weight matrix that shares the backbone; The representation layer normalization operator is used to stabilize the data range; This represents the linear rectified activation function, used to preserve effective features; This is a random deactivation operator used to reduce the risk of the model over-memorizing training samples.

[0051] The acceptance probability head outputs the acceptance probability according to the formula. :

[0052]

[0053] in , These represent the weights and biases of the input and output terminals, respectively. Indicates transpose; This is the Sigmoid function, used to convert the judgment result into an acceptance probability between 0 and 1; The closer it is to 1, the more likely the dual-head neural network is to retain the candidate high-level action; The closer the value is to 0, the more likely the candidate's actions are to be unsafe or non-compliant.

[0054] The action probability head outputs the action probability distribution using the formula. :

[0055]

[0056] in , These represent the weights and biases at the output of the alternative action, respectively. Used to convert the output into probabilities of each action; This represents a probability distribution consisting of the probabilities of six high-level actions, where the six probabilities correspond to... , , , , and All six probabilities are not less than zero, and the sum of the six probabilities is one.

[0057] Using regulatory expert functions Automatically generated sample sets Jointly supervised training of dual-head neural networks using the following formula :

[0058]

[0059] in, This represents the joint training loss of a dual-head neural network; , A value greater than 0 indicates the loss weights at both outputs; This indicates the binary cross-entropy loss function used for the acceptance judgment; This represents the multi-class cross-entropy loss function used for alternative action selection; This indicates the accept label generated by the regulatory expert function; This represents the action labels generated by the regulatory expert function. Training uses the Adam optimizer, resulting in a dual-head neural network for the runtime phase. .

[0060] The specific implementation process of step S4 is as follows: When the vehicle is running, real-time target detection is performed on the current forward view image according to the method described in step S1, the top K key targets are selected, and together with the vehicle speed, lane, and traffic signal status, a current structured scene is formed. .

[0061] Large language models based on the same structured scenario From the action set The process proposes a candidate high-level action, denoted as... The candidate's actions can only achieve... , , , , , One of them. The large language model is only responsible for proposing candidate high-level actions and does not directly control the vehicle.

[0062] Actions of the candidates for senior management Compared with the current structured scenario The vectors are concatenated into a feature vector as described in step S3, and then fed into the dual-head neural network trained in step S3. Perform a forward computation to obtain the acceptance probability. and action probability distribution .

[0063] The final high-level action after arbitration is obtained by following the formula. :

[0064]

[0065] in The acceptance threshold; This represents the candidate high-level actions proposed by the large language model; This represents the probability that the dual-head neural network accepts the candidate high-level action; This represents the probability distribution of safe alternative actions output by the action probability head; This indicates the final high-level action following the arbitration.

[0066] when This indicates that the dual-head neural network determines that the candidate high-level actions proposed by the large language model comply with regulatory requirements and the current scenario's safety requirements; therefore, the candidate high-level actions proposed by the large language model are retained. .when If the candidate's actions pose a risk of insecurity or non-compliance, then the original recommendation from the large language model is not followed; instead, the action probability distribution is considered. The action with the highest probability is selected as the safe alternative action.

[0067] Final high-level actions following arbitration The data is fed into the underlying trajectory generation module and the PID controller. The underlying trajectory generation module then determines the final high-level motion. A driving trajectory is generated, and the PID controller adjusts the vehicle's steering, throttle, and braking based on this trajectory. Therefore, the only source of the vehicle's high-level actions is the final high-level action arbitrated by a dual-head neural network. Instead of unchecked candidate high-level actions from large language models . Figure 1 The final vehicle action is the final high-level action after arbitration; this high-level action is then sent to the low-level trajectory generation module to generate the trajectory, and is converted into steering, throttle and braking control quantities by the PID controller.

[0068] To verify the effectiveness of the method of this invention, a test set was collected on the CARLA simulation platform. The test set contained 5406 samples, covering eight typical driving scenarios, including cruising, following, braking, steering, and pedestrian risk. During the test, the large language model first analyzed the structured scenario... Output candidate high-level actions The dual-head neural network obtained by training this invention Conduct arbitration and output the final high-level actions following the arbitration. .

[0069] The method of this invention is compared with the baseline of the highD open-source dataset. The highD open-source dataset baseline refers to a comparative baseline formed based on publicly available natural driving trajectory samples from the highD dataset. This baseline reflects the action distribution characteristics in publicly available real-world driving data, but does not include the dual-headed safety arbitration module obtained through imitation learning based on the Road Traffic Safety Law in this invention. This baseline is used to illustrate that the action reference results formed solely based on publicly available driving data still have room for improvement in terms of regulatory consistency, the ability to intercept unsafe actions, and the ability to retain legal actions.

[0070] like Figure 2 As shown, the evaluation metrics include regulatory compliance rate, unsafe action interception rate, and legal action retention rate. Regulatory compliance rate represents the proportion of final output actions that comply with regulatory rules; unsafe action interception rate represents the proportion of times the method can successfully prevent a candidate high-level action from being directly executed when the candidate high-level action has unsafe or non-compliant risks; and legal action retention rate represents the proportion of times the method can retain a candidate high-level action when it complies with regulatory and scenario security requirements. Figure 2 In this paper, the baseline representation of the highD open-source dataset represents the comparison results formed based on the publicly available natural driving trajectory samples of the highD dataset; the method of this invention represents that the large language model first proposes candidate high-level actions, and then the dual-head neural network performs safety arbitration to output the final high-level action.

[0071] In the CARLA simulation platform test set, the regulatory compliance rate of the highD open-source dataset baseline was 81.72%, the unsafe action interception rate was 80.36%, and the legal action retention rate was 82.15%. The regulatory compliance rate of the method described in this invention was 88.64%, the unsafe action interception rate was 86.91%, and the legal action retention rate was 89.27%. This indicates that, compared to the highD open-source dataset baseline based on publicly available natural driving data, the method described in this invention can further improve the regulatory compliance rate of the final output actions and more effectively intercept unsafe or non-compliant actions while maintaining a high retention rate of legal actions. In other words, the method described in this invention does not simply prohibit the output actions of large language models, but rather, while retaining legal candidate actions, it safely replaces risky candidate actions, thereby improving the safety and regulatory consistency of autonomous driving decisions.

[0072] The above results demonstrate that the method of this invention divides the work between the large language model and the security arbitration model: the large language model is responsible for determining the structured scenario. A candidate high-level action is proposed, and a dual-head neural network is responsible for arbitrating the safety of the candidate high-level action based on the judgment ability learned through imitation of the Road Traffic Safety Law. When the candidate high-level action meets the regulatory requirements and the safety requirements of the current scenario, the method of this invention retains the action; when the candidate high-level action poses an unsafe or non-compliant risk, the method of this invention replaces it with a safe alternative action. Finally, only the final high-level action after arbitration is sent to the underlying trajectory generation module and PID controller for execution, thereby improving the regulatory consistency and driving safety of autonomous driving decisions.

[0073] The method of this invention does not rely solely on safety arbitration to directly accept or reject vehicle actions. This is because although safety arbitration can determine whether a candidate action complies with regulatory requirements, it usually only provides a "permitted" or "prohibited" result, making it difficult to proactively integrate information such as vehicles ahead, pedestrians, cyclists, vehicle speed, lanes, and traffic signals to propose appropriate driving intentions in complex road scenarios. This invention first uses a large language model to generate candidate high-level actions, and then uses a safety arbitration model trained based on the Road Traffic Safety Law to check and replace them as necessary, thereby balancing the ability to suggest actions in complex scenarios with regulatory safety.

Claims

1. A driving action arbitration method based on a large language model of road traffic safety law imitation learning, characterized in that, Includes the following steps: Step S1: Acquire an image of the road ahead and use a real-time object detector to identify each target in the road ahead. The targets are vehicles, pedestrians, and cyclists identified in the image. A priority score is calculated for each target using the following formula, and the top K priority targets with the highest scores are selected, where K is the preset number of priority targets. These top K priority targets are then combined with the vehicle's speed, lane information, and traffic signal status to form a structured scene. Structured scenarios This is current road condition data organized according to fixed fields: in, Indicate target Priority score, Indicate target Basic risk classification, Indicate target The distance levels that participate in the distance bonus determination include close range and medium range. Indicates close proximity. Indicates medium distance; Indicate target The directions involved in the direction bonus judgment include directly in front, left front, and right front. Indicates directly in front. Indicates the left front side. Indicates the right front side; Indicate target The width of the detection frame; Indicate target The height of the detection frame, This indicates taking the smaller of the two values; Indicate target Category; This represents a set of vulnerable traffic participant categories, consisting of pedestrian and cyclist categories. , This is the distance bonus factor. , This is the direction addition factor. Addition factor for vulnerable traffic participants; This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Step S2: Convert Articles 22, 35, 38, 42, 43, 47, and 67 of the "Road Traffic Safety Law of the People's Republic of China" into a set of computer-executable safety rules, denoted as... , to These correspond to the seven safety rules formed by Articles 22, 35, 38, 42, 43, 47, and 67 of the "Road Traffic Safety Law of the People's Republic of China." The set of safety rules is used to determine whether candidate high-level actions comply with regulations in the current scenario and to provide safe alternative actions when they do not. Candidate high-level actions represent the vehicle's overall driving intention for the next step, not steering wheel angle, throttle opening, or brake pressure. Six sets of optional high-level actions for the vehicle are defined: Among them, the action set , , , , , , Indicates cruising. Indicates deceleration. Indicates stopping the car. Indicates acceleration. Indicates a left turn. Indicates a right turn; for each structured scenario and the actions of each candidate for senior management Through regulatory expert functions This refers to a program that automatically makes judgments based on a set of security rules, generating label pairs for training the model. in This indicates acceptance of the label. This indicates that the actions of the candidate high-level executives do not violate the set of security rules. The rules in This indicates that the actions of the candidate high-level executives violated a set of security rules. The rules in; This indicates that the actions of the candidate high-level executives violated a set of security rules. Time by set of security rules The derived safe alternative actions, and ; Step S3: Based on the structured scenario and the actions of the candidates for senior management Constructing eigenvectors: The feature vector is then input into a dual-head neural network. , This represents all trainable parameters in a dual-headed neural network; a dual-headed neural network is a decision model with two outputs: one output determines whether to retain a candidate high-level action, and the other output provides a safe action when a candidate high-level action needs to be replaced. in, This represents the hidden features of the shared backbone output; This represents a function that converts the calculation results of the alternative action output into an action probability distribution; This is the Sigmoid function, used to convert the judgment result into an acceptance probability between 0 and 1; This represents the feature matrix of the first K key targets selected in step S1, where K is the preset number of key targets in step S1. This represents a summary feature of the scene, consisting of the vehicle's speed, the lane it is in, and the traffic signal status. This means expanding the matrix into vectors one by one. This indicates the set of actions. The actions in the code are converted into one-hot codes that are easy for the model to process; This represents the basic computational component shared by both outputs, i.e., the weight matrix that shares the main structure. Representing the eigenvector , This indicates that the weights and biases at the output end are accepted for judgment. Indicates transpose; This represents the weights and biases at the output of the alternative action; This represents the layer normalization operator used to stabilize the data range; Represents the linear rectified activation function. Represents the random deactivation operator. This represents a probability distribution consisting of the probabilities of six high-level actions, where the six probabilities correspond to... , , , , and All six probabilities are not less than zero, and the sum of the six probabilities is one; use regulatory expert functions. Automatically generated training samples Jointly train the dual-head neural network according to the following formula. : in, This represents the joint training loss of a dual-head neural network; and This represents the loss weights at the two outputs, and and All are greater than zero; This indicates the binary cross-entropy loss function used for the acceptance judgment; This represents the multi-class cross-entropy loss function used for alternative action selection; This indicates the accept label generated by the regulatory expert function; This indicates the action label generated by the regulatory expert function; Step S4: During vehicle operation, the large language model is based on the same structured scene. From the action set The process proposes a candidate high-level action, denoted as... The large language model is only responsible for making suggestions and does not directly control the vehicle; it will select high-level candidates for actions. After being concatenated with the current scene features, it is fed into the dual-head neural network trained in step S3. Perform a forward calculation to obtain the final high-level action after arbitration: in, To accept the threshold, and The acceptance threshold is a probability threshold used to determine whether a candidate high-level action should be retained. When a candidate high-level action passes arbitration, the action proposed by the large language model is retained. When a candidate high-level action fails arbitration, the action with the highest probability in the alternative action output is selected as the safe alternative action. The action with the highest probability refers to the action corresponding to the highest probability value in the action probability distribution, i.e. ; The final high-level actions are then fed into the low-level trajectory generation module and PID controller. The low-level trajectory generation module generates the vehicle's driving trajectory based on the final high-level actions, and the PID controller adjusts the vehicle's steering, throttle, and braking according to the driving trajectory. As the vehicle's only source of high-level actions.

Images