End-to-end interpretable automatic driving control method and system based on large language model

Abstract

The invention discloses an end-to-end interpretable automatic driving control method and system based on a large language model, and the method comprises the steps: taking a video frame captured by a foresight camera as an input, predicting a control signal at a next moment, and outputting a text description of a current behavior of a vehicle; wide world knowledge of a language model is utilized to endow automatic driving with stronger logical reasoning ability; a new control signal prediction head is designed, different from a previous simple prediction head, cross covariance pooling and normalization operations are introduced to aggregate segmented words and explore richer features, and prediction of control signals is facilitated. The automatic driving system disclosed by the invention not only can provide accurate control signal prediction, but also can provide explanation based on a natural language, so that the transparency and the interpretability of the system are enhanced, and the improvement of the safety of automatic driving and the credibility of a user is facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically relates to an end-to-end interpretable autonomous driving control method and system based on a large language model. Background Art

[0002] The core goal of an autonomous driving system is to achieve highly accurate perception of the surrounding environment, make real-time decisions, and perform highly reliable vehicle control without human intervention. In addition, the system should ensure safety and comfort throughout the journey, providing a seamless intelligent travel experience. To achieve these goals, autonomous driving systems must integrate a variety of advanced technologies to cope with complex traffic scenarios and unforeseen external environmental factors. With the rapid development of deep learning technology, autonomous driving model algorithms have also benefited from it. Classic autonomous driving algorithms usually adopt a modular-based idea, including a series of tasks such as perception, prediction, decision-making, and planning and control, and deploy independent models for each task. Although this design can simplify the difficulty of research and development, the downstream planning and control module cannot access the original sensor data. The accumulated errors generated only based on the perception or decision-making module will seriously affect the planning and control results, and even cause safety problems. In addition, the decoupling of each task into independent modules may cause information loss and cause unnecessary losses.

[0003] In recent years, some methods have proposed to train networks in an end-to-end manner. End-to-end autonomous driving solutions take raw sensor data as input, optimize all features within a network, and directly generate planned routes or output control results. This greatly improves the defects of modular design. However, the current challenge facing end-to-end autonomous driving models is that these systems are like a "black box" in the decision-making and planning process, without interpretability and reasoning capabilities. Especially when the model faces unsolvable situations, it becomes particularly difficult to understand the system's erroneous decision-making process. If the fault cannot be corrected, it may affect the safety and controllability of the vehicle. Therefore, making the system interpretable is a top priority. Some previous works have explored methods to generate maps, 3D detections and their future trajectories or visualize causal attention in the intermediate layer to enhance the interpretability of the system. However, these methods are difficult for users to understand.

[0004] The emergence of large language models (LLMs) provides a new idea for explainable autonomous driving. Since LLMs have extensive world knowledge, they can provide powerful logical reasoning for autonomous driving. Therefore, autonomous driving systems that incorporate language models can better analyze and generate vehicle control signals. However, existing language model-based autonomous driving methods rarely use raw sensor data to directly output control results. Most of them use language models to enhance scene understanding and planning capabilities. In addition, even if there is an end-to-end method from sensor to control signal, the vehicle control signal is only output through a simple linear layer or a fully connected layer, and there is not much exploration of the prediction head.

[0005] Therefore, the present invention proposes an end-to-end interpretability method and system for a large language model for autonomous driving, whose input is a video frame captured by a forward-looking camera, and whose output is a predicted control signal at the next moment. At the same time, the system also provides the driving state of the vehicle at the current moment (such as the vehicle is stopping slowly) and the reason for the behavior (such as because there is a car driving slowly in front), so as to enhance the interpretability of the system. In addition, two different prediction heads are designed for control signals and behavior reasoning respectively. In particular, for the control signal prediction head, covariance pooling and normalization operations are introduced, which can effectively aggregate the word segmentation output by the language model and explore more useful features. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides an end-to-end interpretable autonomous driving control method and system based on a large language model, which uses the extensive world knowledge of the language model to give autonomous driving a more powerful logical reasoning ability. The input of the present invention is a video frame captured by a forward-looking camera, and the output is a predicted control signal at the next moment. At the same time, the driving state of the vehicle at the current moment (such as the vehicle is stopping slowly) and the reason for the behavior (such as because there is a car driving slowly in front) are also provided to enhance the interpretability of the system. In addition, two different prediction heads are designed for control signals and behavior reasoning respectively. In particular, for the control signal prediction head, covariance pooling and normalization operations are introduced, which can effectively aggregate the word segmentation output by the language model and explore more useful features.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] The present invention designs an end-to-end interpretable autonomous driving control method and system based on a large language model (hereinafter referred to as the autonomous driving system or system). The control method comprises the following steps:

[0009] Step 1: The video data captured by the single front-view camera is used as the input of the autonomous driving system and is uniformly sampled into video frames;

[0010] Step 2: Use the spatiotemporal adapter as a visual encoder to extract features and convert video frames into word segmentation;

[0011] Step 3: Send the segmented words into the large language model decoder to obtain the predicted new segmented words;

[0012] Step 4: Send the predicted new word segmentation to the control signal prediction head and the text reasoning head respectively. The control signal prediction head outputs the control signal at the next moment, and the text reasoning head outputs the vehicle driving status and reason explanation.

[0013] Furthermore, for the input video data in step 1, a uniform sampling strategy with a fixed time interval is adopted to extract a certain number of video frames (for example, 8 frames, 16 frames or 32 frames) as an effective information source for the input of the autonomous driving system.

[0014] The sampled video frame can be represented as a four-dimensional tensor, denoted by V∈R T×H×W×C , where T is the number of sampled frames, C is the number of channels (RGB three channels), H and W represent the height and width of the image respectively.

[0015] Furthermore, the core design of the spatiotemporal adapter is based on the basic structure of the visual Transformer, but additional spatiotemporal modeling operations are introduced, which consist of upsampling, deep convolution, and downsampling operations. The pre-trained knowledge of the model can be extracted and utilized, which is beneficial for video understanding.

[0016] The specific process is:

[0017] Input feature V∈R T×H×W×C First, a slicing operation is performed to divide the video frame into several small blocks. Then, these small blocks are flattened, that is, these small blocks are converted into one-dimensional vectors. Then, the obtained vectors are transformed into the input dimensional space of the model using a projection operation. Finally, each frame of the video is converted into a feature vector, and the feature X∈R is obtained. N ×d , where N is the number of slices and d represents the dimension after the change.

[0018] The feature X is fed into the spatiotemporal adapter, which introduces a spatiotemporal operator implemented by a deep convolution. It can be expressed as:

[0019] Z(X)=X+f(DWConv(XW d ))W u

[0020] Among them, W d and W u Represent the dimensionality reduction operation and dimensionality increase operation respectively, W d =W u=Conv(x), Conv(x) uses a one-dimensional convolution to reduce the computational burden. DWConv stands for deep convolution, Z(X) stands for the output of the spatiotemporal adapter, and f stands for the spatiotemporal operator.

[0021] Then, following the standard visual attention, multi-head attention, feedforward network and normalization operation are used on Z(X) to obtain the word segmentation Z. It can be expressed as:

[0022]

[0023] Among them, Norm means normalization, MSA means multi-head self-attention, FFN means feedforward neural network, Represents the result of normalized multi-head attention processing.

[0024] Furthermore, the language model decoder in step 3 uses the Bidirectional Encoder Representation from Transformer (BERT).

[0025] BERT receives word segmentation input obtained from the spatiotemporal adapter and encodes information from both the forward and backward directions of the input sequence, which enables the model to fully consider the forward and backward dependencies between video frames when processing spatiotemporal features.

[0026] Fine-tuning is performed by adding an extra output layer that adjusts the feature representation of the word segmentation to generate new predicted word segmentations based on the task at hand, i.e., behavior reasoning and control signal generation in autonomous driving.

[0027] Furthermore, the control signal prediction head in step 4 uses cross-covariance pooling and normalization operations.

[0028] The specific process is:

[0029] First, perform covariance pooling on all word segments, that is,

[0030] Y=Z T Z

[0031] The second-order representation Y of the iterative matrix square root is generated, so that all the word segments are aggregated in a high-dimensional space. Then, Y is normalized. It can be expressed formally as:

[0032] Y′=Norm(Y)

[0033] Where Norm represents the iterative matrix square root normalization method, and Y′ represents the normalized second-order representation.

[0034] Furthermore, iterative matrix square root normalization consists of three consecutive non-linear layers.

[0035] Specifically:

[0036] The first layer is a pre-normalization layer, which uses the Frobenius norm (F norm) calculation to ensure the convergence of subsequent iterations:

[0037]

[0038] A represents the pre-normalized result.

[0039] The second layer computes the square root of the covariance matrix by repeating A a fixed number of times using the Newton-Schultz iteration method.

[0040] The Newton-Schultz iteration is developed based on the approximation of the connection between the matrix symbolic function and the matrix square root. It can be formulated as:

[0041] Y k =f(Y k-1 ,Z k-1 )

[0042] Z k =g(Y k-1 ,Z k-1 )

[0043] Among them, Y0=A, Z0=I, k=1,..,N. Both f(·) and g(·) are matrix operations involving polynomials. After a certain number of iterations, the approximate matrix square root, Y k , Z k Represents the square root of the covariance matrix calculated at the kth iteration.

[0044] The third layer is post-compensation. Because pre-normalization causes data changes, this layer performs compensation operations on it:

[0045]

[0046] C represents the result after compensation operation, Y N represents the square root of the covariance matrix iterated N times.

[0047] Finally, a linear layer is used to obtain the prediction result S of the control signal:

[0048] S=Liear(C)

[0049] Where S = {s1,s2,...,s T}, where s1, s2, …, s T Represents each predicted control signal and Liear represents the linear layer.

[0050] Furthermore, the text reasoning head in step 4 uses a multi-layer perceptron to first extract features from each word layer by layer and generate a corresponding high-dimensional representation. These representations are then mapped to the vehicle's behavior description space. The text reasoning head can output the current state of the vehicle and infer the reason for the behavior.

[0051] Based on the above method, the present invention also proposes an end-to-end interpretable autonomous driving control system based on a large language model, comprising:

[0052] Video sampling module: can implement the content of step 1 above;

[0053] Visual encoder module: can implement the content of step 2 above;

[0054] Decoder module: can implement the content of step 3 above;

[0055] Reasoning and prediction module: The root can implement the content of step 4 above.

[0056] The present invention also includes: placing the autonomous driving system model on the NVIDIA 4090 GPU for end-to-end training, and for the loss function, directly adding the losses of the two prediction heads:

[0057] L=L signal +L text

[0058] Among them, L signal represents the loss of the control signal prediction head, L text represents the loss of the text reasoning head.

[0059] Beneficial effects of the present invention:

[0060] 1. This paper designs an interpretable end-to-end autonomous driving system guided by a large language model. The system takes the video frames captured by the forward-looking camera as input, predicts the control signal at the next moment, and outputs a text description of the vehicle's current behavior. By introducing a language model, the system can not only provide accurate control signal predictions, but also give natural language-based explanations, thereby enhancing the transparency and interpretability of the system, and helping to improve the safety of autonomous driving and user trust.

[0061] 2. The present invention enhances the interpretability of the autonomous driving system through language description, and provides clear semantic interpretation and behavioral reasoning for the driving process. Specifically, the system can output the current driving status of the vehicle in real time and give the reason for the behavior. For example, the system will prompt "the vehicle is slowly stopping" and further explain "because there is a car in front that is slowing down." This language description not only improves the transparency of the system, but also provides users with intuitive feedback, helping them understand the behavioral logic of the vehicle, reducing the psychological burden when using the autonomous driving system, and making autonomous driving more humane and friendly.

[0062] 3. The present invention designs a novel control signal prediction head, which has significant advantages over the traditional simple prediction structure. The control signal prediction head introduces cross-covariance pooling and normalization operations to enable the features between word segments to be more richly aggregated and expressed in high-dimensional space. This design effectively captures the global correlation between word segments and enhances the system's ability to predict vehicle behavior in complex scenarios. In particular, through the iterative matrix square root normalization operation, the system can generate more stable and accurate control signals while maintaining computational efficiency. Compared with previous autonomous driving prediction heads, the design of the present invention can better adapt to complex dynamic driving scenarios and provide the system with more powerful prediction capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The overall structure diagram of an end-to-end interpretable autonomous driving control method and system based on a large language model provided by the present invention;

[0064] Figure 2 A structural diagram of the space-time adapter provided by the present invention;

[0065] Figure 3 A diagram showing the prediction results provided by the present invention; DETAILED DESCRIPTION

[0066] In order to make the purpose and technical solution of the present invention more clearly explained, the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0067] The present invention proposes an end-to-end interpretable autonomous driving control method and system based on a large language model, such as Figure 1 As shown, the overall structure diagram of a large language model end-to-end interpretable control method and system for autonomous driving provided for the implementation of this application mainly includes the following steps:

[0068] Step 1: The video data captured by the single front-view camera is used as the input of the autonomous driving system and is uniformly sampled into video frames;

[0069] like Figure 1The overall structure of the autonomous driving system shown in the figure, the system continuously captures video data in the road scene through a single forward-looking camera. In order to perform subsequent processing and analysis on these video data, a uniform sampling strategy with a fixed time interval is adopted to extract a certain number of video frames (e.g., 8 frames, 16 frames, or 32 frames) as an effective information source for the autonomous driving system input.

[0070] The sampled video frame can be represented as a four-dimensional tensor, denoted by V∈R T×H×W×C , where T is the number of sampled frames, C is the number of channels (RGB three channels), H and W represent the height and width of the image respectively.

[0071] Step 2: Use the spatiotemporal adapter as a visual encoder to extract video features and convert video frames into word segmentation;

[0072] like Figure 2 As shown in the structure diagram of the spatiotemporal adapter, the core design of the spatiotemporal adapter is based on the basic structure of the visual Transformer, but with additional spatiotemporal modeling operations, which consist of upsampling, deep convolution, and downsampling operations. The pre-trained knowledge of the model can be extracted and utilized, which is beneficial to video understanding.

[0073] The specific process is:

[0074] Input feature V∈R T×H×W×C First, a slicing operation is performed to divide the video frame into several small blocks. Then, these small blocks are flattened, that is, these small blocks are converted into one-dimensional vectors. Then, the obtained vectors are transformed into the input dimensional space of the model using a projection operation. Finally, each frame of the video is converted into a feature vector, and the feature X∈R is obtained. N ×d , where N is the number of slices and d represents the dimension after the change.

[0075] The feature X is fed into the spatiotemporal adapter, which introduces a spatiotemporal operator implemented by a deep convolution. It can be expressed as:

[0076] Z(X)=X+f(DWConv(XW d ))W u

[0077] Among them, W d and W u They represent dimensionality reduction and dimensionality increase operations, respectively, which can reduce the computational burden. DWConv stands for deep convolution.

[0078] Then, following the standard visual attention, multi-head attention, feedforward network and normalization operation are used on Z(X) to obtain the word segmentation Z. It can be expressed as:

[0079]

[0080] Among them, Norm means normalization, MSA means multi-head self-attention, and FFN means feedforward neural network.

[0081] Step 3: Send the segmented words obtained from the spatiotemporal adapter to the language model decoder to obtain the predicted new segmented words;

[0082] The language model decoder in the present invention uses the bidirectional encoder representation from Transformer (BERT).

[0083] BERT receives word segmentation input obtained from the spatiotemporal adapter and encodes information from both the forward and backward directions of the input sequence, which enables the model to fully consider the forward and backward dependencies between video frames when processing spatiotemporal features.

[0084] Fine-tuning is performed by adding an extra output layer that adjusts the feature representation of the word segmentation to generate new predicted word segmentations based on the task at hand, i.e., behavior reasoning and control signal generation in autonomous driving.

[0085] Step 4: Send the predicted new word segmentation to the control signal prediction head and the text reasoning head respectively;

[0086] like Figure 1 The text reasoning head in the lower left corner is shown. The text reasoning head uses a multi-layer perceptron to first extract features from each word layer by layer to generate a corresponding high-dimensional representation. These representations are then mapped to the vehicle's behavior description space. For example, the text reasoning head can output the state that the vehicle is slowly stopping, and also infer the reason for the behavior, such as a car ahead is slowing down.

[0087] like Figure 1 As shown in the control signal prediction head in the lower right corner, cross covariance pooling and normalization operations are used respectively.

[0088] The specific process is:

[0089] First, perform covariance pooling on all word segments, that is,

[0090] Y=Z T Z

[0091] A second-order representation Y of the square root of the matrix is ​​generated, so that all the word segments are aggregated in a high-dimensional space.

[0092] Then, normalize Y. It can be expressed formally as:

[0093] Y′=Norm(Y)

[0094] Where Norm represents the iterative matrix square root normalization method, and Y′ represents the normalized second-order representation.

[0095] Furthermore, iterative matrix square root normalization consists of three consecutive non-linear layers.

[0096] The specific design is:

[0097] The first layer is a pre-normalization layer, which uses the Frobenius norm (F norm) calculation to ensure the convergence of subsequent iterations:

[0098]

[0099] ||·|| F represents the Frobenius norm.

[0100] The second layer computes the square root of the covariance matrix by repeating A a fixed number of times using the Newton-Schultz iteration method.

[0101] The Newton-Schultz iteration is developed based on the approximation of the connection between the matrix symbolic function and the matrix square root. It can be formulated as:

[0102] Y k =f(Y k-1 ,Z k-1 )

[0103] Z k =g(Y k-1 ,Z k-1 )

[0104] Where Y0=A, Z0=I, k=1, .., N. Both f(·) and g(·) are matrix operations involving polynomials. After a certain number of iterations, the approximate matrix square root can be obtained.

[0105] The third layer is post-compensation. Because pre-normalization causes data changes, this layer performs compensation operations on it:

[0106]

[0107] Finally, a linear layer is used to obtain the prediction result S of the control signal:

[0108] S=Liear(C)

[0109] Where S = {s1,s2,...,s T}.

[0110] Step 5: Perform end-to-end training on the built model system to obtain the predicted control signal at the next moment;

[0111] The built model is trained on the NVIDIA 4090 GPU.

[0112] The specific parameters are set as follows: the optimizer uses AdamW, the training batch size is 3, and the initial learning rate is 0.0001. A total of 32 iterations are trained.

[0113] For the settings of other hyperparameters: the maximum number of frames T is set to 32, each frame is randomly cropped to a size of 224×224 in length and width, and the number of iterations in the control signal prediction head is set to 5.

[0114] Trained and tested on the BDD-X dataset.

[0115] During training, for the loss function, the losses of the two prediction heads are directly added together:

[0116] L=L signal +L text

[0117] Among them, L signal represents the loss of the control signal prediction head, L text represents the loss of the text reasoning head.

[0118] During testing, two types of metrics were used to evaluate the system, including:

[0119] Several metrics widely used in the NLP field: BLEU4, METEOR, and CIDEr. These metrics mainly evaluate the accuracy of the current behavior description and causal reasoning. Among them, BLEU4 evaluates the quality of generated text, METEOR considers the accuracy and recall based on the entire corpus, and CIDEr is closer to the principle of humans to judge whether two sentences are similar.

[0120] Vehicle control signal prediction metrics: mean square error (RMSE) and threshold accuracy in is the error tolerance value. When the error between the predicted result and the true value is less than When , the predicted control signal is considered to be correct. They are set to {0.1, 0.5, 1.0, 5.0} respectively. Among them, the commonly used turning angle and speed control signals are used for measurement.

[0121] The system achieves high accuracy in both evaluation indicators: BLEU4, METEOR, and CIDEr achieve 35.6, 251.9, 30.6, and 10.7, 93.0, 15.2 in vehicle behavior description and reasoning, respectively. 5.0 The control signal prediction under the threshold achieved a turning angle accuracy of 97.4% and a speed accuracy of 98.3%.

[0122] like Figure 3 As shown, the system finally provides the current driving state of the vehicle (such as the vehicle is stopping slowly) and the reason for the behavior (such as because there is a slow-moving vehicle in front), and outputs a low-level control signal for the next moment, including the turning angle and speed.

[0123] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. An end-to-end interpretable autonomous driving control method based on a large language model, characterized in that: The steps include: Step 1: uniformly sample the video data captured by a single front-view camera to obtain a video frame; Step 2: Use the spatiotemporal adapter as a visual encoder to extract features and convert video frames into word segmentation. Step 3: Send the segmented words into the large language model decoder to obtain the predicted new segmented words; Step 4: Send the predicted new word segmentation to the control signal prediction head and the text reasoning head respectively; the text reasoning head outputs the vehicle driving status and reason, and the control signal prediction head outputs the predicted value of the vehicle control signal.

2. The end-to-end interpretable autonomous driving control method based on a large language model according to claim 1, characterized in that: The specific implementation of step 1 includes the following: For the input video data, a fixed time interval is used for uniform sampling to extract a certain number of video frames, and the sampled video frames are represented as a four-dimensional tensor, denoted as V∈R T×H×W×C , where T is the number of sampled frames, C is the number of channels (RGB three channels), H and W represent the height and width of the image respectively.

3. The end-to-end interpretable autonomous driving control method based on a large language model according to claim 1, characterized in that: The specific implementation of step 2 includes the following: The spatiotemporal adapter samples the basic structure of the visual Transformer and introduces additional spatiotemporal modeling operations, which consist of upsampling, deep convolution, and downsampling operations to extract and utilize the pre-trained knowledge of the model. The specific process is as follows: Input feature V∈R T×H×W×C First, a slicing operation is performed to divide the video frame into several small blocks; then, these small blocks are flattened, that is, these small blocks are converted into one-dimensional vectors; then, the obtained vectors are transformed into the input dimensional space using a projection operation; finally, each frame of video is converted into a feature vector, and the feature X∈R is obtained. N×d , where N is the number of slices and d represents the dimension after the change; The feature X is fed into the spatiotemporal adapter and a spatiotemporal operator is introduced, which is implemented by deep convolution and is expressed as: Z(X)=X+f(DWConv(XW d ))W u Among them, W d and W u They represent dimensionality reduction and dimensionality increase operations respectively, and DWConv represents deep convolution; Then, following the standard visual attention, we use multi-head attention, feedforward network and normalization operation on Z(X) to obtain the word segmentation Z, which can be expressed as: Among them, Norm means normalization, MSA means multi-head self-attention, and FFN means feedforward neural network.

4. The end-to-end interpretable autonomous driving control method based on a large language model according to claim 1, characterized in that: The large language model decoder in step 3 uses the bidirectional encoder representation BERT of Transformer, and the specific process is as follows: BERT receives the word segmentation input obtained from the spatiotemporal adapter and encodes information from both the front and back directions of the input sequence, so that the model can fully consider the front and back dependencies between video frames when processing spatiotemporal features; Fine-tuning is performed by adding an additional output layer that is able to adjust the feature representation of the word segmentation according to the current task, namely behavior reasoning and control signal generation in autonomous driving, thereby generating new predicted word segmentations.

5. The end-to-end interpretable autonomous driving control method based on a large language model according to claim 1, characterized in that: The control signal prediction head in step 4 uses cross covariance pooling and normalization operations, and the specific process is as follows: First, covariance pooling is performed on all the segmented words to generate the second-order representation Y of the square root of the iterative matrix, so that all the segmented words are aggregated in a high-dimensional space; that is, Y=Z T Z Then, Y is normalized and can be expressed as: AND ′ =Norm(Y) Where Norm represents the iterative matrix square root normalization operation, Y ′ represents the normalized second-order representation.

6. The end-to-end interpretable autonomous driving control method based on a large language model according to claim 5, characterized in that: The iterative matrix square root normalization consists of three consecutive nonlinear layers, specifically: The first layer is a pre-normalization layer, which uses the Frobenius norm (F norm) to ensure the convergence of subsequent iterations: The second layer calculates the square root of the matrix, and uses the Newton-Schultz iteration method to repeat A for a fixed number of times to approximate the square root of the covariance matrix. The formula is as follows: Y k =f(Y k-1 ,Z k-1 ) Z k =g(Y k-1 ,Z k-1 ) Where Y0=A, Z0=I, k=1,..,N, f(·) and g(·) represent Newton-Schultz iterative matrix operations containing polynomials. After a certain number of iterations, the approximate matrix square root can be obtained; The third layer is post-compensation. Because pre-normalization causes data changes, this layer performs compensation operations on it: Finally, a linear layer is used to obtain the prediction S of the control signal: S=Liear(C) Where S = {s1,s2,...,s T }.

7. The end-to-end interpretable autonomous driving control method based on a large language model according to claim 1, characterized in that: The text reasoning head in step 4 uses a multi-layer perceptron to first perform layer-by-layer feature extraction on each word segment to generate a corresponding high-dimensional representation, which is then mapped to the vehicle's behavior description space. The text reasoning head outputs the current state of the vehicle and infers the reason for the behavior.

8. An end-to-end interpretable autonomous driving control system based on a large language model, characterized by: include Video sampling module: capable of implementing the content of step 1 in claim 1; Visual encoder module: capable of implementing the content of step 2 in claim 1; Decoder module: capable of implementing the content of step 3 in claim 1; Reasoning and prediction module: The root can implement the content of step 4 in claim 1.

9. The end-to-end interpretable autonomous driving control system based on a large language model according to claim 8, characterized in that: The training sampling end-to-end training method of the autonomous driving control system is specifically set as follows: End-to-end training was performed on the NVIDIA 4090 GPU dataset. The specific parameters were set as follows: the optimizer used AdamW, the training batch size was 3, the initial learning rate was 0.0001, and a total of 32 iterations were trained; For the settings of other hyperparameters: the maximum number of frames T is set to 32, each frame is randomly cropped to a length and width of 224×224, and the number of iterations in the control signal prediction head is set to 5.

10. The end-to-end interpretable autonomous driving control system based on a large language model according to claim 8 or 9, characterized in that: The loss function of the autonomous driving control system during training is to add the losses of the two prediction heads: L=L signal +L text Among them, L signal represents the loss of the control signal prediction head, L text represents the loss of the text reasoning head.

Images