A visual large language model-based automatic driving vehicle collision risk assessment method and system

Abstract

The present application relates to a kind of automatic driving vehicle collision risk assessment method and system based on visual large language model, the method includes the following steps: step 1, utilize structure theme model to model collision accident narrative data set;Step 2, use XGBoost model and Shapley explanation to establish the relationship between theme and collision accident severity;Step 3, establish the visual big model training data set for collision risk assessment;Step 4, based on visual large language model, the fine-tuning training of visual large language model is carried out for collision risk assessment;Step 5, build joint simulation platform and carry out risk assessment simulation verification.The method improves the perception understanding ability of automatic driving vehicle to traffic scene, accurately analyzes the potential collision threat of current driving under complex traffic environment, realizes the assessment of collision risk in different scenarios, effectively improves the accuracy and practical value of collision risk assessment, and improves the safety of automatic driving car driving.

Description

Technical Field

[0001] This invention belongs to the field of traffic engineering technology, specifically relating to a method and system for assessing collision risks of autonomous vehicles based on a visual large language model. Background Technology

[0002] Visual large language models are a class of models that not only inherit the powerful text processing capabilities of traditional large language models but also extend their ability to understand and analyze various data formats such as images, audio, and video. Visual large language models can combine image data captured by cameras with environmental information provided by onboard sensors to perform deep learning and pattern recognition, thereby more accurately assessing potential collision risks. Furthermore, visual large language models can continuously optimize their risk assessment algorithms by analyzing historical data and real-time feedback, improving the system's reliability and robustness. Therefore, using visual large language models for collision risk assessment of autonomous vehicles can not only improve the safety performance of autonomous vehicles but also provide a feasible and effective solution to address the bottlenecks in existing technologies. Current collision risk assessment technologies for autonomous vehicles are mainly based on traditional physical kinematic indicators, traditional machine learning models, or early single-modal and simple multimodal models. These technologies generally suffer from poor scene adaptability, difficulty in effectively processing unstructured visual data, insufficient depth of multimodal information fusion, and low efficiency in model adaptation and fine-tuning. They also fail to establish a deep correlation between accident causes and real-time risk levels, making it difficult to meet the safety assessment needs in complex traffic environments.

[0003] Therefore, there is an urgent need for a collision risk assessment method and system for autonomous vehicles based on a visual large language model, which can help autonomous vehicles to anticipate potential collision risks while driving and assess collision risks in different scenarios. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a method and system for assessing collision risks of autonomous vehicles based on a visual large language model.

[0005] This invention relates to a collision risk assessment method for autonomous vehicles based on a visual large language model, comprising the following steps:

[0006] Step 1: Model the collision incident narrative dataset using a structural topic model;

[0007] Step 2: Use the XGBoost model and Shapley explanation to establish the relationship between the subject and the severity of the collision.

[0008] Step 3: Establish a large visual model training dataset for collision risk assessment;

[0009] Step 4: Collision risk assessment based on the visual large language model; fine-tuning and training of the visual large language model.

[0010] Step 5: Build a joint simulation platform to conduct risk assessment simulation verification.

[0011] Further, in step 1, collision accident reports are downloaded from the collision accident dataset. Data parsing tools are used to extract the required data and remove invalid information to obtain the collision accident narrative dataset. Each collision accident narrative is converted into a bag-of-words matrix and a metadata matrix. The collision accident narrative dataset is then modeled using a structural topic model, with the following topic proportions:

[0012]

[0013] Where, θ d Let μ be the topic distribution vector of the d-th collision incident narrative data. d For θ d The mean vector of the Logistic Normal distribution it follows, μ d，k Let μ be the mean vector d The k-th element, X d Let Σ be a 1×p-dimensional vector based on the topic popularity covariate vector; Σ be a (k-1)×(k-1) covariance matrix; γ be a p×(k-1) coefficient matrix. k Let be the k-th column vector of the coefficient matrix γ;

[0014] Keyword distribution:

[0015]

[0016] Where v is the identifier for the collision incident topic, d is the identifier for the collision incident narrative document, and β d,v k This represents the log-expected distribution of keywords related to the keyword "k" under the collision incident narrative document labeled "d" and the collision incident topic labeled "v", where "k" is an identifier used to distinguish different bias terms and interaction terms, and "m" represents the distribution of keywords related to the keyword "k". v The expected logarithmic value of the frequency of occurrence of the identifier v for the topic of collision incidents in the document; k k Topic-specific deviation; For covariate group bias; The interaction between topic bias and covariate bias;

[0017] Topic distribution and word distribution:

[0018]

[0019] Where n is the location identifier within the collision incident description document, Zd,n This indicates the topic assignment of position identifier n in the collision accident narrative identified as d; w d,n For word distribution based on selected topics; β d,r This represents the word probability distribution in the collision accident narrative data labeled d, where r is the label of the associated topic, and the ~ in the formula indicates that it follows a certain probability distribution.

[0020] Further, in step 1, the required data includes a description of the collision incident in natural language, the time of the collision incident, its severity, and other information related to the collision incident; the invalid information includes stop words, punctuation marks, and numbers, as well as words related to location, manufacturer, and report; the bag-of-words matrix is ​​a digital representation of the collision incident description understood and analyzed by the structured topic model; the metadata matrix includes the time and severity of the collision incident.

[0021] Furthermore, in step 2, one tree in the XGBoost model is:

[0022]

[0023] In the formula, f represents the predicted value of the i-th collision accident sample, x is the input feature vector related to the collision accident, and i represents the index of the collision accident sample; c Let be the prediction function of the k-th regression tree, F be the space of the regression tree, q be the leaf weights, q(x) be the function that maps x to a leaf node of the regression tree, and w be the prediction function of the k-th regression tree. q(x) This represents the weight value corresponding to the leaf node, c is the index of the regression tree, and C is the total number of regression trees in the XGBoost model;

[0024] The goal of the XGBoost model is:

[0025]

[0026] In the formula, θ is the parameter set of the XGBoost model, l is a differentiable convex loss function, and y i Let be the true collision severity label for the i-th collision accident sample, s be the total number of collision accident samples, and Ω be the regularization term;

[0027] The regularization function is:

[0028]

[0029] In the formula, Ω is the regularization term, T is the number of leaves, η is the penalty coefficient for the number of leaf nodes, λ is the L2 regularization coefficient for the leaf weights, and w is the weight vector of the leaf nodes in the regression tree.

[0030] Metadata containing collision accident thematic features and autonomous driving collision accident severity is randomly divided into training and testing data. Data oversampling is used to balance the training data. An XGBoost model is used to train the training data, with the topic distribution matrix of collision accident narratives used as the input to the XGBoost model and the collision accident severity in the metadata used as the model output. Training and testing sets are constructed to train an XGBoost model to evaluate the severity of collision accidents. The trained XGBoost model is loaded, and the Shapley value interpretation method is used to calculate the SHAP value. SHAP analysis is performed according to the severity of collision accidents to extract key causal themes.

[0031] The logarithmic loss used to optimize hyperparameters is:

[0032]

[0033] In the formula, N is the number of samples, M is the number of categories, j is the category index used to traverse different categories, and y ij p is a binary variable with the expected class. ij This represents the estimated probability of the sample.

[0034] Further, in step 3, a car driving accident dataset is obtained by capturing real traffic accident video data and related annotations using visual sensors installed on driving vehicles. The car driving accident dataset includes a collision accident dataset and a regular driving dataset. For the video images in the collision accident dataset, collision accident / regular driving image processing is performed, and the YOLO model is used to annotate pedestrians and vehicles in the images. Based on the key causal themes of collision accidents extracted in step 2 through the XGBoost model and Shapley explanation, a collision risk assessment reasoning chain is constructed. Combined with the annotation and labeling information in the car driving accident dataset, image description prompts are generated and combined with the processed collision accident / regular driving images to form image-text pairs, ultimately constructing a large visual model training dataset.

[0035] Furthermore, in step 3, the specific steps of the collision accident / normal driving image processing are as follows:

[0036] Step 3-1: Extract regular driving video frame images as training images for low collision risk; for collision accident images, select video frames before the collision accident based on time annotations, and the extracted collision accident image dataset contains accident time annotations.

[0037] Step 3-2: Use the Laplace operator to filter collision accident / normal driving images. For a given two-dimensional image function f(a,b), where a represents the horizontal coordinates of the image and b represents the vertical coordinates of the image, its Laplace operator is expressed in the following second-order partial differential equation form:

[0038]

[0039] The second-order differential of the Laplacian operator with respect to the two-dimensional image function f(a,b) is decomposed into partial derivatives along the spatial coordinate axes, and its expansion is as follows:

[0040]

[0041] Δf(a,b) expands as follows:

[0042]

[0043] Based on the above differential expression, a corresponding Laplacian convolution kernel is constructed. This operator uses a 3×3 neighborhood calculation method, taking the pixel to be processed (a,b) as the core, and performing second-order difference operations on the gray values ​​of its four neighborhoods. The template operator is defined as follows:

[0044] Δf(a,b)=f(a+1,b)+f(a-1,b)+f(a,b+1)+f(a,b-1)-4f(a,b) (11)

[0045] Based on the aforementioned second-order differential equation, the corresponding Laplacian convolution operator is derived. This operation template adopts a centrally symmetric structure, taking the pixel point (a,b) to be processed as the reference, and performs second-order differential calculation on the pixel values ​​of its four adjacent orthogonal directions.

[0046] Step 3-3: Each image selected in step 3-2 is represented as follows:

[0047] f(a,b)=f i (a,b)×f r (a,b) (12)

[0048] In the formula, f i (a,b) represents the illumination of the image; f r (a,b) represents the reflection intensity of the graph;

[0049] By performing a logarithmic transformation on the original image, its original multiplicative features are transformed into additive features. The transformation result is expressed as follows:

[0050] lnf(a,b)=lnf i (a,b)+lnf r (a,b) (13)

[0051] After performing a Fourier transform, the signal is mapped to the frequency domain; a frequency domain filter is then used for filtering, and the mathematical expression of this filter in the frequency domain is:

[0052] H(u,e)=(γ H -γ L )H hp (u,e)+γ L (14)

[0053] In the formula, u is the horizontal frequency coordinate in the frequency domain, e is the vertical frequency coordinate in the frequency domain, and γ... H γ L To adjust the amplitude response range of the filter, H hp (u,e) represents the designed frequency domain filter;

[0054] Steps 3-4: Use the YOLO model to annotate road vehicles and pedestrians. The specific formula is as follows:

[0055]

[0056] In the formula, F(a,b) is an element in the feature map, I(a+m,b+n) is a local region of the input image, K(m,n) is the convolution kernel, and m and n are the dimension indices of the convolution kernel K;

[0057] The YOLO model output format is [a1, b1, w1, h1, conf], where a1 and b1 are the coordinates of the center point of the detection box, w1 and h1 are the width and height of the detection box, and conf is the confidence score of the current detection box. The input coordinates of the large language model are the coordinates of the four corner points of the detection box [a_min, b_min, a_max, b_max]. The formula conversion is as follows:

[0058]

[0059] The collision risk assessment reasoning chain architecture is as follows: The driving environment is used to describe the driving scenario; key objects in the driving scenario that affect the collision risk of the vehicle are identified; the characteristics and potential impacts of the identified key objects on the vehicle are analyzed to achieve collision risk assessment; based on the well-structured collision risk assessment reasoning chain, a large visual model training dataset is constructed; the large visual model training dataset is divided into inference stage dataset instructions and fine-tuning stage dataset instructions.

[0060] The inference stage dataset instructions are based on the required input and target output dataset instructions, by concatenating the system role prompts and instruction prompts in the dataset, adding prompts, and using the dialogue content expression method of the pre-training stage.

[0061] The fine-tuning stage dataset instructions are based on the required input and target output dataset instructions, and are used as the target for fine-tuning to optimize the parameters of the LoRA module in the large model through back gradient propagation.

[0062] Furthermore, in step 3-2, the collision accident / normal driving images are filtered using the following method:

[0063] Step 3-2-1: Read the image files sequentially from the video extraction folder;

[0064] Step 3-2-2: Check if the current image is the last image in the folder. If it is, terminate the entire filtering process; otherwise, proceed with the subsequent processing.

[0065] Step 3-2-3: Convert the color image to grayscale and apply the Laplacian operator to calculate the image sharpness index;

[0066] Step 3-2-4: Determine the ambiguity of the grayscale image using a preset threshold;

[0067] Step 3-2-5: After the image results in Step 3-2-4 meet the resolution standards, store them in the corresponding directory and the system will automatically load the next image to be processed.

[0068] Step 3-2-6: Terminate the entire filtering process when the last image is detected.

[0069] Furthermore, in step 4, the visual large language model is expressed by the following formula:

[0070] LLM(O,V)=f ε (E O (O),E V (V)) (17)

[0071] In the formula, O represents text input, V represents visual input, and ε represents the set of learnable parameters for the visual large language model.

[0072] E O E is a text embedding function. V For visual embedding functions, f ε Let ε represent the function of the visual large language model;

[0073] Fine-tuning of the visual large language model is achieved through the LoRA model, and the specific method is as follows:

[0074] Assuming W is the weight matrix in the Transformer layer, LoRA decomposes it into...

[0075] W = W0 + UN T (18)

[0076] In the formula, W0 is the original weight matrix, U and N are low-order matrices learned during fine-tuning, and T represents the transpose of matrix N;

[0077] This adaptation applies to the attention layer and feedforward layer of the Transformer. The mathematical representation of the adapter in the attention layer is:

[0078]

[0079] In the formula, Q is the query vector matrix, F is the key vector matrix, G is the value vector matrix, and ΔF is the change in the key matrix learned during fine-tuning; d F is the dimension of the embedding vector.

[0080] Further, in step 5, select ScaneR TM The studio simulation platform was used for risk assessment, and PyTorch was used as the computational framework for neural network modeling. The experimental environment was configured, and three typical collision risk scenarios were constructed: a rear-end collision scenario where the vehicle in front suddenly decelerates, causing a forward rear-end collision; a lane-changing scenario where the vehicle is traveling straight and another vehicle changes lanes, causing a conflict; and a merging scenario where the vehicle changes lanes and merges into another lane, causing a conflict with vehicles traveling in the original lane. The Collision Risk Index (CPI) was selected as the assessment standard for vehicle collision risk, expressed by the following formula:

[0081]

[0082] In the formula, For vehicles Potential collision index, For vehicles The initial time interval, in seconds; For vehicles The final time interval is in seconds; Δt is the observation time interval, in seconds. For vehicles The total simulation time, in seconds; An index for vehicles, used to identify different vehicles. For vehicles The dynamic risk assessment coefficient at time t is used to quantify the vehicle. Real-time collision risk level at time t For vehicles The minimum acceptable dynamic risk at time t, where σ is the risk weight coefficient;

[0083] The K-means clustering method was used to classify the risk level of vehicles into three categories: low risk, medium risk, and high risk. The distance between the three cluster centers was used as a threshold grading index to classify the three collision risk levels. The accuracy of the visual large language model before and after training and other machine learning models in assessing collision risk in various scenarios was compared to verify the effectiveness and necessity of model training.

[0084] The present invention also relates to a system for an autonomous vehicle collision risk assessment method based on a visual large language model, the system comprising a computer module for running the autonomous vehicle collision risk assessment method based on a visual large language model.

[0085] Beneficial effects

[0086] This invention proposes a collision risk assessment method based on a visual large language model, which improves the perception and understanding capabilities of autonomous vehicles in traffic scenarios, accurately analyzes potential collision threats in complex traffic environments, and enables the assessment of collision risks in different scenarios. This effectively improves the accuracy and practical value of collision risk assessment and enhances the safety of autonomous vehicles. Attached Figure Description

[0087] Figure 1 This is a schematic diagram of the structural theme model of the present invention;

[0088] Figure 2 This is a flowchart of the XGBoost model development process of the present invention;

[0089] Figure 3 This is a flowchart of the video data processing of the present invention;

[0090] Figure 4 This is a flowchart of the fuzzy image filtering process of the present invention;

[0091] Figure 5 This is a diagram illustrating the homomorphic filtering process of the present invention;

[0092] Figure 6 This is a schematic diagram of the collision risk assessment reasoning thought chain architecture of the present invention;

[0093] Figure 7 This is a graph showing the change of the loss function during the LoRA fine-tuning training process of this invention;

[0094] Figure 8 This is a simulated overall road plan view of the present invention;

[0095] Figure 9a This is a scenario diagram illustrating the forward rear-end collision risk of the present invention.

[0096] Figure 9b This is a scene diagram illustrating a conflict caused by a vehicle changing lanes and merging into another vehicle, as described in this invention.

[0097] Figure 9c This is a scene diagram illustrating a conflict between the vehicle of this invention and a vehicle traveling in the original lane;

[0098] Figure 10 A flowchart for creating the evaluation dataset of this invention;

[0099] Figure 11a This is a schematic diagram of the confusion matrix for rear-end collision scene effects in this invention;

[0100] Figure 11b This is a schematic diagram of the confusion matrix for the merging effect of vehicles ahead in this invention;

[0101] Figure 11c This is a schematic diagram of the confusion matrix for the lane-changing scene effect of the vehicle in this invention;

[0102] Figure 12a This is a schematic diagram of the confusion matrix of the original model risk assessment effect in a rear-end collision scenario in an embodiment of the present invention;

[0103] Figure 12b This is a schematic diagram of the confusion matrix of the original model risk assessment effect in the vehicle merging scenario in an embodiment of the present invention;

[0104] Figure 12c This is a schematic diagram of the confusion matrix of the risk assessment effect of the original model in the vehicle lane change scenario in an embodiment of the present invention;

[0105] Figure 13 This is a schematic diagram illustrating the improvement in model training evaluation results before and after training in various scenarios in embodiments of the present invention. Detailed Implementation

[0106] The following combination Figures 1 to 13 This implementation method will be described in detail.

[0107] This invention proposes a visual large language model for autonomous driving collision risk assessment. Causal analysis of autonomous driving accidents is conducted, and latent topics are extracted from unstructured autonomous vehicle collision narratives using Structural Topic Modeling (STM) technology, a natural language processing technique. An XGBoost model combined with collision metadata is used to establish a relationship model between topics and collision severity. The SHAP interpretation method is then used to analyze the causal topics related to the severity of autonomous vehicle collision accidents. This analysis lays the foundation for the proposed collision risk reasoning chain.

[0108] This invention utilizes the LoRA algorithm and a constructed training dataset, based on the Qwen2-VL visual large language model, to perform supervised fine-tuning (SFT) on the visual large language model, optimizing the model's inference parameters and constructing a collision risk assessment visual large language model. Based on a simulation platform, three conflict scenarios—rear-end collision, lane change, and merging—were implemented to verify and evaluate the model's performance.

[0109] This invention relates to a collision risk assessment method for autonomous vehicles based on a visual large language model, comprising the following steps:

[0110] Step 1: Model the collision incident narrative dataset using a structural topic model.

[0111] Download the collision accident reports from the collision accident dataset in PDF format. Use the data parsing tool pdftools to extract the required data, including a natural language description of the collision incident, the time of the collision, its severity, and other relevant information. Remove invalid information from the required data, including stop words, punctuation marks, and numbers, as these typically do not provide meaningful and unique information about individual collisions. Other words commonly found in collision accident reports but not providing useful information about the collisions should also be removed from the collision accident narratives, including words related to location, manufacturer, and report. After removing invalid information, the collision accident narrative dataset is obtained and saved in CSV format. The collision accident narrative dataset requires preprocessing, where each collision accident narrative is converted into a bag-of-words matrix and a metadata matrix. The bag-of-words matrix is ​​a numerical representation of the collision accident narrative that a Structured Topic Model (STM) can understand and analyze; the metadata matrix includes the time and severity of the collision.

[0112] A structural topic model was used to model the collision incident narrative dataset. A schematic diagram of the structural topic model is shown below. Figure 1 As shown in the diagram. Here, X is the topic popularity covariate, μ is the mean vector of the probability distribution followed by the topic distribution θ, θ is the topic distribution, Z is the topic assignment for each word, w is the word, β is the topic-word distribution, and Y is the topic content covariate. The structural topic model consists of three basic components: the topic popularity model controls how words are assigned to topics, the topic content model controls the frequency of words in topics, and the language model combines the popularity and content models to generate the actual words.

[0113] Download collision accident reports from the collision accident dataset, use data parsing tools to extract the required data, and remove invalid information to obtain the collision accident narrative dataset. Each collision accident narrative is converted into a bag-of-words matrix and a metadata matrix. The collision accident narrative dataset is then modeled using a structural topic model, with the following topic proportions:

[0114]

[0115] Where, θ d Let μ be the topic distribution vector of the d-th collision incident narrative data. dFor θ d The mean vector of the Logistic Normal distribution it follows, μ d，k Let μ be the mean vector d The k-th element, X d Let Σ be a 1×p-dimensional vector based on the topic popularity covariate vector; Σ be a (k-1)×(k-1) covariance matrix; γ be a p×(k-1) coefficient matrix. k Let be the k-th column vector of the coefficient matrix γ;

[0116] Keyword distribution:

[0117]

[0118] Where v is the identifier for the collision incident topic, d is the identifier for the collision incident narrative document, and β d,v k This represents the log-expected distribution of topic terms related to identifier k under the collision incident narrative document identified as d and the collision incident topic identified as v, where k is an identifier used to distinguish different bias terms and interaction terms, and m... v The expected logarithmic value of the frequency of occurrence of the identifier v for the topic of collision incidents in the document; k k Topic-specific deviation; For covariate group bias; The interaction between topic bias and covariate bias;

[0119] Topic distribution and word distribution:

[0120]

[0121] Where n is the location identifier within the collision incident description document, Z d,n This indicates the topic assignment of position identifier n in the collision accident narrative identified as d; w d,n For word distribution based on selected topics; β d,r This represents the word probability distribution in the collision accident narrative data with the identifier d, where r is the identifier of the associated topic, and the ~ in the formula indicates that it follows a certain probability distribution;

[0122] Step 2: Use the XGBoost model and Shapley explanation to establish the relationship between the subject and the severity of the collision.

[0123] The XGBoost model is a decision tree-based model that is easier to interpret than other "black box" machine learning models. The interpretation of the XGBoost model can quantify and understand the relationship between the subject and the severity of a collision, which is helpful in providing suggestions for improving the safety of autonomous vehicles and reducing the severity of collisions. Compared with deep learning models such as convolutional neural networks and recurrent neural networks, the XGBoost model requires far fewer computational resources.

[0124] A tree in the XGBoost model is:

[0125]

[0126] In the formula, Let f represent the predicted value of the i-th collision accident sample, where x is the input feature vector related to the collision accident, and i represents the index of the collision accident sample; c Let be the prediction function of the k-th regression tree, F be the space of the regression tree, q be the leaf weights, q(x) be the function that maps x to a leaf node of the regression tree, and w be the prediction function of the k-th regression tree. q(x) This represents the weight value corresponding to the leaf node, c is the index of the regression tree, and C is the total number of regression trees in the XGBoost model;

[0127] The goal of the XGBoost model is:

[0128]

[0129] In the formula, θ is the parameter set of the XGBoost model, l is a differentiable convex loss function, and y i Let be the true collision severity label for the i-th collision accident sample, s be the total number of collision accident samples, and Ω be the regularization term;

[0130] The regularization function is:

[0131]

[0132] In the formula, Ω is the regularization term, T is the number of leaves, η is the penalty coefficient for the number of leaf nodes, λ is the L2 regularization coefficient for the leaf weights, and w is the weight vector of the leaf nodes in the regression tree.

[0133] To develop the XGBoost model, the data, including the collision accident topic distribution extracted by the structural topic model and the severity of autonomous driving collision accidents, is first randomly divided into training and testing data, and this partitioning process is repeated. Furthermore, a synthetic minority oversampling technique is used to balance the training data to overcome the data imbalance problem. Then, for each partitioned training and testing set, the XGBoost model is trained on the training data, and its hyperparameters are determined according to... Figure 2The process shown is optimized. The topic distribution matrix of the collision incident narrative is used as the input to the XGBoost model, and the collision incident severity in the metadata is used as the output of the model. Training and test sets are constructed to train an XGBoost model that can evaluate the severity of collision incidents.

[0134] The logarithmic loss used to optimize hyperparameters is:

[0135]

[0136] In the formula, N is the number of samples, M is the number of categories, j is the category index used to traverse different categories, and y ij p is a binary variable with the expected class. ij This represents the estimated probability of the sample.

[0137] Shapley Explanation (SHAP) is a method used to explain the relationship between topics learned by XGBoost and the severity of collisions. SHAP values ​​are additive, meaning that the contribution of each feature to the final prediction can be calculated independently. After training the model, the model is loaded, and the TreeExplainer from the shap library is used to estimate the SHAP values. The larger the absolute value of the SHAP value of a topic, the more important it is to the model's output. This is because important topics represent specific information related to the severity of collisions, while other topics represent relatively general information.

[0138] Step 3: Establish a large visual model training dataset for collision risk assessment.

[0139] A Car Crush Dataset (CCD) is obtained by capturing real traffic accident videos and their associated annotations using visual sensors installed on driving vehicles. The CCD includes a collision accident dataset and a regular driving dataset. For the video images in the collision accident dataset, collision accident / regular driving image processing is performed, and pedestrians and vehicles in the images are annotated using the YOLO model. Based on the key causal themes of collision accidents extracted in step 2 using the XGBoost model and Shapley explanation, a collision risk assessment reasoning chain is constructed. Combined with the annotations and labeling information in the Car Crush Dataset (CCD), image description prompts are generated and combined with the processed collision accident / regular driving images to form image-text pairs, ultimately constructing a large-scale visual model training dataset.

[0140] To improve image quality in subsequent experimental phases, and considering that avoiding vehicle collisions in the real world requires not only considering vehicles ahead but also pedestrians and adhering to current road rules, it is necessary to re-annotate the video data and perform image enhancement operations. See below for the specific processing flow. Figure 3 .

[0141] The specific steps for processing collision / normal driving images are as follows:

[0142] Step 3-1: Extract regular driving video frames as training images for low-collision-risk scenarios. For collision accident images, since assessing collision risk after a collision is irrelevant, select video frames before the collision based on time stamps. The extracted collision accident image dataset contains accident time stamps, for example, [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1], where 0 represents no accident in this frame and 1 represents an accident in this frame. The open-source multimedia processing tool FFmpeg is used to extract the video frames.

[0143] Step 3-2: Addressing the issue of blurred images in the original CCD dataset, this invention employs image sharpness-based data filtering. An image sharpness evaluation method is introduced during the data preprocessing stage to automatically filter training samples. The Laplacian operator is used to filter collision / normal driving images. This operator, based on the second-order differential principle, effectively quantifies abrupt changes in grayscale values ​​within the image and is primarily used in edge enhancement and blur analysis in computer vision. Specifically, for a given two-dimensional image function f(a,b), where a represents the horizontal coordinates of the image and b represents the vertical coordinates, its Laplacian operator is expressed as the following second-order partial differential equation:

[0144]

[0145] The second-order differential of the Laplacian operator with respect to the two-dimensional image function f(a,b) is decomposed into partial derivatives along the spatial coordinate axes, and its expansion is as follows:

[0146]

[0147] Δf(a,b) expands as follows:

[0148]

[0149] Based on the above differential expression, a corresponding Laplacian convolution kernel is constructed. This operator uses a 3×3 neighborhood calculation method, taking the pixel to be processed (a,b) as the core, and performing second-order difference operations on the gray values ​​of its four neighborhoods. The template operator is defined as follows:

[0150] Δf(a,b)=f(a+1,b)+f(a-1,b)+f(a,b+1)+f(a,b-1)-4f(a,b) (31)

[0151] Based on the aforementioned second-order differential equation, the corresponding Laplacian convolution operator is derived. This operation template adopts a centrally symmetric structure, taking the pixel point (a,b) to be processed as the reference, and performs second-order differential calculation on the pixel values ​​of its four adjacent orthogonal directions (i.e., horizontal and vertical).

[0152] The process for filtering blurry images is as follows: Figure 4 As shown, the method for filtering collision accident / normal driving images is as follows:

[0153] Step 3-2-1: Read the image files sequentially from the video extraction folder;

[0154] Step 3-2-2: Check if the current image is the last image in the folder. If it is, terminate the entire filtering process; otherwise, proceed with the subsequent processing.

[0155] Step 3-2-3: Convert the color image to grayscale and apply the Laplacian operator to calculate the image sharpness index;

[0156] Step 3-2-4: Determine the ambiguity of the grayscale image using a preset threshold;

[0157] Step 3-2-5: After the image results in Step 3-2-4 meet the resolution standards, store them in the corresponding directory and the system will automatically load the next image to be processed.

[0158] Step 3-2-6: Terminate the entire filtering process when the last image is detected.

[0159] Step 3-3: Image enhancement processing is performed using homomorphic filtering. First, the image selected in Step 3-2 is modeled as the product of illuminance and reflectance components. The illuminance component reflects changes in illumination, and the reflectance component reflects object characteristics. After converting the illuminance and reflectance components to the frequency domain using Fourier transform, different filter parameters are applied for processing: the low-frequency illuminance component is attenuated, and the high-frequency reflectance component is enhanced. After frequency domain processing, the image is restored to the spatial domain using inverse Fourier transform, finally obtaining the enhanced output result. Specific steps are as follows... Figure 5 As shown.

[0160] Each image is represented as:

[0161] f(a,b)=f i (a,b)×f r (a,b) (32)

[0162] In the formula, fi (a,b) represents the illumination of the image; f r (a,b) represents the reflection intensity of the graph;

[0163] By performing a logarithmic transformation on the original image, its original multiplicative features are transformed into additive features. The transformation result is expressed as follows:

[0164] lnf(a,b)=lnf i (a,b)+lnf r (a,b) (33)

[0165] After performing a Fourier transform, the signal will be mapped to the frequency domain; a frequency domain filter is then used for filtering, and the mathematical expression of this filter in the frequency domain is:

[0166] H(u,e)=(γ H -γ L )H hp (u,e)+γ L (34)

[0167] In the formula, u is the horizontal frequency coordinate in the frequency domain, e is the vertical frequency coordinate in the frequency domain, and γ... H γ L To adjust the amplitude response range of the filter, H hp (u,e) represents the designed frequency domain filter;

[0168] Steps 3-4: Use the YOLO (You Only Look Once) model to annotate road vehicles and pedestrians. YOLO is a one-stage object detection algorithm based on a convolutional neural network architecture, offering high detection speed and accuracy. The YOLO model consists of four core modules: Input, Backbone, Neck, and Head. The Input preprocessing module is primarily responsible for preprocessing the raw data. The Backbone module is crucial for extracting multi-level feature representations from the input image. The Neck module integrates feature information at different scales through structures such as feature pyramids, and also acts as a bridge between the Backbone module and the Head. The Head module not only performs feature decoding but also generates the target bounding box coordinates and their corresponding class probability distributions.

[0169] The YOLO model is used to annotate road vehicles and pedestrians, and the specific formula is as follows:

[0170]

[0171] In the formula, F(a,b) is an element in the feature map, I(a+m,b+n) is a local region of the input image, K(m,n) is the convolution kernel, and m and n are the dimension indices of the convolution kernel K;

[0172] In this invention, the objects with the highest risk of vehicle collision are vehicles and pedestrians on the road. The categories "car," "trunk," and "person" are selected as the detection output, which includes the detection category and the four coordinates of the detection box. The YOLO model output format is [a1, b1, w1, h1, conf], where a1 and b1 are the coordinates of the detection box's center point, w1 and h1 are the width and height of the detection box, and conf is the confidence score of the current detection box. The subsequent large language model input requires the coordinates of the four corner points of the detection box [a_min, b_min, a_max, b_max], which are converted using the following formula.

[0173]

[0174] Based on the transformed detection box corner coordinates and corresponding category information, and combined with the key causal themes of collision accidents extracted through XGBoost model and SHAP analysis in step 2, a collision risk assessment reasoning chain (CoT) is constructed. Then, it is associated with the original annotations and labeling information in the vehicle driving accident dataset (CCD) to generate image description prompts containing "environmental features + target information + risk reasoning". The generated image description prompts are then matched one-to-one with the collision accident / normal driving images filtered in step 3-2 and labeled in step 3-4 to form standardized image-text pairs. Finally, all image-text pairs are divided into training and validation subsets according to a preset ratio, completing the construction of the large-scale visual model training dataset.

[0175] Large language models can significantly improve their performance when dealing with complex reasoning problems by generating intermediate derivation processes. This prompting strategy, which encourages the model to gradually output reasoning logic, is called the Chain of Thought (CoT) method in research. A schematic diagram of the collision risk assessment CoT architecture is shown below. Figure 6 As shown, the specific explanation is as follows:

[0176] (1) Description of driving scenarios

[0177] Vehicle driving environment (such as weather and road conditions) has a certain impact on vehicle collision risk. Therefore, the model is first prompted to output a linguistic description of the driving environment, E = {E...} weather E time E road E lane It includes four conditions, each representing a key aspect affecting the environment.

[0178] E weather This indicates the weather conditions in which the vehicle is driving, including sunny, snowy, rainy, or foggy weather. Some adverse weather conditions may increase the risk of a collision due to reduced visibility and road grip.

[0179] E time This represents the time of day, including both daytime and nighttime driving. Nighttime driving is characterized by reduced visibility, requiring cautious driving strategies.

[0180] E road The road types are categorized into urban roads, national highways, and expressways, each presenting unique challenges.

[0181] E lane This indicates the driving lane and determines the vehicle's current lane. This type of information is crucial for making correct lane selection and ensuring the safety of lane changes.

[0182] (2) Key object identification

[0183] Besides environmental conditions, various objects in a driving scenario significantly impact a vehicle's collision risk. Unlike traditional autonomous driving perception modules (which detect all objects within a specific range), we focus solely on identifying the key objects most likely to affect the current situation—those with the most significant impact on current driving risk. Each key object contains two attributes: the object category and its approximate bounding box coordinates on the image (a_min, b_min, a_max, b_max). The category and coordinates are mapped linguistically to their corresponding linguistic token_id, enabling seamless integration into subsequent modules.

[0184] (3) Collision risk assessment

[0185] First, a key object analysis is required. After identifying the key objects, analyze their characteristics and potential impact on the vehicle. Characteristics include three aspects of the key objects: static attributes C... s motion state C m and specific behavior C b Static property C s Describing the inherent characteristics of an object, such as visual cues from a roadside billboard or oversized cargo on a truck, is crucial to understanding its impact on collision risk. State of Motion C m The dynamics of the object are described, including its position, orientation, and motion. These dynamics are crucial for predicting the object's future trajectory and potential interactions with its own vehicle. Specific behavior C b This refers to specific actions or gestures that may directly influence one's next driving decision. We do not need a model to analyze the three features (C) of all objects.s C m C b In reality, most of the time only one or two are applicable to key objects.

[0186] Based on the previously processed labels and the collision risk assessment reasoning chain, and considering the usage characteristics of large models, the instructions for constructing a training dataset for large visual models are rebuilt. This dataset construction is divided into two phases: inference phase dataset instructions and fine-tuning phase dataset instructions.

[0187] (1) Dataset instructions during the inference phase

[0188] During the large model inference process, the required input and target output dataset instructions need to be combined with the system role prompts and instruction prompts in the dataset, and a prompt message needs to be added, while using the dialogue content expression method from the pre-training stage.

[0189] (2) Fine-tuning stage dataset instructions

[0190] During the fine-tuning of the large model, the required input and target output dataset instructions must be found in the dataset instructions from the inference stage mentioned above: <s> Assistant:...< / s> Standard answer text is added to the model and used as a target for fine-tuning to optimize the parameters of the LoRA module in the large model through backpropagation of gradients.

[0191] Step 4: Collision risk assessment based on the visual large language model. Fine-tuning training of the visual large language model.

[0192] The visual large language model is expressed by the following formula:

[0193] LLM(O,V)=f ε (E O (O),E V (V)) (37)

[0194] In the formula, O represents text input, V represents visual input, and ε represents the set of learnable parameters for the visual large language model.

[0195] E O E is a text embedding function. V For visual embedding functions, f ε Let ε represent the function of the visual large language model;

[0196] Fine-tuning of the visual large language model is achieved through the LoRA model, and the specific method is as follows:

[0197] Assuming W is the weight matrix in the Transformer layer, LoRA decomposes it into...

[0198] W = W0 + UN T (38)

[0199] In the formula, W0 is the original weight matrix, U and N are low-order matrices learned during fine-tuning, and T represents the transpose of matrix N.

[0200] This adaptation is used in the Transformer's attention and feedforward layers. The attention layer is the module that implements the interactive computation of attention weights through "query (Q), key (F), value (G)"; the feedforward layer is a fully connected feedforward network following the attention layer, used for further feature transformation of the attention output, enabling the model to learn task-specific patterns while utilizing the extensive knowledge it acquires during pre-training. The mathematical representation of the adapter in the attention layer is:

[0201]

[0202] In the formula, Q is the query vector matrix, F is the key vector matrix, G is the value vector matrix, and ΔF is the change in the key matrix learned during fine-tuning; d F is the dimension of the embedded vector.

[0203] After completing the LoRA adaptation of the attention layer and the feedforward layer, the visual large language model is fine-tuned using the large visual model training dataset constructed in step 3: the image is input into the visual embedding module E. V To generate visual features, the text prompt word input text embedding module E T To generate text features, and then pass them through the model function f ε Multimodal feature fusion and training are carried out, and the specific pattern of collision risk assessment task is learned by using the low-rank matrices U and N of LoRA, while retaining the general knowledge accumulated in the pre-training stage.

[0204] After fine-tuning, the traffic scene image to be evaluated is input into the visual large language model: First, the YOLO model detects vehicles and pedestrians in the image and converts them into the coordinates of the four corners of the detection box. Combined with scene information, text prompts containing environmental descriptions, target information, and collision risk inference logic are generated. Then, the image and prompts are input into the model together, and the model uses LLM(O,V)=f ε (E O (O),E V (V)) Completes multimodal reasoning and finally outputs the collision risk in JSON format, thus realizing the assessment of collision risk.

[0205] Step 5: Build a joint simulation platform to conduct risk assessment simulation verification.

[0206] Select ScaneR TMThe studio simulation platform was used for risk assessment, and PyTorch was used as the computational framework for neural network modeling. The experimental environment was configured, and three typical collision risk scenarios were constructed: a rear-end collision scenario where the vehicle in front suddenly decelerates, causing a forward rear-end collision; a lane-changing scenario where the vehicle is traveling straight and another vehicle changes lanes, causing a conflict; and a merging scenario where the vehicle changes lanes and merges into another lane, causing a conflict with vehicles traveling in the original lane. The Collision Risk Index (CPI) was selected as the assessment standard for vehicle collision risk, expressed by the following formula:

[0207]

[0208] In the formula, For vehicles Potential collision index, For vehicles The initial time interval, in seconds; For vehicles The final time interval, in seconds; Δ t The observation time interval is expressed in seconds. For vehicles The total simulation time, in seconds; An index for vehicles, used to identify different vehicles. For vehicles The dynamic risk assessment coefficient at time t is used to quantify the vehicle. Real-time collision risk level at time t For vehicles The minimum acceptable dynamic risk at time t, where σ is the risk weighting coefficient.

[0209] The K-means clustering method was used to classify the risk level of vehicles into three categories: low risk, medium risk, and high risk. The distance between the three cluster centers was used as a threshold grading index to classify the three collision risk levels. The accuracy of the visual large language model before and after training and other machine learning models in assessing collision risk in various scenarios was compared to verify the effectiveness and necessity of model training.

[0210] The present invention also relates to a system for an autonomous vehicle collision risk assessment method based on a visual large language model, the system comprising a computer module for running the autonomous vehicle collision risk assessment method based on a visual large language model.

[0211] Example

[0212] A collision risk assessment method for autonomous vehicles based on a visual large language model includes the following steps:

[0213] Step 1: Model the collision incident narrative dataset using a structural topic model.

[0214] The collision accident dataset uses statistical text data from autonomous driving collision accidents reported by the California Department of Motor Vehicles (DMV) from 2018 to 2024. Collision accident reports from the dataset were downloaded as PDFs, and the necessary data was extracted using the data parsing tool pdftools. This data included a natural language description of the collision, the time of the collision, its severity, and other relevant information. Invalid information, including stop words, punctuation, and numbers, was removed. Other words commonly found in collision accident reports but not providing useful information were also removed from the descriptions, including words related to location, manufacturer, and report. After removing invalid information, the resulting collision accident description dataset was obtained and saved as a CSV file. The text removed from the collision accident descriptions is shown in Table 1.

[0215] Table 1. Text removed from collision accident descriptions

[0216]

[0217]

[0218] This study uses the residual index as an evaluation method for model fit to determine the appropriate number of topics. To select an STM model with relatively small residuals, test experiments were conducted using different numbers of topics. Considering the performance of models with different numbers of topics, a model with 15 topics was chosen because it achieved the lowest residual value.

[0219] After determining the number of topics, a topic model was used to extract target topics and topic words for each topic. This study used frequency, FREX (Frequency Exclusion), and Lift to interpret the topics. The frequency of words in a topic was estimated based on the distribution of topic words; FREX weighted words by their total frequency and their uniqueness to the topic; the Lift value was a quantitative indicator obtained by comparing the frequency of a specific word in the actual corpus with its expected probability in the baseline distribution. Table 2 below shows the extracted topics and the probability of topic words in a single topic, along with the top three words for FREX and Lift.

[0220] Table 2. Subject keyword distribution table

[0221]

[0222] Based on the words they contain, the topics are categorized into behavior-related topics, transportation-related topics, location-related topics, and other topics. After converting the collision incident narrative datasets into topics, each collision incident narrative dataset can be represented by 15 topics at different proportions. This process transforms unstructured collision incident narrative datasets into structured data, resulting in an N-by-15 matrix, where N represents the number of collision incident narratives.

[0223] Step 2: Use the XGBoost model and Shapley explanation to establish the relationship between the subject and the severity of the collision.

[0224] To develop the XGBoost model, the data, including the collision accident topic distribution extracted by the structural topic model and the severity of autonomous driving collision accidents, was first randomly divided into training and testing data, and this division process was repeated five times. Furthermore, a synthetic minority class oversampling technique was used to balance the training data to overcome the data imbalance problem. Then, for each training and testing set obtained from the previous division, the XGBoost model was trained on the training data, and its hyperparameters were determined according to... Figure 2 The process shown is optimized; the topic distribution matrix of the collision accident narrative is used as the input of the XGBoost model, and the collision accident severity in the metadata is used as the output of the model. Training and test sets are constructed to train an XGBoost model that can evaluate the severity of collision accidents.

[0225] To further clarify the relationship between themes and the severity of collision accidents, and to determine the impact of each theme on the SHAP value of a single collision accident severity, SHAP analysis was conducted on collision accidents with no injuries, minor injuries, and moderate to severe injuries. The characteristics of key themes for collision severity classification were summarized, and the four themes with the greatest impact on high-injury accidents and the three themes with the greatest impact on moderate-injury accidents were selected as key causative themes, serving as the core components of the subsequent model reasoning thought chain.

[0226] Step 3: Establish a large visual model training dataset for collision risk assessment.

[0227] The Car Crush Datasets (CCD) are created by capturing real traffic accident videos and their associated annotations using vision sensors installed on driving vehicles. The CCD includes collision accident datasets and regular driving datasets. Each collision accident video is 5 seconds long, containing 50 frames, and played back at 10 frames per second. To further enrich the video content, time stamps, environmental attributes (such as day or night, snow, rain, or clear weather), information on whether the vehicle was involved, details of the accident participants, and a description of the accident's cause were added. The videos of vehicles driving normally are 3000 randomly selected vehicle driving segments and their labels from the BDD100K autonomous driving dataset. Table 3 shows the basic information of the collision accident dataset.

[0228] Table 3. Basic Information on the Collision Accident Dataset

[0229] Collision accident 1500 5s 10 15000 Normal driving 3000 5s 10 30000

[0230] The core files of the CCD dataset consist of two parts. The first part contains 5-second videos, including videos of normal driving and videos of collisions. The second part contains video annotations, including annotations at the entire video level, annotations of detected instances within the video, and annotations of the time of the accident.

[0231] To improve the image quality in subsequent experimental phases, and considering that avoiding vehicle collisions in the real world requires not only considering vehicles ahead, but also pedestrians and adhering to current road rules, it is necessary to re-annotate the video data and perform image enhancement operations.

[0232] Step 3-1: Extract regular driving video frames as training images for low-collision-risk scenarios. For collision accident images, since assessing collision risk after a collision is irrelevant, select video frames before the collision based on time stamps. The extracted collision accident image dataset contains accident time stamps, for example, [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1], where 0 represents no accident in this frame and 1 represents an accident in this frame. The open-source multimedia processing tool FFmpeg is used to extract the video frames.

[0233] Step 3-2: Addressing the issue of blurred images in the original CCD dataset, this invention employs image sharpness-based data filtering. An image sharpness evaluation method is introduced during the data preprocessing stage to automatically filter training samples. The Laplacian operator is used to filter collision / normal driving images. This operator, based on the second-order differential principle, effectively quantifies abrupt changes in grayscale values ​​within the image and is primarily used in edge enhancement and blur analysis in computer vision. Specifically, for a given two-dimensional image function f(a,b), where a represents the horizontal coordinates of the image and b represents the vertical coordinates, its Laplacian operator is expressed as the following second-order partial differential equation:

[0234]

[0235] The second-order differential of the Laplacian operator with respect to the two-dimensional image function f(a,b) is decomposed into partial derivatives along the spatial coordinate axes, and its expansion is as follows:

[0236]

[0237] Δf(a,b) expands as follows:

[0238]

[0239] Based on the above differential expression, a corresponding Laplacian convolution kernel is constructed. This operator uses a 3×3 neighborhood calculation method, taking the pixel to be processed (a,b) as the core, and performing second-order difference operations on the gray values ​​of its four neighborhoods. The template operator is defined as follows:

[0240] Δf(a,b)=f(a+1,b)+f(a-1,b)+f(a,b+1)+f(a,b-1)-4f(a,b) (44)

[0241] Based on the aforementioned second-order differential equation, the corresponding Laplacian convolution operator is derived. This operation template adopts a centrally symmetric structure, taking the pixel point (a,b) to be processed as the reference, and performs second-order differential calculation on the pixel values ​​of its four adjacent orthogonal directions (i.e., horizontal and vertical).

[0242] The images extracted from each video in the car driving accident dataset are processed and filtered as follows:

[0243] Step 3-2-1: Read the image files sequentially from the video extraction folder;

[0244] Step 3-2-2: Check if the current image is the last image in the folder. If it is, terminate the entire filtering process; otherwise, proceed with the subsequent processing.

[0245] Step 3-2-3: Convert the color image to grayscale and apply the Laplacian operator to calculate the image sharpness index;

[0246] Step 3-2-4: Determine the ambiguity of the grayscale image using a preset threshold;

[0247] Step 3-2-5: After the image results in Step 3-2-4 meet the resolution standards, store them in the corresponding directory and the system will automatically load the next image to be processed.

[0248] Step 3-2-6: Terminate the entire filtering process when the last image is detected.

[0249] Step 3-3: Image enhancement processing is performed using homomorphic filtering. First, the input image filtered in Step 3-2 is modeled as the product of illuminance and reflectance components. The illuminance component reflects changes in illumination, and the reflectance component reflects object characteristics. After converting the illuminance and reflectance components to the frequency domain using Fourier transform, different filter parameters are applied: the low-frequency illuminance component is attenuated, and the high-frequency reflectance component is enhanced. After frequency domain processing, the image is restored to the spatial domain using inverse Fourier transform, finally obtaining the enhanced output result. Specific steps are as follows... Figure 5 As shown.

[0250] Each image is represented as:

[0251] f(a,b)=f i (a,b)×f r (a,b) (45)

[0252] In the formula, f i (a,b) represents the illumination of the image; f r (a,b) represents the reflection intensity of the graph;

[0253] By performing a logarithmic transformation on the original image, its original multiplicative features are transformed into additive features. The transformation result is expressed as follows:

[0254] lnf(a,b)=lnf i (a,b)+lnf r (a,b) (46)

[0255] After performing a Fourier transform, the signal will be mapped to the frequency domain; a frequency domain filter is then used for filtering, and the mathematical expression of this filter in the frequency domain is:

[0256] H(u,e)=(γ H -γ L )H hp (u,e)+γ L (47)

[0257] In the formula, u is the horizontal frequency coordinate in the frequency domain, e is the vertical frequency coordinate in the frequency domain, and γ... H γ L To adjust the amplitude response range of the filter, H hp (u,e) represents the designed frequency domain filter.

[0258] Steps 3-4: Use the YOLO (You Only Look Once) model to annotate road vehicles and pedestrians. YOLO is a one-stage object detection algorithm based on a convolutional neural network architecture, offering high detection speed and accuracy. The YOLO model consists of four core modules: Input, Backbone, Neck, and Head. The Input preprocessing module is primarily responsible for preprocessing the raw data. The Backbone module is crucial for extracting multi-level feature representations from the input image. The Neck module integrates feature information at different scales through structures such as feature pyramids, and also acts as a bridge between the Backbone module and the Head. The Head module not only performs feature decoding but also generates the target bounding box coordinates and their corresponding class probability distributions.

[0259] The YOLO model is used to annotate road vehicles and pedestrians, and the specific formula is as follows:

[0260]

[0261] In the formula, F(a,b) is an element in the feature map, I(a+m,b+n) is a local region of the input image, K(m,n) is the convolution kernel, and m and n are the dimension indices of the convolution kernel K;

[0262] In this invention, the objects with the highest risk of vehicle collision are vehicles and pedestrians on the road. The categories "car," "trunk," and "person," with a confidence level of 0.8 or higher, are selected as the detection output. The output includes the detection category and the four coordinates of the detection box. The YOLO model output format is [a1, b1, w1, h1, conf], where a1 and b1 are the coordinates of the detection box's center point, w1 and h1 are the width and height of the detection box, and conf is the confidence level of the current detection box. The subsequent large language model input requires the coordinates of the four corner points of the detection box [a_min, b_min, a_max, b_max], which are converted using the following formula.

[0263]

[0264] Large language models can significantly improve their performance when dealing with complex reasoning problems by generating intermediate derivation processes. This prompting strategy, which encourages the model to gradually output reasoning logic, is called the Chain of Thought (CoT) method in research. A schematic diagram of the collision risk assessment CoT architecture is shown below. Figure 6 As shown, the specific explanation is as follows:

[0265] (1) Description of driving scenarios

[0266] Vehicle driving environment (such as weather and road conditions) has a certain impact on vehicle collision risk. Therefore, the model is first prompted to output a linguistic description of the driving environment, E = {E...} weather E time E road E lane It includes four conditions, each representing a key aspect affecting the environment.

[0267] E weather This indicates the weather conditions in which the vehicle is driving, including sunny, snowy, rainy, or foggy weather. Some adverse weather conditions may increase the risk of a collision due to reduced visibility and road grip.

[0268] E time This represents the time of day, including both daytime and nighttime driving. Nighttime driving is characterized by reduced visibility, requiring cautious driving strategies.

[0269] E road The road types are categorized into urban roads, national highways, and expressways, each presenting unique challenges.

[0270] E lane This indicates the driving lane and determines the vehicle's current lane. This type of information is crucial for making correct lane selection and ensuring the safety of lane changes.

[0271] (2) Key object identification

[0272] Besides environmental conditions, various objects in a driving scenario significantly impact a vehicle's collision risk. Unlike traditional autonomous driving perception modules (which detect all objects within a specific range), we focus solely on identifying the key objects most likely to affect the current situation—those with the most significant impact on current driving risk. Each key object contains two attributes: the object category and its approximate bounding box coordinates on the image (a_min, b_min, a_max, b_max). The category and coordinates are mapped linguistically to their corresponding linguistic token_id, enabling seamless integration into subsequent modules.

[0273] (3) Collision risk assessment

[0274] First, a key object analysis is required. After identifying the key objects, analyze their characteristics and potential impact on the vehicle. Characteristics include three aspects of the key objects: static attributes C... s motion state C m and specific behavior C b Static property C s Describing the inherent characteristics of an object, such as visual cues from a roadside billboard or oversized cargo on a truck, is crucial to understanding its impact on collision risk. State of Motion C m The dynamics of the object are described, including its position, orientation, and motion. These dynamics are crucial for predicting the object's future trajectory and potential interactions with its own vehicle. Specific behavior C b This refers to specific actions or gestures that may directly influence one's next driving decision. We do not need a model to analyze the three features (C) of all objects. s C m C b In reality, most of the time only one or two are applicable to key objects.

[0275] Based on the previously processed labels and the collision risk assessment reasoning chain, and considering the usage characteristics of large models, the instructions for constructing a training dataset for large visual models are rebuilt. This dataset construction is divided into two phases: inference phase dataset instructions and fine-tuning phase dataset instructions.

[0276] (1) Dataset instructions during the inference phase

[0277] During the large model inference process, the required input and target output dataset instructions need to be combined with the system role prompts and instruction prompts in the dataset, and a prompt message needs to be added, while using the dialogue content expression method from the pre-training stage.

[0278] Examples of clues for the reasoning stage are as follows:

[0279] Prompt words

[0280] <s>System: You are an autonomous vehicle driving on the road. Please assess the collision risk based on the following prompts.

[0281] < / s> <s>Human: Reasoning Process: First, determine the current driving scenario, including weather conditions, road conditions, driving time, and lane conditions; then, identify obstacles ahead and determine the distance between the vehicle and the obstacle vehicle or pedestrian based on the detection box. Considering the overall driving scenario, analyze the characteristics of potential obstacles ahead and their impact on the vehicle to determine the current collision risk.

[0282] Requirements: Follow the prompts step by step, represent any content that cannot be determined as "unknown", and finally output the collision risk in JSON format.

[0283] < / s> <s>assistant:

[0284] (2) Fine-tuning stage dataset instructions

[0285] During the fine-tuning of the large model, the required input and target output dataset instructions must be found in the dataset instructions from the inference stage mentioned above: <s> Assistant:...< / s> Standard answer text is added to the model and used as a target for fine-tuning to optimize the parameters of the LoRA module in the large model through backpropagation of gradients.

[0286] Examples of prompts during the training phase are as follows:

[0287] Prompt words

[0288] <s>System: You are an autonomous vehicle driving on the road. Please assess the collision risk based on the following prompts.

[0289] < / s> <s>Human: Reasoning Process: First, determine the current driving scenario, including weather conditions, road conditions, driving time, and lane conditions; then, identify obstacles ahead and determine the distance between the vehicle and the obstacle vehicle or pedestrian based on the detection box. Considering the overall driving scenario, analyze the characteristics of potential obstacles ahead and their impact on the vehicle to determine the current collision risk.

[0290] Requirements: Follow the prompts step by step, represent any content that cannot be determined as "unknown", and finally output the collision risk in JSON format.

[0291] < / s> <s>assistant: The current weather conditions are... <weather>The current road conditions are as follows: <road>Current driving time is <day>There are / are no obstructing vehicles and pedestrians detected ahead. The detection frame boundary is...<a1,b1,a2,b2> The obstacle is <car>The potential impact on the current vehicle is <reason>Based on the above analysis, the current collision risk is assessed as follows: <high low>.json{"collision risk":"high risk"}

[0292] The processed dataset contains 4,500 videos, which are divided into 3,600 training videos and 900 test videos, as shown in Table 4.

[0293] Table 4. Information on the processed dataset

[0294]

[0295] Step 4: Collision risk assessment based on the visual large language model. Fine-tuning training of the visual large language model.

[0296] The Qianwen model demonstrates higher accuracy in collision risk assessment than other models; therefore, the Qianwen Visual Large Language Model was chosen as the pre-training model for this invention. The Qianwen model offers three versions: 72 billion parameters, 7 billion parameters, and 2 billion parameters. The 72 billion parameter model exhibits the strongest performance. Due to limited device computing power and the fact that autonomous driving assessment is a task highly sensitive to inference speed, considering both model performance and efficiency, Qwen2-VL-7B was selected as the base model for training this invention. When using Qwen2-VL as the pre-training model for the collision risk assessment task, the data format needs to be aligned to the format used during pre-training before fine-tuning.

[0297] In the input data, Qwen2-VL uses special tokens to distinguish between visual and text input. Tokens <|vision_start|> and <|vision_end|> are inserted at the beginning and end of the image feature sequence to segment the image content.

[0298] The model's input format is divided into two types: dialogue format and visual localization format. Because the collision risk task is very sensitive to the position of obstacles in front, it is necessary to combine the two formats.

[0299] LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method specifically designed for large-scale pre-trained language models (including architectures such as BERT and GPT). Due to its high fine-tuning effect and ability to preserve the knowledge of the model's pre-training, as well as its strong adaptability, this invention chooses LoRA as a fine-tuning technique for large language models.

[0300] The visual large language model is expressed by the following formula:

[0301] LLM(O,V)=f ε (E O (O),E V (V)) (50)

[0302] In the formula, O represents text input, V represents visual input, and ε represents the set of learnable parameters for the visual large language model.

[0303] E O E is a text embedding function. V For visual embedding functions, f ε Let ε be a function representing the visual large language model.

[0304] Fine-tuning of the visual large language model is achieved through the LoRA model, and the specific method is as follows:

[0305] Assuming W is the weight matrix in the Transformer layer, LoRA decomposes it into...

[0306] W = W0 + UN T (51)

[0307] In the formula, W0 is the original weight matrix, U and N are low-order matrices learned during fine-tuning, and T represents the transpose of matrix N.

[0308] This adaptation is used in the Transformer's attention and feedforward layers. The attention layer is the module that implements the interaction of "query (Q), key (F), value (G)" to calculate attention weights; the feedforward layer is a fully connected feedforward network following the attention layer, used for further feature transformation of the attention output, enabling the model to learn task-specific patterns while utilizing the vast amount of knowledge it acquires during pre-training. The mathematical representation of the adapter in the attention layer is:

[0309]

[0310] In the formula, Q is the query vector matrix, F is the key vector matrix, G is the value vector matrix, and ΔF is the change in the key matrix learned during fine-tuning; d F is the dimension of the embedding vector.

[0311] The hyperparameters are set as follows in this embodiment:

[0312] Learning rate: Set to 3 × 10 -4 This is a relatively low learning rate, which helps with stable learning during fine-tuning and avoids overreacting to noise in individual data points when adjusting model weights. A cosine with restarts learning rate adjustment strategy is adopted to achieve dynamic adaptive adjustment of the learning rate.

[0313] Batch size (training batch size): Set to 2. This parameter determines the amount of data processed in a single iteration. A smaller batch size helps reduce memory requirements and improves the model's generalization ability on different data samples.

[0314] Num train epochs: Setting this to 50 means the entire training dataset will be processed 50 times. This number of epochs is set based on a balance between model convergence and avoiding overfitting.

[0315] Lora r (Lora parameter r): Set to 8, indicating the rank used for parameterization reparameter techniques. This helps increase the model's learning capacity without significantly increasing the model size.

[0316] Max seq (maximum sequence length): Set to 2048, which is the maximum length of the input text sequence that the model can process.

[0317] The hyperparameter settings are shown in Table 5.

[0318] Table 5 Hyperparameter Settings Table

[0319]

[0320]

[0321] Model training is performed using half-precision floating-point numbers (float16). During the training process... Figure 7 This shows the changes in loss and learning rate for each training round. A model checkpoint is saved every 5 rounds. The graph shows that by the 30th round, the loss has decreased to a very low level, and subsequent checkpoints can be used as alternative models.

[0322] The evaluation framework was chosen from the LlamaFactory large language model library, which includes a pre-built evaluation dataset and evaluation system. For the evaluation criteria, BLEU-4 was selected for the classic image captioning task, while VQA-v2 was chosen for the visual question answering task. The model checkpoints, saved every 5 rounds after 30 rounds, were evaluated using both evaluation criteria. The model at round 30 performed better on both criteria. Therefore, the checkpoint saved at round 30 was selected as the final model used.

[0323] Four key metrics were selected to comprehensively evaluate the model performance, including classification accuracy (Accuracy), precision (Precision, P), recall (Recall, R), and F1 score.

[0324] The inference parameters of a visual large language model are key elements controlling the quality, diversity, and computational efficiency of the model's output, directly affecting the applicability of the generated results. By adjusting the model's decision-making mechanism during the generation process, the large model can adapt to the needs of different application scenarios. Core inference parameters include temperature coefficient, Top-k and Top-p sampling, and repetition penalty, which interact during the generation process to form a complex control network. Choosing appropriate parameter values ​​is crucial for fully utilizing the potential of large language models.

[0325] Testing was conducted within the range of 0 to 1, evaluating the model's collision risk assessment and classification performance under different temperature parameters, and selecting the parameter values ​​with the best performance. Accuracy was used as the evaluation metric to obtain the evaluation results for different parameter values. The model performed well when the temperature parameter was between 0 and 0.8, with the highest accuracy at 0.2. However, a significant drop occurred when the temperature parameter was greater than 0.8. This is because when the temperature parameter is set too high, the generation randomness becomes too large, and the error of each subsequent token accumulates in a multiplicative manner.

[0326] Top_k and Top_p are sampling strategy parameters that control the diversity of generation. The model performed best when Top_k = 0.2 and Top_p = 0.2, and therefore, they were used as the selected parameters in subsequent experiments.

[0327] The repetition penalty coefficient is a parameter used to reduce duplicate content in the generated text. It applies a probability penalty to the generated words, reducing the likelihood of their recurrence. Its value ranges from 1 to 2. When the coefficient is 1, there is no penalty. The larger the coefficient, the stronger the suppression of duplicate words. The evaluation accuracy under different penalty coefficients was obtained. The accuracy performance was the best when the penalty coefficient was 1.0, which is the default no repetition penalty. Therefore, this value was adopted in the subsequent experiments.

[0328] The model's collision risk assessment classification performance was tested using a test set. After the model outputs results, regular expressions were used to extract the collision risk assessment classifications from the output JSON and perform statistical analysis. Based on the evaluation metrics constructed above, the model's classification performance was evaluated. Table 6 compares the trained model with different models on different evaluation metrics.

[0329] Table 6 Comparison of Experimental Results

[0330]

[0331] Step 5: Build a joint simulation platform to conduct risk assessment simulation verification.

[0332] This embodiment constructs a virtual test road segment encompassing various road types, integrating typical road scenarios such as urban roads, highways, and national and provincial trunk roads. The spatial layout relationships of each road type can be found in [reference needed]. Figure 8 Overall plan view. Based on the established terrain and road design scheme, using SCANeR... TM The SCENARIO module of the Studio software platform has completed the parametric modeling of traffic flow characteristics.

[0333] To realistically reflect actual road traffic characteristics, the implementation examples constructed diverse combinations of traffic parameters, including key elements such as traffic flow levels, driving speeds, and vehicle composition. In the urban road simulation environment, a high-saturation mixed vehicle configuration was adopted, covering typical categories such as passenger cars, buses, and engineering vehicles; while the highway simulation scenario focused on reflecting the characteristics of high-speed, homogeneous traffic flow under low-flow conditions; and the ordinary trunk road scenario mainly simulated heterogeneous traffic states with significant speed differences. All simulation scenarios incorporated a dynamic adjustment mechanism for meteorological parameters, thereby achieving a comprehensive digital reconstruction of the real traffic environment. Specific road traffic characteristics are shown in Table 7.

[0334] Table 7 Characteristics of Road Traffic Operation

[0335] City roads 4 Buses, motorcycles, bicycles, and pedestrians 40～60 highway 6 Buses and trucks 120 trunk highway 2 Buses and trucks of all sizes, motorcycles 60

[0336] The implementation examples construct three typical traffic conflict scenarios, as shown in Figure 9. Figure 9a The conflict situation is the risk of a frontal rear-end collision caused by the vehicle in front suddenly slowing down; Figure 9b The conflict occurred when this vehicle was traveling straight and another vehicle from another lane changed lanes and merged into it. Figure 9c The conflict occurs when the vehicle changes lanes and merges into another lane, causing a collision with vehicles traveling in the original lane.

[0337] In the field of vehicle safety, commonly used quantitative indicators for assessing collision risk include Time to Collision (TTC), Distance to Collision (DTC), and Deceleration Rate to Avoid a Crash (DRAC). However, these traditional indicators have significant limitations in practical applications, primarily because they fail to adequately consider key factors such as road surface conditions and vehicle characteristics. In contrast, the Crash Potential Index (CPI), by integrating road surface information under different weather conditions and combining it with the DRAC indicator, can achieve a more comprehensive risk assessment. Based on this, this invention selects the CPI indicator as the assessment standard for vehicle collision risk.

[0338] The Consumer Price Index (CPI) refers to the probability of exceeding the Maximum Available Deceleration Rate (MADR) or occurring within a specified timeframe. The CPI is defined as follows:

[0339]

[0340] In the formula, For vehicles Potential collision index, For vehicles The initial time interval, in seconds; For vehicles The final time interval, in seconds; Δ t The observation time interval is expressed in seconds. For vehicles The total simulation time, in seconds; An index for vehicles, used to identify different vehicles. For vehicles The dynamic risk assessment coefficient at time t is used to quantify the vehicle. Real-time collision risk level at time t For vehicles The minimum acceptable dynamic risk at time t, where σ is the risk weight coefficient;

[0341] The risk categories for indicators are categorized as follows:

[0342] Table 8. Basis for Sample Category Classification

[0343]

[0344]

[0345] After configuring the simulation environment and various simulation scenarios, the dynamic parameters of the vehicles in each scenario need to be configured. The main vehicle is controlled using a connected driving simulator, while the designed conflict vehicles have their trajectories controlled by scripts. This data is recorded as the raw data for the evaluation model dataset.

[0346] The data processing flow is as follows: First, the selected samples are integrated, followed by data preprocessing. Specifically, speed characteristics are uniformly converted to the International System of Units (m / s), and corresponding speed information is extracted based on the preceding vehicle ID. Then, the DRAC index is calculated group by group. Based on this, the CPI value is further calculated by combining road surface condition parameters. Referring to the classification criteria shown in Table 8, the hazard level of each sample is divided, and this level is used as the sample label. Finally, the labeled experimental data is output in CSV format, completing the dataset construction. The specific process is as follows: Figure 10 As shown.

[0347] To verify the collision risk assessment performance of the trained model in different scenarios, a confusion matrix was used for quantitative analysis. Figure 11a , 11b 11c represents the confusion matrix for collision risk assessment and classification using the trained visual big language model in scenarios of braking to avoid a rear-end collision while driving straight, merging into traffic from the front, and changing lanes. The figure shows the model's assessment effect on the three levels of danger.

[0348] In contrast, the collision risk assessment classification effect was evaluated using the pre-trained visual large language model, and the confusion matrix is ​​shown in Figure 12. Figure 12a , 12b 12c represents the confusion matrix used for collision risk assessment and classification by the pre-trained visual big language model in scenarios such as braking to deal with a rear-end collision while driving straight, a vehicle merging into the lane, and a vehicle changing lanes.

[0349] The collision risk assessment accuracy of the model before and after training under different risk levels in various scenarios is shown in the figure. Figure 13 .

[0350] The above description of the present invention is only a preferred embodiment of the present invention and is not intended to limit the implementation of the present invention. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection claimed in the claims.< / high> < / reason> < / car> < / day> < / road> < / weather> < / s> < / s>

Claims

1. A collision risk assessment method for autonomous vehicles based on a visual large language model, characterized in that, Includes the following steps: Step 1: Model the collision incident narrative dataset using a structural topic model; Step 2: Use the XGBoost model and Shapley explanation to establish the relationship between the subject and the severity of the collision. Step 3: Establish a large visual model training dataset for collision risk assessment; A car driving accident dataset is obtained by capturing real traffic accident data videos and their related annotations using visual sensors installed on driving vehicles. The car driving accident dataset includes a collision accident dataset and a regular driving dataset. For video images in the collision accident dataset, collision accident / normal driving image processing is performed, and the YOLO model is used to annotate pedestrians and vehicles in the images. Based on the key causes of collision accidents extracted in step 2 through the XGBoost model and Shapley explanation, a collision risk assessment reasoning chain is constructed. Combined with the annotation and labeling information in the car driving accident dataset, image description prompts are generated and combined with the processed collision accident / normal driving images to form image-text pairs, and finally a large visual model training dataset is constructed. Step 4: Collision risk assessment based on the visual large language model; fine-tuning and training of the visual large language model. Step 5: Build a joint simulation platform to conduct risk assessment simulation verification; The SCANeR™ studio simulation platform was selected for risk assessment, and PyTorch was used as the computational framework for neural network modeling to configure the experimental environment. Three typical collision risk scenarios were constructed, including a rear-end collision scenario where the vehicle in front suddenly decelerates, causing a forward rear-end collision risk; and a lane-changing scenario where the vehicle is traveling straight and other vehicles change lanes to merge, causing a conflict. This scenario involves a vehicle changing lanes and merging into another lane, resulting in a collision with vehicles in the original lane. The Collision Risk Assessment (CPI) is selected as the standard for evaluating vehicle collision risk, expressed by the following formula: (20) In the formula, For vehicles Potential collision index, For vehicles The initial time interval, in seconds; For vehicles The final time interval is in seconds; Δt is the observation time interval, in seconds. For vehicles The total simulation time, in seconds; An index for vehicles, used to identify different vehicles. For vehicles The dynamic risk assessment coefficient at time t is used to quantify the vehicle. Real-time collision risk level at time t For vehicles The minimum acceptable dynamic risk at time t This refers to the risk weighting coefficient. The K-means clustering method was used to classify the risk level of vehicles into three categories: low risk, medium risk, and high risk. The distance between the three cluster centers was used as a threshold grading index to classify the three collision risk levels. The accuracy of the visual large language model before and after training and other machine learning models in assessing collision risk in various scenarios was compared to verify the effectiveness and necessity of model training.

2. The method for assessing collision risk of autonomous vehicles based on a visual large language model according to claim 1, characterized in that, In step 1, collision accident reports are downloaded from the collision accident dataset. Data parsing tools are used to extract the required data and remove invalid information to obtain the collision accident narrative dataset. Each collision accident narrative is converted into a bag-of-words matrix and a metadata matrix. The collision accident narrative dataset is then modeled using a structural topic model, with the following topic proportions: Where, θ d Let μ be the topic distribution vector of the d-th collision incident narrative data. d For θ d The Logistic Relationship The mean vector of the Normal distribution, μ d，k Let μ be the mean vector d The k-th element, 𝑋 d Let be a 1×p-dimensional vector based on the topic popularity covariate vector; Σ be a (k-1)×(k-1) covariance matrix; 𝛾 be an 𝑝×(k−1) coefficient matrix, 𝛾 k Let K be the k-th column vector of the coefficient matrix K. Keyword distribution: Where 𝑣 is the identifier for the collision incident topic, d is the identifier for the collision incident narrative document, and β d,v k This represents the log-expected distribution of topic terms related to label k under a collision incident narrative document labeled d and a collision incident topic labeled v, where k is an identifier used to distinguish different bias terms and interaction terms. 𝑣 The expected logarithmic value of the frequency of the identifier 𝑣 for the topic of collision accidents in the document; 𝑘 𝑘 Topic-specific deviation; For covariate group bias; The interaction between topic bias and covariate bias; Topic distribution and word distribution: (3) Where n is the location identifier within the collision incident description document. This indicates the topic assignment of position identifier n in the collision incident narrative identified as d; 𝑤 d,𝑛 For word distribution based on selected topics; β d,r This represents the word probability distribution in the collision accident narrative data labeled d, where r is the label of the associated topic, and the ~ in the formula indicates that it follows a certain probability distribution.

3. The method for assessing collision risk of autonomous vehicles based on a visual large language model according to claim 2, characterized in that, In step 1, the required data includes a description of the collision in natural language, the time of the collision, its severity, and other information related to the collision; the invalid information includes stop words, punctuation marks, and numbers, as well as words related to location, manufacturer, and report; the bag-of-words matrix is ​​a numerical representation of the collision narrative understood and analyzed by the structured topic model; the metadata matrix includes the time and severity of the collision.

4. The method for assessing collision risk of autonomous vehicles based on a visual large language model according to claim 1, characterized in that, In step 2, one tree in the XGBoost model is: (4) In the formula, Let f represent the predicted value of the i-th collision accident sample, where x is the input feature vector related to the collision accident, and i represents the index of the collision accident sample; c Let F be the prediction function of the k-th regression tree, F be the space of the regression tree, and q be the leaf weight. Let x be the function that maps x to a leaf node of the regression tree. q(x) This represents the weight value corresponding to the leaf node, c is the index of the regression tree, and C is the total number of regression trees in the XGBoost model; The goal of the XGBoost model is: (5) In the formula, It is the parameter set of the XGBoost model, where l is a differentiable convex loss function, and y i Let be the true severity label of the i-th collision accident sample, and s be the total number of collision accident samples. For regularization terms; The regularization function is: (6) In the formula, Here, T is the regularization term, and T is the number of leaves. The penalty coefficient for the number of leaf nodes. leaf weights The regularization coefficient, 𝑤 is the weight vector of the leaf nodes in the regression tree; Metadata containing collision accident thematic features and autonomous driving collision accident severity is randomly divided into training and testing data. Data oversampling is used to balance the training data. An XGBoost model is used to train the training data, with the topic distribution matrix of collision accident narratives used as the input to the XGBoost model and the collision accident severity in the metadata used as the model output. Training and testing sets are constructed to train an XGBoost model to evaluate the severity of collision accidents. The trained XGBoost model is loaded, and the Shapley value interpretation method is used to calculate the SHAP value. SHAP analysis is performed according to the severity of collision accidents to extract key causal themes. The logarithmic loss used to optimize hyperparameters is: (7) In the formula, 𝑁 represents the number of samples, 𝑀 represents the number of categories, j is the category index used to traverse different categories, and 𝑦 𝑖𝑗 For a binary variable with the expected class, 𝑝 𝑖𝑗 This represents the estimated probability of the sample.

5. The method for assessing collision risk of autonomous vehicles based on a visual large language model according to claim 1, characterized in that, In step 3, the specific steps for processing the collision accident / normal driving image are as follows: Step 3-1: Extract regular driving video frame images as training images for low collision risk; for collision accident images, select video frames before the collision accident based on time annotations, and the extracted collision accident image dataset contains accident time annotations. Step 3-2: Use the Laplace operator to filter collision accident / normal driving images. For a given two-dimensional image function f(a,b), where a represents the horizontal coordinates of the image and b represents the vertical coordinates of the image, its Laplace operator is expressed in the following second-order partial differential equation form: （8) The second-order differential of the Laplacian operator with respect to the two-dimensional image function f(a,b) is decomposed into partial derivatives along the spatial coordinate axes, and its expansion is as follows: (9) ∆f(a,b) expands as follows: (10) Based on the above differential expressions (8), (9), and (10), the corresponding Laplacian convolution kernel is constructed; this operator adopts a 3×3 neighborhood calculation method, taking the pixel to be processed (a,b) as the core, and performing second-order difference operations on the gray values ​​of its four neighborhoods; the template operator is defined as follows: (11) Based on the aforementioned second-order differential equation, the corresponding Laplacian convolution operator is derived. This operation template adopts a centrally symmetric structure, taking the pixel point (a,b) to be processed as the reference, and performs second-order differential calculation on the pixel values ​​of its four adjacent orthogonal directions. Step 3-3: Each image selected in step 3-2 is represented as follows: （12） In the formula, f i (a,b) represents the illumination of the image; f r (a,b) represents the reflection intensity of the graph; By performing a logarithmic transformation on the original image, its original multiplicative features are transformed into additive features. The transformation result is expressed as follows: （13） After performing a Fourier transform, the signal is mapped to the frequency domain; a frequency domain filter is then used for filtering, and the mathematical expression of this filter in the frequency domain is: （14） In the formula, u is the horizontal frequency coordinate in the frequency domain, and e is the vertical frequency coordinate in the frequency domain. γ H、 γ L To adjust the amplitude response range of the filter, H hp (u,e) represents the designed frequency domain filter; Steps 3-4: Use the YOLO model to annotate road vehicles and pedestrians. The specific formula is as follows: （15） In the formula, 𝐹(a,b) is an element in the feature map, 𝐼(a+𝑚,b+𝑛) is a local region of the input image, 𝐾(𝑚,𝑛) is the convolution kernel, and 𝑚 and 𝑛 are the dimension indices of the convolution kernel 𝐾. The YOLO model output format is [a1, b1, w1, h1, conf], where a1 and b1 are the coordinates of the center point of the detection box, w1 and h1 are the width and height of the detection box, and conf is the confidence score of the current detection box. The input coordinates of the large language model are the coordinates of the four corner points of the detection box [a_min, b_min, a_max, b_max]. The formula conversion is as follows: (16) The collision risk assessment reasoning chain architecture is as follows: The driving environment is used to describe the driving scenario; key objects in the driving scenario that affect the collision risk of the vehicle are identified; the characteristics and potential impacts of the identified key objects on the vehicle are analyzed to achieve collision risk assessment; based on the well-structured collision risk assessment reasoning chain, a large visual model training dataset is constructed; the large visual model training dataset is divided into inference stage dataset instructions and fine-tuning stage dataset instructions. The inference stage dataset instructions are based on the required input and target output dataset instructions, by concatenating the system role prompts and instruction prompts in the dataset, adding prompts, and using the dialogue content expression method of the pre-training stage. The fine-tuning stage dataset instructions are based on the required input and target output dataset instructions, and are used as the target for fine-tuning to optimize the parameters of the LoRA module in the large model through back gradient propagation.

6. The method for assessing collision risk of autonomous vehicles based on a visual large language model according to claim 5, characterized in that, In step 3-2, the images of collision accidents / normal driving are filtered as follows: Step 3-2-1: Read the image files sequentially from the video extraction folder; Step 3-2-2: Check if the current image is the last image in the folder. If it is, terminate the entire filtering process; otherwise, proceed with the subsequent processing. Step 3-2-3: Convert the color image to grayscale and apply the Laplacian operator to calculate the image sharpness index; Step 3-2-4: Determine the ambiguity of the grayscale image using a preset threshold; Step 3-2-5: After the image results in Step 3-2-4 meet the resolution standards, store them in the corresponding directory and the system will automatically load the next image to be processed. Step 3-2-6: Terminate the entire filtering process when the last image is detected.

7. The method for assessing collision risk of autonomous vehicles based on a visual large language model according to claim 1, characterized in that, In step 4, the visual large language model is expressed by the following formula: (17) In the formula, O represents text input, and V represents visual input. This is the set of learnable parameters for a visual large language model. E O E is a text embedding function. V For visual embedding functions, For parameters A function representing a large visual language model; Fine-tuning of the visual large language model is achieved through the LoRA model, and the specific method is as follows: Assuming W is the weight matrix in the Transformer layer, LoRA decomposes it into... （18） In the formula, W0 is the original weight matrix, U and N are low-order matrices learned during fine-tuning, and T represents the transpose of matrix N; This adaptation applies to the attention layer and feedforward layer of the Transformer. The mathematical representation of the adapter in the attention layer is: （19） In the formula, Q is the query vector matrix, F is the key vector matrix, G is the value vector matrix, and ΔF is the change in the key matrix learned during fine-tuning; d F is the dimension of the embedding vector.

8. A system for implementing the collision risk assessment method for autonomous vehicles based on a visual large language model as described in any one of claims 1 to 7, characterized in that, The system includes a computer module for running a collision risk assessment method for autonomous vehicles based on a visual large language model.

Images