Method, device and equipment for predicting severity of vehicle collision accident and storage medium

Abstract

The invention discloses a vehicle collision accident severity prediction method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting the multi-modal data of a vehicle collision accident, and carrying out the preprocessing of the multi-modal data; converting the pre-processed structured numerical data into a first feature map through a Grubrum angle field method; converting the preprocessed unstructured text data into a text semantic vector through a pre-training language model, and converting the text semantic vector into a second feature map; performing size alignment on the first feature map and the second feature map, and performing splicing on a channel dimension to obtain a multi-channel fusion image; and inputting the multi-channel fusion image into a first deep learning model, and outputting an accident severity prediction result. According to the method, the recognition sensitivity and the prediction recall rate of serious injury accidents are effectively improved, meanwhile, complex artificial feature engineering is avoided, and the generalization ability and the interpretability of the model are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a method and device for predicting the severity of a vehicle collision accident, equipment and a storage medium. BACKGROUND

[0002] Traffic accidents are one of the main causes of death and serious injury worldwide. Quickly and accurately predicting the severity of an accident is of great value for optimizing emergency medical response and rationally allocating rescue resources.

[0003] Currently, the mainstream technical solutions in this field mainly rely on traditional machine learning models such as logistic regression, random forest, and gradient boosting tree to analyze structured data such as vehicle speed, vehicle type, and road conditions.

[0004] However, these methods generally have low prediction recall rates for severe and fatal injuries (high AIS (Abbreviated Injury Scale) levels) when dealing with real traffic accident data. On the one hand, there is a serious class imbalance in traffic accident data. The majority of samples in the data are light injuries or no damage (AIS 0-2 levels), while the proportion of severe and fatal injury samples (AIS 5-6 levels) is very small. Traditional models optimize overall accuracy as the target, and their training process tends to ignore the minority class, resulting in insufficient recognition ability for high AIS levels. On the other hand, existing models mainly rely on numerical structured data, which further limits the model's ability to identify severe accident features.

[0005] Low prediction recall rate for severe and fatal injuries means that in actual application, there is a risk of misjudging severe accidents as ordinary accidents. This false negative will result in failure to prioritize response to the most in need of rescue, thereby affecting the efficiency of emergency rescue. Therefore, improving the prediction recall rate for high AIS levels is a technical problem that needs to be solved in this field. SUMMARY

[0006] The present application provides a method and device for predicting the severity of a vehicle collision accident, which can effectively improve the prediction recall rate for high AIS level accidents, minimize false negatives, and further improve the prediction sensitivity of severe accidents.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a method for predicting the severity of a vehicle collision accident, comprising: Collecting multi-modal data of a vehicle collision accident and preprocessing the multi-modal data; wherein the multi-modal data includes structured numerical data and unstructured text data; convert the preprocessed structured numerical data into a first feature map by a Gram angle field (GAF) method; convert the preprocessed unstructured text data into a text semantic vector by a pre-trained language model, and convert the text semantic vector into a second feature map; align the first feature map and the second feature map in size and splice them in a channel dimension to obtain a multi-channel fusion image; input the multi-channel fusion image into a first deep learning model, and output an accident severity prediction result.

[0008] In some possible implementation manners, the converting the preprocessed structured numerical data into a first feature map by a Gram angle field (GAF) method comprises: performing value range scaling on the preprocessed structured numerical data to obtain first structured numerical data, the numerical value of the first structured numerical data being located in a first interval; mapping the first structured numerical data into a first polar coordinate angle by an inverse cosine function; obtaining a first Gram angle field matrix according to the first polar coordinate angle, and taking the first Gram angle field matrix as a first feature map.

[0009] In some possible implementation manners, the converting the text semantic vector into a second feature map comprises: inputting the text semantic vector into a fully connected layer, performing linear transformation and dimension reduction on the text semantic vector to obtain a second text semantic vector; reshaping the second text semantic vector into a two-dimensional pseudo image, and taking the pseudo image as a second feature map.

[0010] In some possible implementation manners, the inputting the multi-channel fusion image into a first deep learning model and outputting an accident severity prediction result comprises: inputting the multi-channel fusion image into a first deep learning model, the first deep learning model being a convolutional neural network trained by using a focal loss function as a loss function; obtaining a probability distribution output by the convolutional neural network, the probability distribution representing the possibility that input data belongs to each injury level; taking an injury level with the highest probability value as an accident severity prediction result.

[0011] In some possible implementation manners, the converting the preprocessed unstructured text data into a text semantic vector by a pre-trained language model comprises: inputting the preprocessed unstructured text data into a pre-trained language model; extract a hidden state corresponding to a classification mark in a last transformer block in the pre-trained language model; take the hidden state as a text semantic vector representing semantics of the whole text.

[0012] In some possible implementation manners, the first deep learning model is a convolutional neural network (CNN), and the CNN includes a plurality of convolutional blocks connected in sequence, and each convolutional block includes a convolutional layer, a batch normalization layer, a self-gated activation function layer, and a pooling layer.

[0013] In some possible implementation manners, after the output of the accident severity prediction result, the method further includes: adopting a gradient-weighted class activation mapping method to generate a visualized heat map of a feature region that contributes most to the accident severity prediction result.

[0014] In a second aspect, the present application provides a device for predicting severity of a vehicle collision accident, and the device includes: an acquisition module, configured to collect multi-modal data of a vehicle collision accident, and pre-process the multi-modal data; wherein the multi-modal data includes structured numerical data and unstructured text data; a conversion module, configured to convert the pre-processed structured numerical data into a first feature map by using a Gram angular field (GAF) method, convert the pre-processed unstructured text data into a text semantic vector by using a pre-trained language model, and convert the text semantic vector into a second feature map; a fusion module, configured to perform size alignment on the first feature map and the second feature map, and perform splicing in a channel dimension to obtain a multi-channel fusion image; an output module, configured to input the multi-channel fusion image into a first deep learning model, and output an accident severity prediction result.

[0015] In a third aspect, the present application provides a computing device including a memory and a processor; wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the computing device performs the method according to any one of the first aspect.

[0016] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program, and the computer program is used to perform the method according to any one of the first aspect.

[0017] Fifthly, this application provides a computer program product comprising one or more computer instructions, wherein when the computer instructions are executed by a computer, the computer performs the method as described in any one of the first aspects.

[0018] As can be seen from the above technical solution, this application has at least the following beneficial effects: In this application, multimodal accident data containing both structured numerical data and unstructured text data is collected, and the heterogeneous data is uniformly converted and fused into an image format for processing. Specifically, the numerical data is encoded into a first feature map using Gram angle field encoding, and the text semantic vector is mapped and reshaped into a second feature map through a fully connected layer. The two are then concatenated along the channel dimension to form a multi-channel fused image, which is input into a deep learning model for end-to-end prediction. In existing technologies, traditional methods mainly rely on structured data, making it difficult to effectively utilize textual information and resulting in low recall rates for high-injury-level samples. Therefore, this application, through a multimodal image fusion method, fully leverages the complementary information of text and numerical data, effectively improving the sensitivity and predictive recall rate for identifying severe injury accidents, while avoiding complex manual feature engineering and enhancing the model's generalization ability and interpretability.

[0019] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0020] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for predicting the severity of a vehicle collision accident, as provided in an embodiment of this application; Figure 3 A general flowchart provided for embodiments of this application; Figure 4 A schematic diagram of a Gram angle field image encoding process provided in an embodiment of this application; Figure 5This is a schematic diagram of a multimodal feature fusion process provided in an embodiment of this application; Figure 6 A specific architecture diagram of a deep learning model provided in this application embodiment; Figure 7 An example diagram illustrating model interpretability provided in this application embodiment; Figure 8 A recall rate comparison curve between the present application and a comparative model at various AIS levels is provided as an embodiment of this application. Figure 9 A schematic diagram of a vehicle collision accident severity prediction device provided in an embodiment of this application; Figure 10 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0021] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0022] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0023] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first: Currently, the prediction of vehicle collision severity is mainly based on traditional machine learning methods, such as logistic regression, random forests, or gradient boosting decision trees. These methods typically rely on manually selected structured features, including numerical parameters such as vehicle speed, vehicle type, and road conditions, while neglecting the textual descriptions in accident reports that contain crucial semantic information. This single-modal approach makes it difficult for models to fully capture the complex features of accidents, particularly failing to identify key information in the text that indicates a serious accident, such as "vehicle overturned" or "person trapped."

[0024] In real-world applications, traffic accident data suffers from severe class imbalance, with minor or no injuries making up the vast majority of cases, while serious and fatal injuries are extremely rare. Traditional models, which optimize overall accuracy, systematically favor the majority class, resulting in very low recall rates for high-injury cases. This means that a large number of serious accidents requiring emergency rescue are missed, potentially leading to severe consequences in practice.

[0025] In view of this, the present application provides a method for predicting the severity of vehicle collision accidents. In this method, structured numerical data and unstructured text data are uniformly converted into image representations through multimodal data image fusion technology, and end-to-end feature learning and classification prediction are performed using deep convolutional neural networks, which effectively solves key problems such as incomplete information utilization and class imbalance in the prior art.

[0026] To make the technical solution of this application clearer and easier to understand, the application scenarios of the technical solution of this application are described below with reference to the accompanying drawings. Figure 1 As shown in the figure, this figure is a schematic diagram of an application scenario provided by an embodiment of this application.

[0027] In this application scenario, the system mainly comprises two core components: an accident data acquisition terminal 101 and an intelligent analysis server 102. The accident data acquisition terminal 101 is deployed at the scene of a traffic accident or in mobile enforcement equipment of traffic management departments. It is responsible for collecting structured numerical data, including vehicle speed and collision type, as well as unstructured textual description data from on-site investigation reports. The intelligent analysis server 102 is deployed in the cloud or at a traffic management center. It receives multimodal data uploaded by the terminal and uses the method proposed in this application to intelligently predict the severity of the accident.

[0028] The specific workflow is as follows: the accident data acquisition terminal 101 transmits the collected multimodal data to the intelligent analysis server 102; the intelligent analysis server 102 converts the received structured numerical data into a first feature map using the Gram corner field method, and converts the unstructured text data into text semantic vectors using a pre-trained language model and further converts them into a second feature map; then, the two images are aligned in size and stitched together in channel dimension to form a multi-channel fused image; finally, the fused image is input into a trained deep learning model, which outputs the accident severity prediction result and returns the analysis result to the visualization interface of the accident data acquisition terminal 101 or the traffic command center.

[0029] To this end, this application achieves a complete closed loop from accident scene data collection to cloud-based intelligent analysis through a collaborative architecture of terminals and servers. It is evident that this solution effectively utilizes the data collection capabilities of the terminal and the powerful computing resources of the server, providing reliable technical support for rapid response and rescue in traffic accidents, and possesses significant practical value and promising prospects for wider application.

[0030] To make the technical solution of this application clearer and easier to understand, the following describes a method for predicting the severity of vehicle collision accidents provided by an embodiment of this application, in conjunction with the above application scenarios. Figure 2 As shown in the figure, this is a flowchart of a method for predicting the severity of a vehicle collision accident provided in an embodiment of this application.

[0031] This method is applied to a processing device, and the method includes: S201. The processing equipment collects multimodal data of vehicle collision accidents and preprocesses the multimodal data.

[0032] Specifically, the processing equipment retrieves multimodal data of historical collision accidents from a traffic management database. Each data record contains structured numerical data and unstructured text data. The structured numerical data is an m-dimensional feature vector, including continuous variables and categorical variables encoded using one-hot encoding; m is the feature dimension of the structured numerical data, and m is a positive integer. The unstructured text data is a textual description of the accident, derived from on-site investigation reports or alarm records.

[0033] Next, the multimodal data is preprocessed, which includes two main steps: data cleaning and data normalization. In the data cleaning step, for continuous variables, the median of the feature column is used to fill missing values; for categorical variables, an unknown category is added separately for processing. The interquartile range (IQR) method is used for outlier detection and correction. The IQR method is a statistical method that uses the difference between the first and third quartiles of data to measure the degree of data variation. By calculating IQR = Q3 - Q1, where IQR is the interquartile range, Q3 is the third quartile, and Q1 is the first quartile, the range of normal values ​​is determined to be [Q1 - 1.5 × IQR, Q3 + 1.5 × IQR]. Values ​​outside this range are considered outliers and replaced with corresponding boundary values, effectively eliminating the interference of extreme values ​​on model training.

[0034] In the data normalization step, all continuous variables are subjected to max-min normalization, which linearly transforms them to the [0,1] interval to eliminate the influence of dimensions. The normalization formula is:

[0035] in, Let j be the original feature value of the j-th feature. The first structured numerical data for the j-th feature. Let be the minimum value of the j-th feature. It represents the maximum value of the j-th feature.

[0036] For unstructured text data, preprocessing includes two steps: text cleaning and word segmentation and embedding. First, regular expressions are used to remove special characters, punctuation marks, and numbers from the text, convert them to lowercase, and remove stop words such as "the", "is", and "at". Second, the WordPiece word segmenter is used to segment the processed text and generate word sequences.

[0037] After the above preprocessing, we obtain preprocessed structured numerical data and preprocessed unstructured text data. The preprocessed structured numerical data refers to numerical feature vectors that have undergone data cleaning and normalization, and can be directly used for subsequent image encoding; the preprocessed unstructured text data refers to word sequences that have undergone text cleaning and word segmentation, providing standardized text input for subsequent semantic vector conversion.

[0038] like Figure 3 The overall flowchart of this application, as shown, indicates that preprocessing is the initial step in the entire process, laying the data foundation for subsequent image conversion. The beneficial effect of this preprocessing step is that, through systematic data cleaning and standardization, it effectively improves data quality, eliminates the negative impact of missing values, outliers, and dimensional differences on model performance, and provides a high-quality, standardized data foundation for subsequent multimodal feature fusion and deep learning modeling, thereby improving the accuracy and robustness of accident severity prediction.

[0039] S202, The processing device converts the preprocessed structured numerical data into a first feature map using the Gram angle field (GAF) method.

[0040] Gramian Angular Field (GAF) is an encoding technique that transforms one-dimensional time series or feature vectors into two-dimensional image matrices. Its core idea is to map normalized numerical data onto a polar coordinate system and, by calculating the trigonometric function values ​​of the sum (or difference) of every two points, generate a Gram matrix that retains the complete information of the original data and explicitly represents the relationships between data points; this matrix serves as the feature image. Figure 4 The process of encoding Gram angle field images is demonstrated, including three key sub-steps: normalization, polar coordinate mapping, and GAF ​​matrix calculation, which ultimately generates a numerical feature image.

[0041] The processing device utilizes a convolutional neural network for image feature extraction and preserves the inherent correlations between structured features. This application employs Gram corner field technology to convert the normalized feature vectors into an image representation. The specific process is as follows: The processing device performs value range scaling on the preprocessed structured numerical data to obtain first structured numerical data, wherein the values ​​of the first structured numerical data lie within a first interval; specifically, the first interval is the interval [-1, 1], and the calculation method of the first structured numerical data is as follows:

[0042] Next, the first structured numerical data is mapped to polar coordinate angle space using the inverse cosine function, converting the numerical value into angle information to obtain the first polar coordinate angle; the formula for calculating this first polar coordinate angle is: ,

[0043] in, Let the first polar coordinate angle of the j-th feature be . The first structured numerical data for the j-th feature. Let be the original feature value of the j-th feature, and arccos is the inverse cosine function.

[0044] The first Gram angle field matrix G is obtained based on the first polar coordinate angle. Each element in the matrix is ​​defined as the cosine value of the sum of different characteristic angles, and the calculation formula is as follows:

[0045] Where G is the first Gram angular field matrix, Let the first polar coordinate angle of the i-th feature be . Let be the first polar coordinate angle of the j-th feature, m be the feature dimension of the structured numerical data, and m be a positive integer.

[0046] The first Gram angular field matrix G is used as the first feature map I. num (exist Figure 3 In this first feature map (represented by I_num), the main diagonal elements (i=j) encode the information of the original feature value itself, while the off-diagonal elements encode the interaction relationship between different features i and j, thus presenting the correlation between structured features in the form of image pixels.

[0047] S203 The processing device converts the pre-processed unstructured text data into text semantic vectors through a pre-trained language model, and then converts the text semantic vectors into a second feature map.

[0048] Pre-trained language models are deep learning models that are pre-trained on large-scale unlabeled text data through unsupervised or self-supervised learning methods. Their core is to model the semantics, syntax, contextual relationships, and other information of the text, thereby possessing powerful natural language understanding and generation capabilities.

[0049] It learns the general representation ability of text by completing pre-training tasks on large-scale corpora, and can then be fine-tuned to adapt to specific scenarios in specific downstream tasks (such as text classification, semantic understanding, machine translation, etc.).

[0050] In this application, the pre-trained language model uses the BERT-Base model to convert unstructured accident scene text descriptions (such as investigation reports) into text semantic vectors, thereby realizing the structured encoding of text semantics and providing a semantic basis for subsequent conversion into a second feature map, fusion with the first feature map, and accident severity prediction.

[0051] BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) adopts a bidirectional transformer architecture, which can utilize both left and right context information simultaneously. It is a milestone model in pre-trained language models and has spawned versions such as BERT-base and BERT-large.

[0052] This step aims to convert unstructured text data, such as accident scene investigation reports, into an image format that can be fused with structured data, namely, a second feature map. Specifically, it includes two sub-processes: text semantic vector extraction and semantic vector dimensionality reduction and reshaping. The specific steps are as follows: The text semantic vector extraction process involves the processing device inputting pre-processed unstructured text data into a pre-trained language model. The pre-processed unstructured text data, such as "the accident scene was on a main urban road, the vehicle involved was a small sedan, the collision was a frontal rear-end collision, and the driver was not injured," is processed through redundancy removal, word segmentation, padding, or truncation to a fixed length.

[0053] Next, following the input format of the pre-trained language model, such as adding the [CLS] prefix and [SEP] separator, the data is input into the pre-trained language model. The hidden states corresponding to the classification labels in the last layer of the transformer block are extracted. These hidden states are then used as the text semantic vector representing the semantics of the entire text. The pre-trained language model consists of multiple stacked Transformer modules. After the model's forward computation is complete, the hidden states corresponding to the [CLS] classification labels in the last layer of the Transformer module are extracted. These hidden states represent the model's global encoding of the entire text's semantics; therefore, they are used as the text semantic vector representing the semantics of the entire text. The calculation formula is as follows:

[0054] in, Here, is the text semantic vector, BERT is the pre-trained language model, CLS is the classification label in the input format of the pre-trained language model (such as BERT), which is added at the beginning of the text to allow the model to learn the global semantics of the entire text, R is the real number space, and T is the unstructured text data.

[0055] Since the text semantic vector is a high-dimensional one-dimensional vector, it cannot be directly fused with the first feature map (two-dimensional matrix) obtained by GAF encoding. Therefore, it needs to be converted into a two-dimensional pseudo-image, which is then used as the second feature map. Specifically, the text semantic vector is input into a fully connected layer W, where W∈R. d×768(Where d = H′ × W′, d is the dimension, H′ is the preset height of the second feature map, and W′ is the preset width of the second feature map.) A linear transformation and dimensionality reduction are performed on the text semantic vector to obtain the second text semantic vector; the second text semantic vector is then reshaped into a two-dimensional pseudo-image I. text (exist Figure 3 (represented by I_text in the text), the pseudo image I text As a second feature map, it transforms the semantic information of the text into a spatial feature map form.

[0056] S204. The processing device aligns the size of the first feature map and the second feature map, and stitches them together in the channel dimension to obtain a multi-channel fused image.

[0057] Since the initial sizes of the first feature map (numerical feature image) and the second feature map (text pseudo-image) may be inconsistent, they need to be adjusted to the same size through an upsampling operation. The first feature map (numerical feature image) is denoted as I. num Its initial size is an m×m two-dimensional matrix (m is the feature dimension of the structured numerical data), and its corresponding mathematical representation is I. num ∈R m×m (R represents the real number space, and m×m represents the number of rows × the number of columns of the matrix); the second feature map (text pseudo-image) is denoted as I. text Its size is H×W (which is the size of the two-dimensional matrix after semantic vector reshaping in step S203, for example, 28×28, where H is the height of the second feature map and W is the width of the second feature map).

[0058] Bilinear interpolation algorithm is used for I num Upsampling is performed by calculating the weighted average of the four original pixels surrounding the target pixel, thus enlarging the image size while preserving feature smoothness. I num Upsampling to I text For the same size H×W, we obtain a size-aligned numerical feature image, denoted as I. num up (exist Figure 5 In Chinese, I_num_up is used, and its mathematical representation is I. num up ∈R H×W .

[0059] Two size-aligned images are stitched together along the channel dimension (i.e., the third dimension of the image, such as the red, green, and blue channels of an RGB image) to obtain a multi-channel fused image. Images are typically stored in computers as a three-dimensional tensor of height × width × number of channels. num up with I textCurrently, they are all single-channel H×W two-dimensional matrices (which can be regarded as a three-dimensional tensor with 1 channel, i.e., H×W×1).

[0060] Numerical image I after size alignment num up With text pseudo-image I text Concatenation (Concat) is performed in the channel dimension (the third dimension), which involves superimposing the channel numbers of two single-channel images to generate a multi-channel fused image. Its mathematical representation is as follows:

[0061] Among them, I fused For multi-channel fused images, I fused ∈R H×W×2 (exist Figure 3 (referred to as I_fused in the text) This fused image simultaneously contains the structured information of numerical features and the semantic information of text features, providing input for subsequent deep learning models.

[0062] Figure 5 This demonstrates the entire process of multimodal feature fusion, including text semantic vector extraction, text feature visualization, and size alignment and channel concatenation of numerical and text images, ultimately generating a multi-channel fused image. This combination provides the input carrier for subsequent deep learning models, ensuring that the model can simultaneously utilize information from both types of data to predict accident severity.

[0063] S205 The processing device inputs the multi-channel fused image into the first deep learning model and outputs the accident severity prediction result.

[0064] The first deep learning model is a Convolutional Neural Network (CNN), which includes multiple sequentially connected convolutional blocks. Each convolutional block contains a convolutional layer, a batch normalization layer, a self-gated activation function layer, and a pooling layer. Figure 6 The specific architecture of the deep learning model used in this application is shown, which includes a feature extraction backbone network with four convolutional blocks and a classifier, used to learn features from fused images and output prediction results.

[0065] The processing device constructs an end-to-end deep convolutional neural network to obtain the first deep learning model, whose input is the fused image I. fused The specific architecture is as follows: The feature extraction backbone network consists of four consecutive convolutional blocks. Each convolutional block contains: a two-dimensional convolutional layer (Conv2D) using 3×3 convolutional kernels, with the number of filters increasing progressively with each block (e.g., 32, 64, 128, 256); a batch normalization layer to accelerate training and improve stability; and a self-gated activation function, Swish. A 2×2 max pooling layer (MaxPooling2D) is used to provide smooth and non-monotonic activation characteristics; a 2×2 max pooling layer (MaxPooling2D) is used to reduce dimensionality and expand the receptive field.

[0066] After flattening the feature map output from the last convolutional block into a one-dimensional vector, it is fed into two fully connected layers (with 512 and 128 neurons, respectively). Finally, it is passed through an output layer (7 neurons, corresponding to the 7 levels of AIS0-6) and the Softmax activation function is used to output the probability distribution of each damage level.

[0067] in, This represents the predicted probability that a sample belongs to AIS level 0. This indicates the predicted probability that the sample belongs to AIS level 1. This indicates the predicted probability that the sample belongs to AIS level 2. This indicates the predicted probability that the sample belongs to AIS level 3. This indicates the predicted probability that the sample belongs to AIS level 4. This indicates the predicted probability that the sample belongs to AIS level 5. This indicates the predicted probability that the sample belongs to AIS level 6.

[0068] To effectively address the extreme imbalanced distribution of classes (AIS levels) in the dataset and avoid model bias towards the majority class, this application employs Focal Loss as the loss function. Focal Loss is a loss function specifically designed to solve the class imbalance problem. It dynamically reduces the contribution of easily classified samples (usually the majority class with abundant data) to the total loss during training by modifying the standard cross-entropy loss function, thereby forcing the model to focus more on learning those difficult-to-classify samples (usually the minority class with less data). Its expression is:

[0069] in, Here, N is the focus loss function, and N is the number of samples in the batch. The true class label for sample n; The model predicts that sample n belongs to its true class. The probability; γ (focusing parameter, γ≥0) is used to adjust the weight of difficult samples, which are usually minority class samples that are misclassified. The larger γ is, the more the model focuses on samples that are difficult to classify. In this embodiment, γ=2.0 is set; These are the class weight coefficients, used to balance the importance of different classes. They are set to be proportional to the class weights in the training set. The frequency is inversely proportional to the frequency, that is: ∝1 / frequency( This assigns higher loss weights to high AIS levels (such as fatal wounds) where samples are scarce, forcing the model to optimize its classification performance for these critical minority classes.

[0070] The AdamW optimizer is used, which introduces weight decay regularization to help prevent overfitting. The initial learning rate is set to 1×10⁻⁶. −4 A cosine annealing decay strategy is employed to dynamically adjust the learning rate, helping the model escape local optima and converge to a better solution. Performance is monitored on the validation set (e.g., macro-average F1 score). If performance no longer improves within a certain number of consecutive training epochs, training is terminated early to prevent overfitting.

[0071] The multi-channel fused image is input into a first deep learning model, which is a convolutional neural network trained with a focus loss function as the loss function; the probability distribution of the output of the convolutional neural network is obtained, which represents the probability that the input data belongs to each injury level; the injury level with the highest probability value is taken as the accident severity prediction result.

[0072] Specifically, for new and unknown accident data, the same preprocessing, image encoding, and feature fusion are first performed strictly according to the above process to obtain the corresponding new multi-channel fused image I. fused new The image is then fed into a pre-trained deep learning model, which outputs a 7-dimensional probability vector P. new The category with the highest probability is selected as the final predicted AIS level, i.e., the predicted AIS for accident severity.

[0073] Where i1 is the category index.

[0074] In some embodiments, after outputting the accident severity prediction results, a gradient-weighted class activation mapping method is also used to generate a visual heatmap of the feature region that contributes the most to the accident severity prediction results.

[0075] To enhance the credibility and acceptability of the model's prediction results, this application utilizes gradient-weighted class activation mapping (Grad-CAM) technology to visualize and explain the model's decision-making process.

[0076] Specifically, for the input image I fused new Along with its predicted category, Grad-CAM generates a heatmap by calculating the gradient of the target category score relative to the feature map of the last convolutional layer.

[0077] This heatmap can be overlaid on the original input image, visually showing which regions in the fused image the model's decisions primarily rely on. For example, the heatmap clearly shows that the model makes high AIS-level predictions based on feature regions in the numerical image representing "high speed" and "no airbags," and feature regions in the text image representing the semantics of "rolling" and "trapped."

[0078] like Figure 7 As shown, Figure 7 This application provides an example diagram of model interpretability, showcasing a visual heatmap generated using Gradient Weighted Class Activation Mapping (Grad-CAM) technology. This heatmap, by tracing back the model's decision-making process, clearly identifies the feature regions that contribute most to the prediction results within the multi-channel fused image.

[0079] The arrows in the image clearly indicate the two key areas of focus in the heatmap: the left arrow points to the numerical feature image area, which corresponds to key structured features such as "the change in collision speed is higher than the threshold" and "the airbags did not deploy"; the right arrow points to the text feature image area, which corresponds to high-risk semantic features such as "vehicle rollover" and "people trapped".

[0080] This diagram visually demonstrates, through arrow directions and area markings, how the model comprehensively assesses key evidence from both numerical and textual modalities when predicting high-AIS-level incidents. This visualization method effectively enhances the interpretability of the model's decision-making process, providing users with clear and reliable predictive data.

[0081] This provides traffic management personnel and emergency rescue personnel with a window to understand model-based decision-making, making it no longer a black box, thus greatly enhancing the application value of this method in the real world.

[0082] Figure 8The comparison curves clearly demonstrate that this application maintains a high recall rate across all AIS levels, outperforming traditional machine learning models and single-modal deep learning methods. Curve 1 (traditional machine learning model): using traditional methods such as XGBoost, heavily reliant on structured data, resulting in a decrease in recall rate; Curve 2 (single-modal deep learning model): using only numerical or text-based single-modal deep learning methods, performance is improved but still limited; Curve 3 (this application): an innovative method based on multimodal data visualization and deep learning, maintaining a high recall rate across all AIS levels.

[0083] To make the technical solution of this application clearer and easier to understand, the technical solution of this application will be described in detail below with reference to specific embodiments.

[0084] Specifically, the CIDAS (China In-Depth Accident Study) traffic accident dataset was used. To ensure data quality and model relevance, the following screening was performed: only collision accident records involving two motor vehicles were retained; samples with missing key features (such as vehicle speed and severity of injury) or obvious distortion were removed; the final result was a dataset containing 35,000 valid accident records.

[0085] Each record contains 28 structured features, including continuous variables (such as delta_v (collision speed change), vehicle_curb_weight (vehicle curb weight), and person_age (person's age)) and categorical variables (such as collision_type (collision type), airbag_deployed (airbag deployment status), and weather_condition (weather conditions)). All categorical variables are one-hot encoded.

[0086] Text description: Derived from the narrative field of the police report, it describes the course of the accident, the scene environment, and the status of vehicles and personnel.

[0087] Injury Labels: The severity of injury to each occupant is labeled by professional coders according to the AIS2015 standard as a target for model prediction (7 levels in total: AIS0 to 6).

[0088] Next, we will set up the experiment.

[0089] 1) Preprocessing: Numerical feature selection and normalization. The text is extracted into a 768-dimensional vector using the BERT-base-uncased model (a case-insensitive BERT base model).

[0090] The 28 continuous features were cleaned (missing values ​​were filled with median and outliers were handled by IQR method) and then normalized to the [0,1] interval by maximum-min.

[0091] The BERT model is pre-trained using the bert-base-uncased model from the transformers library. After text cleaning and word segmentation, it is input into the BERT model to extract the final hidden state of the [CLS] tag as a 768-dimensional semantic vector representation.

[0092] 2) Visualization: The 28-dimensional numerical vector is converted into a 28×28 GAF image. The text vector is mapped and reshaped into a 16×16 image through a fully connected layer.

[0093] The 28-dimensional normalized feature vector is converted into a 28×28 pixel grayscale image using the GAF method.

[0094] The 768-dimensional text vector is mapped to 256 dimensions through a fully connected layer (without activation function), and then reshaped into a 16×16 pseudo-image.

[0095] 3) Fusion: Upsample the numerical image to 16×16 and stitch it with the text image to form a fused image of 16×16×2.

[0096] The 28×28 numerical image was upsampled to 16×16 size using bilinear interpolation.

[0097] This image is then stitched together with a 16×16 text image in the channel dimension to generate the final fused image (size: 16×16×2).

[0098] 4) Model and Training: Using Figure 6 The CNN architecture shown contains four convolutional blocks: convolutional block 1, convolutional block 2, convolutional block 3, and convolutional block 4, with filter numbers of 32, 64, 128, and 256, respectively. The Focal Loss parameters are set to γ=2.0 and α=[0.1,0.1, 0.1, 0.2, 0.8, 1.5, 3.0], where α is the class weight coefficient. The AdamW optimizer (learning rate 1×10⁻⁶) is used. -4 The dataset was trained for 100 epochs. The dataset was divided into training, validation, and test sets in a 6:2:2 ratio.

[0099] Adopting such Figure 6The diagram shows a custom convolutional neural network (CNN). This network contains four convolutional blocks, each consisting of a Conv2D layer (3×3 kernels, with 32, 64, 128, and 256 filters respectively), a BatchNorm layer, a Swish activation layer, and a MaxPooling2D layer (2×2). Following the convolutional layers are Flatten layers and two fully connected layers (512 and 128 neurons respectively), culminating in a 7-neuron output layer using the Softmax activation function.

[0100] Focal Loss was employed. The focusing parameter was set to γ=2.0 to increase attention to hard-to-classify samples; the class weight coefficients were set to α=[0.1, 0.1, 0.1, 0.2, 0.8, 1.5, 3.0], which are inversely proportional to the frequency of each AIS level in the training set, thereby significantly improving the model's sensitivity to high AIS levels (4-6).

[0101] Using the AdamW optimizer (initial learning rate = 1 × 10⁻⁶) -4 Weight decay = 1 × 10 -4 The batch size was set to 64. The dataset was randomly divided into a training set (21,000 records), a validation set (7,000 records), and a test set (7,000 records) in a 6:2:2 ratio. The model was trained for 100 epochs, and an early stopping strategy (Patience=10) was used on the validation set to prevent overfitting.

[0102] To evaluate the performance of this application, it was compared with two baseline models on an independent test set. All experiments were repeated five times, and the results are expressed as mean ± standard deviation, as shown in Table 1.

[0103] Table 1: Performance Comparison Table (%, Mean ± Standard Deviation)

[0104] The multimodal image fusion framework proposed in this application possesses high versatility. Its innovation lies in transforming heterogeneous data into a unified image representation and performing deep fusion. It is not only applicable to vehicle collision accident severity prediction, but with appropriate adjustments, it can also be widely applied to other fields requiring severity grading or risk prediction of multi-source data. For example: Industrial safety: It can integrate equipment sensor data (numerical values) with maintenance log text to predict the risk level of equipment failure.

[0105] Emergency medicine: Patient vital signs data (numerical values) can be integrated with the chief complaint text description to predict the severity of emergency triage.

[0106] Financial risk control: It can integrate transaction data (numerical values) with customer behavior description text to assess the risk level of fraudulent transactions.

[0107] In such applications, it is only necessary to replace the feature engineering in S1 according to the data structure of the specific domain and adjust the category output by the first deep learning model. The main framework of this application has great reusability.

[0108] Based on the above, this application achieves accurate prediction of the severity of vehicle collision accidents through an innovative method of multimodal data image fusion and deep learning. Specifically, this method converts structured numerical data into image representations using Gram corner field technology, converts unstructured text data into semantic vectors using a pre-trained language model, and further reconstructs them into pseudo-images. Deep fusion of multimodal features is achieved through channel-dimensional concatenation. End-to-end training is performed using a convolutional neural network optimized with a focus loss function, effectively solving the class imbalance problem and improving the recall rate for predicting severe injury accidents. Simultaneously, gradient-weighted class activation mapping technology is combined to provide a visual explanation of the decision, enhancing the model's interpretability and practical value. Experimental results show that this method maintains a high recall rate across all AIS levels, significantly outperforming traditional machine learning methods and single-modal deep learning methods, providing reliable technical support for intelligent traffic management and emergency rescue.

[0109] The above text combined Figures 1 to 8 The method for predicting the severity of vehicle collision accidents provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0110] This application also provides a device for predicting the severity of vehicle collision accidents, such as... Figure 9 As shown in the figure, this is a schematic diagram of a vehicle collision accident severity prediction device provided in an embodiment of this application. The device includes: The acquisition module 901 is used to collect multimodal data of vehicle collision accidents and preprocess the multimodal data; wherein, the multimodal data includes structured numerical data and unstructured text data; The conversion module 902 is used to convert the preprocessed structured numerical data into a first feature map using the Gram angle field (GAF) method; convert the preprocessed unstructured text data into a text semantic vector using a pre-trained language model; and convert the text semantic vector into a second feature map. The fusion module 903 is used to align the first feature map and the second feature map in size and stitch them together in the channel dimension to obtain a multi-channel fused image. The output module 904 is used to input the multi-channel fused image into the first deep learning model and output the accident severity prediction result.

[0111] In some possible implementations, the conversion module 902 is specifically used to scale the value range of the preprocessed structured numerical data to obtain first structured numerical data, wherein the value of the first structured numerical data is located in a first interval; to map the first structured numerical data to a first polar coordinate angle using an inverse cosine function; to obtain a first Gram angle field matrix based on the first polar coordinate angle, and to use the first Gram angle field matrix as a first feature map.

[0112] In some possible implementations, the transformation module 902 is specifically used to input the text semantic vector into a fully connected layer, perform linear transformation and dimensionality reduction on the text semantic vector to obtain a second text semantic vector; reshape the second text semantic vector into a two-dimensional pseudo-image, and use the pseudo-image as a second feature map.

[0113] In some possible implementations, the output module 904 is specifically used to input the multi-channel fused image into a first deep learning model, which is a convolutional neural network trained with a focus loss function as the loss function; obtain the probability distribution of the output of the convolutional neural network, which represents the probability that the input data belongs to each injury level; and take the injury level with the highest probability value as the accident severity prediction result.

[0114] In some possible implementations, the conversion module 902 is specifically used to input the preprocessed unstructured text data into a pre-trained language model; extract the hidden states corresponding to the classification labels in the last layer transformer block of the pre-trained language model; and use the hidden states as text semantic vectors representing the semantics of the entire text.

[0115] In some possible implementations, the first deep learning model is a convolutional neural network (CNN), which includes multiple sequentially connected convolutional blocks, each containing a convolutional layer, a batch normalization layer, a self-gated activation function layer, and a pooling layer.

[0116] In some possible implementations, the device further includes: The generation module is used to generate a visual heatmap of the feature region that contributes the most to the prediction result of the accident severity using a gradient-weighted class activation mapping method.

[0117] The vehicle collision severity prediction device according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the vehicle collision severity prediction device are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0118] This application also provides a computing device. For example... Figure 10 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 400 includes a bus 401, a processor 402, a communication interface 403, and a memory 404. The processor 402, the memory 404, and the communication interface 403 communicate with each other via the bus 401.

[0119] Bus 401 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0120] Processor 402 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0121] The communication interface 403 is used for communication with external devices. For example, if the computing device is a first switch, the communication interface 403 can be used for communication between the first switch and a first user terminal, or for communication between the first switch and a second switch.

[0122] Memory 404 may include volatile memory, such as random access memory (RAM). Memory 404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0123] The memory 404 stores executable code, which the processor 402 executes to perform the aforementioned method for predicting the severity of a vehicle collision accident.

[0124] Specifically, in achievingFigure 9 In the case of the illustrated embodiment, and Figure 9 When the modules or units of the vehicle collision severity prediction device described in the embodiment are implemented by software, the following steps are performed: Figure 9 The software or program code required for the functions of each module / unit can be partially or wholly stored in memory 404. Processor 402 executes the program code corresponding to each unit stored in memory 404 to execute the aforementioned method for predicting the severity of vehicle collision accidents.

[0125] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute the aforementioned method for predicting the severity of a vehicle collision accident.

[0126] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0127] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0128] When the computer program product is executed by a computer, the computer performs any of the aforementioned methods for predicting the severity of vehicle collision accidents. The computer program product can be a software installation package; when any of the aforementioned methods for predicting the severity of vehicle collision accidents needs to be used, the computer program product can be downloaded and executed on the computer.

[0129] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A method for predicting the severity of a vehicle collision accident, characterized in that, The method includes: Multimodal data of vehicle collision accidents are collected, and the multimodal data is preprocessed; wherein, the multimodal data includes structured numerical data and unstructured text data; The preprocessed structured numerical data is converted into a first feature map using the Gram angle field (GAF) method. The preprocessed unstructured text data is converted into text semantic vectors through a pre-trained language model, and the text semantic vectors are then converted into a second feature map. The first feature map and the second feature map are aligned in size and then stitched together along the channel dimension to obtain a multi-channel fused image; The multi-channel fused image is input into the first deep learning model, which outputs the accident severity prediction result.

2. The method according to claim 1, characterized in that, The step of converting the preprocessed structured numerical data into a first feature map using the Gram angle field (GAF) method includes: The preprocessed structured numerical data is scaled in range to obtain the first structured numerical data, and the value of the first structured numerical data is located in the first interval. The first structured numerical data is mapped to a first polar coordinate angle using an inverse cosine function; The first Gram angle field matrix is ​​obtained based on the first polar coordinate angle, and the first Gram angle field matrix is ​​used as the first feature map.

3. The method according to claim 1, characterized in that, The step of converting the text semantic vector into a second feature map includes: The text semantic vector is input into a fully connected layer, and a linear transformation and dimensionality reduction are performed on the text semantic vector to obtain a second text semantic vector. The second text semantic vector is reshaped into a two-dimensional pseudo-image, and the pseudo-image is used as the second feature map.

4. The method according to claim 1, characterized in that, The step of inputting the multi-channel fused image into the first deep learning model and outputting an accident severity prediction result includes: The multi-channel fused image is input into a first deep learning model, which is a convolutional neural network trained with a focus loss function as the loss function. Obtain the probability distribution of the output of the convolutional neural network, whereby the probability distribution represents the likelihood that the input data belongs to each damage level; The injury level with the highest probability value is used as the prediction result for the severity of the accident.

5. The method according to claim 1, characterized in that, The process of converting preprocessed unstructured text data into text semantic vectors using a pre-trained language model includes: The preprocessed unstructured text data is input into the pre-trained language model; Extract the hidden states corresponding to the classification labels in the last transformer block of the pre-trained language model; The hidden state is used as a text semantic vector representing the semantics of the entire text.

6. The method according to claim 1, characterized in that, The first deep learning model is a convolutional neural network (CNN), which includes multiple sequentially connected convolutional blocks. Each convolutional block contains a convolutional layer, a batch normalization layer, a self-gated activation function layer, and a pooling layer.

7. The method according to claim 1, characterized in that, After outputting the accident severity prediction result, the method further includes: A gradient-weighted class activation mapping method is used to generate a visual heatmap of the feature region that contributes the most to the prediction result of the accident severity.

8. A device for predicting the severity of a vehicle collision accident, characterized in that, The device includes: The acquisition module is used to collect multimodal data of vehicle collision accidents and preprocess the multimodal data; wherein, the multimodal data includes structured numerical data and unstructured text data; The conversion module is used to convert preprocessed structured numerical data into a first feature map using the Gram angle field (GAF) method; to convert preprocessed unstructured text data into text semantic vectors using a pre-trained language model; and to convert the text semantic vectors into a second feature map. The fusion module is used to align the first feature map and the second feature map in size and stitch them together in the channel dimension to obtain a multi-channel fused image; The output module is used to input the multi-channel fused image into the first deep learning model and output the accident severity prediction result.

9. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 7.

Images