Automatic driving dangerous scene detection method based on large language model

Abstract

The method aims at solving the problems of synchronization and algorithm complexity in automatic driving dangerous scene detection multi-sensor data fusion at present, difficulty in scene understanding and traffic participant intention prediction in a complex urban environment, insufficient training data diversity and high labeling cost. According to the automatic driving dangerous scene detection method based on the large language model, the powerful language understanding ability, the high-quality generation ability and the powerful generalization ability of the large language model are applied to the field of automatic driving, and the scene understanding and scene reasoning ability of the model is enhanced by analyzing natural language description and instructions; in combination with sensor data and semantic information, abnormal detection and real-time feedback are improved; dynamically adjusting a decision strategy, and selecting a safest and most efficient driving path; diversified driving scenes are generated for training and testing, and an optimization algorithm is continuously learned. The method is combined with a large language model to improve the efficiency and safety level of automatic driving dangerous scene detection, so that an automatic driving automobile can cope with a complex traffic environment more safely and reliably, and a solid foundation is laid for realizing real unmanned driving.

Description

Technical Field

[0001] The present invention relates to a dangerous scene detection method, and in particular to an automatic driving dangerous scene detection method based on a large language model. Background Art

[0002] The development of autonomous driving technology depends on the effective detection of dangerous scenes, which is the key to ensuring driving safety. Dangerous scene detection technology is mainly based on multiple sensors and advanced data processing algorithms to achieve real-time identification and prediction of potential dangers in complex traffic environments.

[0003] First, self-driving cars are equipped with a variety of sensors, such as lidar, cameras, millimeter-wave radar, and ultrasonic sensors, which can provide high-resolution environmental perception data. Secondly, data processing and fusion technology plays a vital role in dangerous scene detection. Multi-sensor fusion technology can integrate data from different sensors to improve the accuracy and reliability of detection. Computer vision and deep learning algorithms, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), are widely used in the analysis of image and video data to identify pedestrians, vehicles, road signs, and other obstacles. These algorithms can not only identify static objects, but also predict the movement trajectory of dynamic objects, thereby predicting possible dangerous situations in advance. Finally, real-time performance is another key factor in dangerous scene detection. The autonomous driving system needs to process and respond to sensor data within milliseconds to achieve rapid danger identification and response. This requires the system to have strong computing power and optimized algorithm design.

[0004] The current synchronization and algorithm complexity problems faced by multi-sensor data fusion in the detection of dangerous scenes in autonomous driving, the difficulties in scene understanding and traffic participant intention prediction in complex urban environments, the insufficient diversity of training data and the high cost of annotation. The present invention utilizes the powerful language understanding ability, high-quality generation ability and strong generalization ability of the large language model and applies it to the field of autonomous driving. By parsing natural language descriptions and instructions, the model's situational understanding and situational reasoning capabilities are enhanced; by combining sensor data and semantic information, anomaly detection and real-time feedback are improved; decision-making strategies are dynamically adjusted to select the safest and most efficient driving path; diversified driving scenarios are generated for training and testing, and algorithms are continuously learned and optimized. The combination with the large language model improves the efficiency and safety level of autonomous driving dangerous scene detection, enabling autonomous driving vehicles to cope with complex traffic environments more safely and reliably, laying a solid foundation for the realization of true unmanned driving. Summary of the invention

[0005] The present invention proposes a method for detecting dangerous scenarios in autonomous driving based on a large language model, which extracts useful features from data from different sensors. The large language model can combine data from multiple sensors to achieve more comprehensive environmental perception through semantic understanding, scenario analysis, and generation of various dangerous scenarios.

[0006] The method for detecting dangerous scenes in autonomous driving based on a large language model described in the present invention first uses multiple sensors to collect data and fuse multi-source data; uses a pre-trained large language model (GPT-4) and optimizes it through transfer learning so that it can understand and process specific tasks related to autonomous driving, and sends the data processed by the large language model to the YOLOv5 model for detection. At the same time, the dangerous scenes generated by the large language model are used as training input to optimize the decision-making strategy of the autonomous driving system; uses the large language model to analyze the weaknesses and deficiencies of the system in the simulation, generates targeted improvement scenarios, and performs iterative training and optimization; generates various dangerous scenes in a virtual environment, and uses the large language model to perform detection and response tests to ensure the performance of the model in various complex situations.

[0007] The present invention has the following advantages: multi-source sensor data is preprocessed using a large language model, and after pre-training is completed, fine-tuning is performed on dangerous scene tasks. The semantic understanding and generation capabilities of the large language model are used to convert dangerous scenes described in natural language into specific simulation scenes and send them to the YOLOv5 model for dangerous scene and abnormal behavior detection; various dangerous scenes are generated in a virtual environment, and the large language model is used for detection and response testing to ensure the performance of the model in various complex situations. By analyzing the dangerous scenes encountered by the automatic driving system during actual driving, the model can continuously learn and improve its detection algorithm to achieve system feedback; the present invention combines the large language model with the YOLOv5 target detection model to achieve the detection of dangerous scenes and abnormal behaviors in automatic driving, thereby making intelligent and safe decisions, and realizing the detection of dangerous road conditions and the avoidance of pedestrians. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is the overall network model structure diagram of the present invention; Figure 2 It is a transformer structure diagram of the present invention; Figure 3 This is a structural diagram of the self-attention mechanism of the present invention; Figure 4 It is the YOLOv5s model structure diagram of the present invention; Figure 5 It is the structure diagram of FPN+PANet of the present invention; DETAILED DESCRIPTION

[0009] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0010] The overall network structure model of the present invention refers to Figure 1 The data acquisition module collects multi-source data from cameras and lidars to ensure that the data covers various driving environments and potential dangerous scenes, and extracts useful features from the data of different sensors, such as pedestrians in the image, vehicles detected by radar, and obstacles measured by lidar. The large language model is used to annotate and enhance the data. At the same time, the large language model is used to convert the dangerous scenes described in natural language into specific simulation scenes, and various dangerous scenes are generated for the training and testing of the autonomous driving system to improve the system's ability to respond to real dangerous situations; the processed data is sent to the deployed trained YOLOv5 model for real-time target detection; according to the current environment and traffic conditions, the model can perform real-time risk assessment, identify potential dangerous scenes and issue warnings. When a dangerous scene is detected, the large language model can assist the autonomous driving system in making emergency braking or avoidance decisions quickly to ensure the safety of the vehicle and passengers. By analyzing the dangerous scenes encountered by the autonomous driving system in actual driving, the model can continuously learn and improve its detection algorithm to achieve system feedback.

[0011] The large language model processing module of the present invention adopts the GPT-4.0 large language model based on the Transformer architecture, which is implemented through unsupervised pre-training and fine-tuning methods. The model has excellent semantic understanding, scene generation and multimodal processing capabilities. It uses a large language model to annotate and enhance data, reducing the number of parameters in the target detection process, thereby improving the real-time performance of detection; the large language model generates rare dangerous scenes for autonomous driving training and testing, improving the accuracy and safety of dangerous scene detection during target detection; when a dangerous scene is detected, the large language model can assist the autonomous driving system to quickly make emergency braking or avoidance decisions to ensure the safety of the vehicle and passengers; at the same time, the model can continuously learn and improve its detection algorithm to achieve system feedback. The combination of the large language model and the dangerous scene detection technology of autonomous driving significantly improves the performance and reliability of detection.

[0012] Transformer structure diagram reference Figure 2. It is a model based on the Encoder-Decoder framework, mainly composed of self-attention mechanism, multi-head attention mechanism, position encoding, residual connection and layer normalization. Among them, the self-attention mechanism, which is one of the core concepts of Transformer, enables the model to consider all positions in the input sequence at the same time, rather than processing them step by step like recurrent neural networks (RNN) or convolutional neural networks (CNN). The self-attention mechanism allows the model to assign different attention weights according to different parts of the input sequence, so as to better capture the semantic relationship.

[0013] Reference to the structure diagram of the self-attention mechanism Figure 3 . Its calculation steps include: S1, input representation: given an input sequence X, which contains n words, each word is represented as a d-dimensional vector. The shape of the input matrix is ​​(n, d). S2, calculation of Query, Key and Value: through linear transformation, the input X is mapped to Query (Q), Key (K) and Value (V) space respectively. These transformations can be expressed as: in , and is the weight matrix to be learned, and its shape is (d, )、(d, )、(d, ). S3, calculate the attention score: by calculating the dot product of the query and the key, we get the attention score of each word to other words. To avoid the value being too large, we usually divide it by To scale: The softmax function ensures that the sum of the attention weights is 1. S4, weighted summation: Use the attention weights to perform weighted summation on the Value to obtain a new representation. The shape of the attention score matrix is ​​(n,n), which represents the attention weight of each position in the sequence to other positions. After weighted summation, the resulting matrix shape is (n, ).

[0014] In order to enhance the expressiveness of the model, Transformer uses a multi-head attention mechanism. The basic idea is to divide the self-attention mechanism into multiple "heads", each head calculates attention independently, and merges their results. The specific steps are as follows: S1, multi-head calculation: pass the input X through different weight matrices , and Mapped to different Query, Key and Value spaces. Each head calculates self-attention independently:

[0015] in , , .

[0016] S2, splicing result: splice the output of all heads together: in, It is a learnable linear transformation matrix used to adjust the concatenated dimensions.

[0017] The autonomous driving target detection module of the present invention adopts the YOLOv5 model, which is a real-time object detection algorithm of the YOLO series, combining high efficiency, accuracy and ease of use. It uses CSPDarknet53 as the backbone network, uses PANet for feature fusion, and outputs the target category and bounding box through YOLO Head. YOLOv5 introduces optimizations such as CSPNet design, Mosaic data enhancement, automatic learning anchor box and CIoU loss function to improve the detection performance. The model has multiple versions (YOLOv5s, YOLOv5m, YOLOv5l, YOLOv5x) to meet different performance and resource requirements. YOLOv5 is widely used in video surveillance, autonomous driving, robots, intelligent security and medical imaging, and continues to maintain its leading position in real-time and accuracy. The network structure of the YOLOv5s model mainly consists of an input end, a backbone network (Backbone), a neck network (Neck), and a head network (Head).

[0018] Mosaic data enhancement is used at the input end to splice four pictures by random scaling, random cropping, and random arrangement, so that the model can recognize targets in a smaller range and detect and identify dangerous scenes and abnormal behaviors, such as the sudden intrusion of pedestrians.

[0019] The main steps of Mosaic data enhancement are: S1, randomly select four images from the dataset as input images.

[0020] S2, randomly crop the four images to ensure that each image retains a certain area that will form part of the final stitched image.

[0021] S3, scales the four cropped images to the same size for subsequent stitching operations.

[0022] S4, stitch the four scaled images together in a 2x2 grid in the order of top left, top right, bottom left, and bottom right to form a new image.

[0023] The backbone network includes Focus structure, CSP structure and SPP structure. The Focus module slices the image, which reduces the information loss in the downsampling process, simplifies numerical calculations and improves the training speed. The CSP module not only reduces the overall computational burden, but also can more effectively extract depth information from features. Two CSP structures are designed in the YOLOv5s model. CSP1_X is applied to the Backbone network, and CSP2_X is applied to the Neck network. The SPP module increases the network's receptive field by converting variable-size feature maps into fixed-size feature vectors. For specific structural diagrams, please refer to Figure 4 .

[0024] The neck network adopts the FPN+PANet structure. The PANet structure is used for dense positioning of high-level features. It performs bottom-up downsampling so that the top-level features contain image location information. The two features are finally fused so that feature maps of different sizes contain image semantic information and image feature information, ensuring accurate prediction of images of different sizes. At the same time, FPN provides powerful underlying semantic features by upsampling the feature map of the last layer of the network, and then fuses them with various multi-scale features to enhance detection capabilities. The FPN network contains a bottom-up circuit, a top-down circuit, and lateral connections. For a specific structural diagram, please refer to Figure 5 .

[0025] The head network performs the final detection at different scales. It outputs a vector containing class probability, confidence score, predicted angle, and bounding box information using anchor boxes. CIOU_LOSS is used as the loss function of the bounding box in the head, which takes into account the overlapping area between the detection box and the target box, the information of the center point distance, the aspect ratio of the bounding box, etc., making the detection of dangerous scenes and abnormal behaviors more accurate.

[0026] In the process of detecting dangerous scenes in autonomous driving, the multi-sensor data is labeled and fused through the large language model, which greatly reduces the number of model parameters and is more conducive to the lightweight target detection model. The preprocessed data and the rare dangerous scene data automatically generated by the large language model are sent to the large language model for pre-training and fine-tuning, and abnormal behaviors and dangerous scenes are preliminarily identified and classified. Finally, the data is sent to the YOLOv5 target detection model to obtain further detection of dangerous scenes. The large language model-based method for detecting dangerous scenes in autonomous driving can greatly enhance the situational understanding, risk assessment, decision support, multi-sensor data fusion, abnormal behavior detection, feedback learning and ethical decision-making capabilities of the autonomous driving system, thereby improving the overall safety and reliability of the system.

Claims

1. A method for detecting dangerous scenes in autonomous driving based on a large language model, characterized in that: include: Data collection, large language model processing, autonomous driving target detection and real-time risk assessment; 1) First, use the data acquisition module to collect multi-source data from cameras and lidar, etc., covering various driving environments and potential dangerous scenarios, and extract useful features from the data of different sensors; 2) The large language model processing module is used to annotate and enhance the data. At the same time, the large language model is used to convert the dangerous scenes described in natural language into specific simulation scenes, and various dangerous scenes are generated for the training and testing of the autonomous driving system to improve the system's ability to respond to real dangerous situations; the processed data is sent to the trained YOLOv5 model for real-time target detection; 3) Based on the current environment and traffic conditions, the autonomous driving target detection module conducts real-time risk assessment, identifies potential dangerous scenarios and issues early warnings; when a dangerous scenario is detected, the large language model can assist the autonomous driving system in making rapid emergency braking or avoidance decisions to ensure the safety of the vehicle and passengers; by analyzing the dangerous scenarios encountered by the autonomous driving system during actual driving, the model can continuously learn and improve its detection algorithm to achieve system feedback.

2. The large language model processing module in the large language model-based automatic driving dangerous scene detection method described in claim 1 is characterized in that: The GPT-4.0 large language model based on the Transformer architecture is adopted, which is implemented through unsupervised pre-training and fine-tuning methods; the large language model is used to annotate and enhance the data, reducing the number of parameters in the target detection process, thereby improving the real-time performance of detection; the large language model generates rare dangerous scenes for autonomous driving training and testing, improving the accuracy and safety of dangerous scene detection during target detection; when a dangerous scene is detected, the large language model can assist the autonomous driving system to quickly make emergency braking or avoidance decisions to ensure the safety of the vehicle and passengers; at the same time, the model can continuously learn and improve its detection algorithm to achieve system feedback.

3. The automatic driving target detection module in the automatic driving dangerous scene detection method based on a large language model as described in claim 1 is characterized in that: The YOLOv5 model is adopted, which is a real-time object detection algorithm in the YOLO series. It uses CSPDarknet53 as the backbone network, PANet for feature fusion, and outputs target categories and bounding boxes through YOLO Head. YOLOv5 introduces optimizations such as CSPNet design, Mosaic data enhancement, automatic learning anchor boxes, and CIoU loss function to improve detection performance.

Images