Driving attention inspired traffic accident risk prediction method and system

Abstract

The invention discloses a traffic accident risk prediction method and system inspired by driving attention, and the method comprises the steps: obtaining a driving scene image, preprocessing the image, and converting the image into continuous image frames; based on attention mechanism text-to-video attention transfer fusion, text information is converted into video representation, and cross-modal features with coherent semantics are generated; semantic context migration is realized on the cross-modal features through a graph convolutional network, and a dual-path model is constructed on the basis of a generated semantic context feature matrix; and a multi-task learning strategy is adopted, and a potential accident risk caused by distraction of the driver is identified and predicted through a dual-path model. According to the invention, key semantic clues are identified in each module by using an attention mechanism, so that the robustness of accident prediction is enhanced, and the driving safety is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of traffic safety, and in particular to a method and system for predicting traffic accident risks inspired by driving attention. Background Art

[0002] Traffic accident prediction aims to identify possible accidents in advance so as to make immediate judgments and effectively prevent traffic accidents. Autonomous driving and assisted driving systems are increasingly focusing on predicting the behavior of nearby road participants in advance, covering trajectory prediction, action prediction, intention prediction, collision avoidance and even end-to-end learning, aiming to improve driving safety and efficiency. However, traffic accidents usually occur at brief and unexpected times and places. Accurate prediction requires identifying sparse spatiotemporal patterns of accidents and mutations from normal to dangerous states. In addition, the diversity of weather and lighting conditions, the complexity of accident types and the high imbalance of data together constitute key factors in this challenge.

[0003] There is an increasing number of studies on video-based traffic accident prediction models, which focus on predicting the time to occurrence (TTA) and type of accidents, especially focusing on the identification of large-scale accidents. In-depth analysis of these models, the core is to explore the coordination or correlation between object trajectory segments and visual context features in deep learning. Spatial interaction models, especially spatiotemporal attention networks, are highly regarded in the field of analyzing visual contexts and accident prediction. However, the occurrence of accidents in complex environmental conditions, such as objects that are difficult to detect or track, is often a key factor that limits the effectiveness of these methods. Some scholars have proposed the use of a gaze-guided deep reinforcement learning accident prediction model. This model constructs a driver attention map during the offline pre-training stage, but it is insufficient in the in-depth exploration of the interaction between cognitive heuristic video understanding and text descriptions.

[0004] Previous studies mainly focused on object-based spatiotemporal correlations, but there are still difficulties in adapting to the inherent long-tail data distribution and significant environmental changes. Summary of the invention

[0005] The purpose of the present invention is to provide a traffic accident risk prediction method and system inspired by driving attention to solve the above problems.

[0006] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for predicting traffic accident risks inspired by driving attention, comprising: Acquire driving scene images and convert them into continuous image frames after preprocessing; Attention-based text-to-video attention transfer fusion, which converts text information into video representation and generates cross-modal features with coherent semantics; The cross-modal features are transferred to the semantic context through the graph convolutional network, and the generated semantic context feature matrix is ​​used to construct a dual-path model. A multi-task learning strategy is adopted to identify and predict potential accident risks caused by driver distraction through a dual-path model.

[0007] Furthermore, the acquisition of the driving scene image and conversion into continuous image frames after preprocessing includes: The size resolution of the collected driving scene image is adjusted to the long side M×the wide side N; Calculate the average pixel value of all images and subtract the average pixel value from each image to eliminate the differences between single-frame images and achieve standardization.

[0008] Furthermore, the attention transfer from text to video based on the attention mechanism is integrated, and the text information is converted into video representation to generate cross-modal features with coherent semantics, including: The video frame with a long side of M and a wide side of N and the text description are processed by patch embedding through a 2D convolution layer with an N×N convolution kernel, and the text is encoded using the pre-trained BERT model to obtain the embedded information of the video and text; The embeddings of video frames and text descriptions are integrated into a multi-head self-attention model with shared weights to capture the correlation between visual features and text words. Construct visual and text embedding vectors that merge displacement fusion mechanism, input position-aware cross-attention cross-modal fusion module, generate cross-modal features that integrate coherent text information, and realize attention-guided displacement fusion between text and video.

[0009] Furthermore, the cross-modal features are transferred to semantic context through graph convolutional networks, including: The integrated cross-modal features are introduced into a layer of graph convolutional network to explore the semantic context in the driving scene and obtain the semantic context feature matrix.

[0010] Furthermore, the generated semantic context feature matrix is ​​used to construct a dual-path model, including: One path involves processing the feature matrix using an accident score decoding module containing gated recurrent units to perform accident prediction at the video level; The other path is used to generate a frame-level driver attention map and input the feature matrix into the module to achieve frame-level reconstruction.

[0011] Furthermore, the multi-task learning strategy is used to identify and predict potential accident risks caused by driver distraction through a dual-path model, including: Driver attention map reconstruction, based on the semantic context feature matrix, uses multi-layer deconvolution layers and self-attention networks to achieve driver attention map reconstruction, providing key traction for semantic learning in accident prediction; The accident score decoding module using gated recurrent units processes the feature matrices of positive and negative examples respectively to capture the dynamic changes of semantic context features in the time dimension, and then calculates the final accident risk score through the softmax function to predict traffic accidents. Furthermore, in model training, the driver attention map reconstruction and accident risk prediction tasks are collaboratively optimized to train the model.

[0012] In a second aspect, the present invention provides a traffic accident risk prediction system inspired by driving attention, comprising: A data acquisition module is used to acquire driving scene images and convert them into continuous image frames after preprocessing; The cross-modal feature generation module is used for attention transfer fusion from text to video based on the attention mechanism, and converts text information into video representation to generate cross-modal features with coherent semantics; The dual-path model building module is used to implement semantic context transfer of cross-modal features through a graph convolutional network, and to construct a dual-path model with the semantic context feature matrix generated thereby; The prediction output module is used to identify and predict potential accident risks caused by driver distraction through a dual-path model using a multi-task learning strategy.

[0013] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a driving attention-inspired traffic accident risk prediction method when executing the computer program.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the traffic accident risk prediction method inspired by driving attention are implemented.

[0015] Compared with the prior art, the present invention has the following technical effects: This paper introduces a traffic accident risk prediction method inspired by driving attention, which aims to optimize the model training process by drawing on the driver's attention pattern and description based on visual text scenes. In our study, we explored the relationship between text descriptions and driver attention in driving situations, aiming to assist in the rapid identification of relevant objects in accidents by providing semantic guidance. Effectively capture the spatiotemporal correlation between objects in the scene. Enhance its adaptability in complex scenes and harsh environments. Significantly enhance the prediction accuracy and reliability of the system.

[0016] The present invention proposes a traffic accident risk prediction method based on driving attention inspiration. The method uses a shared weight self-attention network to process continuous frame images and text descriptions before the accident, generates visual and semantic fusion features, integrates text and visual information through a position-aware cross-attention fusion module to maintain the token position information after the fusion layer, explores the connection between local and global token representations, and assists video-based accident prediction in traffic scenes. The fused cross-modal features are input into two paths through the GCN network, which are used for video-level accident prediction and frame-level driver attention map reconstruction respectively, and the gated recurrent unit is used to associate the changes of semantic context over time. The semantic context feature matrix at time t is used to reconstruct the driver attention map to learn the core semantics in accident prediction. Through multi-task optimization, the model samples positive and negative video clips in the traffic accident prediction path, and finally obtains the predicted accident probability. The text description and driver attention are integrated into the prediction network to provide dense semantic guidance for traffic scene content and efficiently locate key areas closely related to safe driving. The attention mechanism is used to identify key semantic clues in each module to enhance the robustness of accident prediction, thereby significantly improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of the traffic accident risk prediction method inspired by driving attention in the present invention; Figure 2 Schematic diagram of the structure of the traffic accident risk prediction model inspired by driving attention in the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.

[0019] Example 1, please refer to Figure 1 , a traffic accident risk prediction method inspired by driving attention, including: Acquire driving scene images and convert them into continuous image frames after preprocessing; Attention-based text-to-video attention transfer fusion, which converts text information into video representation and generates cross-modal features with coherent semantics; The cross-modal features are transferred to the semantic context through the graph convolutional network, and the generated semantic context feature matrix is ​​used to construct a dual-path model. A multi-task learning strategy is adopted to identify and predict potential accident risks caused by driver distraction through a dual-path model.

[0020] The model training process is optimized by drawing on the driver's attention pattern and description based on visual text scenes. In our research, we explored the relationship between text descriptions and driver attention in driving situations, aiming to assist in the rapid identification of relevant objects in accidents by providing semantic guidance. Effectively capture the spatiotemporal correlation between objects in the scene. Enhance its adaptability in complex scenes and harsh environments. Significantly enhance the prediction accuracy and reliability of the system.

[0021] Embodiment 2, the present invention provides a method for predicting traffic accident risks inspired by driving attention, such as Figure 1 and Figure 2 As shown, the following steps are included: Step S1, capturing a driving scene image from the perspective of a vehicle-mounted camera and converting it into a continuous image frame through a preprocessing technique; Step S2: Attention transfer fusion from text to video based on the attention mechanism, and converting text information into video representation, thereby generating cross-modal features with coherent semantics; Step S3, the cross-modal features are transferred through the graph convolutional network through the attention mechanism to achieve semantic context transfer, and the semantic context feature matrix generated thereby is input into the two paths respectively; Step S4 adopts a multi-task learning strategy to construct a dual-path model, aiming to effectively identify and predict potential accident risks caused by driver distraction.

[0022] Wherein step S1 includes the following steps: Step S11, adjusting the size resolution of the collected driving scene image to long side M×wide side N; Step S12, calculate the average pixel value of all images, and subtract the average pixel value from each image, in order to eliminate the differences between single-frame images and achieve standardized processing.

[0023] Wherein step S2 comprises the following steps: Step S21, the video frame with a long side of M and a wide side of N and the text description are subjected to patch embedding processing respectively through a 2D convolution layer with a convolution kernel of size N×N, and the text is encoded using a pre-trained BERT model, so as to obtain embedded information of the video and text; Step S22, integrating the embeddings of the video frame and the text description into a multi-head self-attention model with shared weights to characterize the correlation between the visual features and the text vocabulary; Step S23, constructing the visual and text embedding vectors of the fused displacement fusion mechanism, and inputting them into the cross-modal fusion module of position-aware cross-attention, aims to generate cross-modal features that integrate coherent text information, thereby achieving accurate attention-guided displacement fusion between text and video.

[0024] Wherein step S3 includes the following steps: Step S31, introducing the integrated cross-modal features into a layer of graph convolutional network to explore the semantic context in the driving scene, thereby obtaining a semantic context feature matrix; Step S32, one path involves processing the feature matrix using an accident score decoding module including a gated recurrent unit to perform accident prediction at the video level; In step S33, another path is used to generate a frame-level driver attention map, and the feature matrix is ​​input into the module to achieve frame-level reconstruction.

[0025] Wherein step S4 comprises the following steps: Step S41, driver attention map reconstruction, based on the semantic context feature matrix, through the use of multiple deconvolution layers, using the self-attention network to achieve driver attention map reconstruction, aiming to provide key traction for semantic learning in accident prediction; Step S42, traffic accident prediction, using the accident score decoding module of the gated recurrent unit to process the feature matrices of the positive and negative examples respectively to capture the dynamic changes of the semantic context features in the time dimension, and then calculate the final accident risk score through the softmax function, so as to achieve effective prediction of traffic accidents; Step S43, multi-task optimization, in model training, the model is trained by collaboratively optimizing the driver attention map reconstruction and accident risk prediction tasks.

[0026] Example 3 Acquire driving scene images from a first-person perspective and perform preprocessing to generate a continuous frame sequence; the specific operation includes inputting 5 frames of image input and uniformly adjusting the resolution of all images to 224×224×3 to optimize computational efficiency. Furthermore, perform a subtraction operation on each frame of the image, subtracting the average frame pixel value of all images, thereby reducing the impact of significant differences between single images on subsequent analysis. A 16×16 2D convolution kernel is used to perform patch embedding operations. In the present invention, the text description is encoded via a pre-trained BERT model, and the token length is specifically set to 15.

[0027] The embedding information of video frames and text descriptions is integrated into a multi-head self-attention model with eight shared weight heads, aiming to characterize the correlation between visual features and text vocabulary. In the visual path, the number of tokens is 49, and in the text path, the number of tokens is 15. Subsequently, the text and visual modalities are integrated through a cross-modal fusion network with a position-aware cross-attention mechanism to generate feature information containing 64 tokens. The position embedding significantly enhances visual semantic learning in the cross-layer shift operation of visual tokens, while distributing coherent text information during the learning process, thereby achieving attention-guided text and video fusion.

[0028] The fused cross-modal features are processed by a layer of graph convolutional network to explore the semantic context of the driving scene. The generated 64×5 semantic context feature matrix is ​​input into two paths, one for video-level accident prediction and the other for constructing a frame-level driver attention map.

[0029] In the accident prediction path, we use an accident score decoding module with a gated recurrent unit to capture and associate the dynamic changes of semantic context features in the time series. By performing a maximum pooling operation on the feature matrix at the node level, we then apply a softmax function to generate the probability of an accident. To optimize the model performance, we adopted a sampling strategy to distinguish between positive and negative examples, where the positive examples are selected based on the last frame of the sample in the accident window, while the negative examples are generated from the time window before the accident to construct a balanced training set. In the path of constructing the driver's attention map at the frame level, the feature matrix is ​​processed through 5 deconvolution layers and a batch normalization layer to generate a driver's attention map. In the present invention, driver attention map reconstruction and traffic accident prediction constitute a multi-task learning problem, and the loss function is:

[0030] where is a parameter for balancing the error gap and is set to 5 in all experiments.

[0031] In an embodiment of the present invention, the system structure adopts a multi-input multi-output model and is optimized through an end-to-end training strategy. In terms of hardware, a high-performance computing platform equipped with NVIDIA RTX3090 GPU and 64G memory is configured. In order to improve the adaptability and accuracy of the model, we set different learning rates for the self-attention module, cross-modal fusion layer, GRU, and the decoding module of the driver's attention map, specifically set to,, and. All experiments were trained within 10 cycles, and the batch size was fixed to 2. In order to improve the network training effect, we expanded the custom-collected DADA-2000 data set, especially before the accident, adding text descriptions based on actual visual observations to enhance data diversity and richness. In the present invention, we experimentally constructed it on the DADA-2000 data set for verification. Then, we also evaluated the performance of the CCD data set. The DADA-2000 data set consists of 658,476 frames of video, with an average video length of nearly 230 frames (based on a rate of 30 frames per second), and records the driver's attention data in each frame of video, covering 54 types of accident scenes. We selected 512 and 168 original video sequences for training and testing. The CCD dataset contains 1500 dashcam accident videos. Each video is segmented into 50 frames with a total duration of 5 seconds. In the CCD dataset, the pre-set test set contains 900 positive samples and no negative samples. In the evaluation process on the DADA-2000 dataset, our method achieved significant performance enhancement. In the evaluation of the CCD dataset, our method shows comparable performance compared with existing methods.

[0032] In yet another embodiment of the present invention, a traffic accident risk prediction system inspired by driving attention is provided, which can be used to implement the above-mentioned traffic accident risk prediction method inspired by driving attention. Specifically, the system includes: A data acquisition module is used to acquire driving scene images and convert them into continuous image frames after preprocessing; The cross-modal feature generation module is used for attention transfer fusion from text to video based on the attention mechanism, and converts text information into video representation to generate cross-modal features with coherent semantics; The dual-path model building module is used to implement semantic context transfer of cross-modal features through a graph convolutional network, and to construct a dual-path model with the semantic context feature matrix generated thereby; The prediction output module is used to identify and predict potential accident risks caused by driver distraction through a dual-path model using a multi-task learning strategy.

[0033] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0034] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of a traffic accident risk prediction method inspired by driving attention.

[0035] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of a traffic accident risk prediction method inspired by driving attention in the above embodiment.

[0036] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0037] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0038] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0039] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A traffic accident risk prediction method inspired by driving attention, characterized in that: include: Acquire driving scene images and convert them into continuous image frames after preprocessing; Attention-based text-to-video attention transfer fusion, which converts text information into video representation and generates cross-modal features with coherent semantics; The cross-modal features are transferred to the semantic context through the graph convolutional network, and the generated semantic context feature matrix is ​​used to construct a dual-path model. A multi-task learning strategy is adopted to identify and predict potential accident risks caused by driver distraction through a dual-path model.

2. The method for predicting traffic accident risk based on driving attention according to claim 1, characterized in that: The step of acquiring the driving scene image and converting it into a continuous image frame after preprocessing includes: The size resolution of the collected driving scene image is adjusted to the long side M×the wide side N; Calculate the average pixel value of all images and subtract the average pixel value from each image to eliminate the differences between single-frame images and achieve standardization.

3. The method for predicting traffic accident risk based on driving attention according to claim 2, characterized in that: The attention-based text-to-video attention transfer fusion converts text information into video representation and generates cross-modal features with coherent semantics, including: The video frame with a long side of M and a wide side of N and the text description are processed by patch embedding through a 2D convolution layer with an N×N convolution kernel, and the text is encoded using the pre-trained BERT model to obtain the embedded information of the video and text; The embeddings of video frames and text descriptions are integrated into a multi-head self-attention model with shared weights to capture the correlation between visual features and text words. Construct visual and text embedding vectors that merge the displacement fusion mechanism, input the cross-modal fusion module of position-aware cross-attention, generate cross-modal features that integrate coherent text information, and realize the displacement fusion between attention-guided text and video.

4. The method for predicting traffic accident risk based on driving attention according to claim 1, characterized in that: The cross-modal features are transferred through graph convolutional networks to achieve semantic context migration, including: The integrated cross-modal features are introduced into a layer of graph convolutional network to explore the semantic context in the driving scene and obtain the semantic context feature matrix.

5. The method for predicting traffic accident risk based on driving attention according to claim 4, characterized in that: The generated semantic context feature matrix is ​​used to construct a dual-path model, including: One path involves processing the feature matrix using an accident score decoding module containing gated recurrent units to perform accident prediction at the video level; The other path is used to generate a frame-level driver attention map and input the feature matrix into the module to achieve frame-level reconstruction.

6. The method for predicting traffic accident risk based on driving attention according to claim 5, characterized in that: The multi-task learning strategy is used to identify and predict potential accident risks caused by driver distraction through a dual-path model, including: Driver attention map reconstruction, based on the semantic context feature matrix, uses multi-layer deconvolution layers and self-attention networks to achieve driver attention map reconstruction, providing key traction for semantic learning in accident prediction; The accident score decoding module using gated recurrent units is used to process the feature matrices of positive and negative examples respectively to capture the dynamic changes of semantic context features in the time dimension. The final accident risk score is then calculated through the softmax function to predict traffic accidents.

7. The method for predicting traffic accident risk based on driving attention according to claim 6, characterized in that: In model training, the driver attention map reconstruction and accident risk prediction tasks are collaboratively optimized to train the model.

8. A traffic accident risk prediction system inspired by driving attention, characterized in that: include: A data acquisition module is used to acquire driving scene images and convert them into continuous image frames after preprocessing; The cross-modal feature generation module is used for attention transfer fusion from text to video based on the attention mechanism, and converts text information into video representation to generate cross-modal features with coherent semantics; The dual-path model building module is used to implement semantic context transfer of cross-modal features through a graph convolutional network, and to construct a dual-path model with the semantic context feature matrix generated thereby; The prediction output module is used to identify and predict potential accident risks caused by driver distraction through a dual-path model using a multi-task learning strategy.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a traffic accident risk prediction method inspired by driving attention as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a traffic accident risk prediction method inspired by driving attention as described in any one of claims 1 to 7 are implemented.

Images