Spatial channel adaptive accident prediction method and system based on eye movement attention guidance

By using the space channel adaptation module and channel adaptation module guided by driver eye movement information, combined with the time information recursive transmission strategy, the problems of high false alarm rate and low accuracy of traffic accident prediction in autonomous driving environments are solved, and smart cars are improved security.

CN118470484BActive Publication Date: 2024-11-26UNIV OF JINAN
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410672169.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-28
Publication Date
2024-11-26
Estimated Expiration
2044-05-28

AI Technical Summary

Technical Problem

The existing traffic accident prediction methods have problems with high false alarm rates and low accuracy in autonomous driving environments, especially when drivers cannot continue to pay attention to the driving environment, it is not suitable to collect eye movement information in real time as input.

Method used

A spatial channel adaptive accident prediction method based on eye movement attention guidance is proposed. Through the training model, the driver's eye movement information is used to guide the extraction of video data of the driving recorder. The space adaptive module and the channel adaptive module are combined to introduce time Recursive information transmission strategy to mitigate the impact of early results in the inference process on model performance.

Benefits of technology

It improves the accuracy and reliability of traffic accident prediction, reduces the false alarm rate, enhances the safety of smart cars, and performs better than the current state-of-the-art models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470484B_ABST
    Figure CN118470484B_ABST
Patent Text Reader

Abstract

The invention belongs to the field of computer vision, and provides a spatial channel adaptive accident prediction method and system based on eye movement attention guidance, comprising acquiring a driving recorder image, performing feature extraction using a trained backbone network, and obtaining a driving recorder spatial channel feature map; generating patches of different scales using residual block outputs of different scales of the backbone network, and obtaining multi-scale multi-view features after splicing the patches of different scales; extracting time features through a temporal self-attention module, a spatial cross-attention module, and a feedforward neural network based on a custom initial historical time series feature map, and obtaining a current time series feature map, wherein the spatial cross-attention module uses the multi-scale multi-view features as input to extract spatial information, and recursively extracts the current time series feature map as a historical time series feature map of the next timestamp; and fusing and classifying the current time series feature map with the driving recorder spatial channel feature map to obtain an accident prediction result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a spatial channel adaptive accident prediction method and system based on eye movement attention guidance. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, autonomous driving technology has made great progress. However, while meeting the needs of convenience and emerging industries, autonomous driving also brings many hidden dangers to traffic safety and arouses people's concerns about traffic accidents. In addition, according to the World Health Organization's "2023 Global Road Safety Report", although the number of deaths in traffic accidents has decreased in recent years, the death toll is still as high as 1.15 million, which is shocking. Therefore, integrating traffic accident prediction into autonomous vehicles and advanced driver assistance systems (ADAS) has become a top priority to improve travel safety. Accurately predicting impending traffic accidents in advance can effectively help autonomous vehicles and human drivers take emergency measures in time, thereby reducing the possibility of accidents. Computer vision researchers have long been committed to exploring the task of accident prediction and regard it as a daunting challenge.

[0004] Computer vision approaches to accident prediction mainly involve analyzing dashcam video data. Given the rapid movement of objects in real driving environments, these video data often require complex spatiotemporal interactions. Initial research efforts focused on anomaly detection in videos, mainly using deep neural networks for object detection and trajectory prediction. In recent years, some studies have gone deeper and aimed to collect accident-related features or relationships from dashcam videos to predict accidents. Their focus is on predicting accidents as early as possible, which can result in a large number of false positives, hindering practical applications.

[0005] Inspired by the human selective attention mechanism, the driver's attention is an active visual cue for safe driving. Critical driving judgments and decisions depend heavily on the driver's observation of the driving environment, so the area they focus on is crucial to autonomous driving and assisted driving systems. Commercial eye tracking devices help capture the driver's gaze area in the driving environment, making the driver's eye movement information a valuable and usable input for autonomous driving and assisted driving systems. However, during autonomous driving operations, some drivers may not be able to continuously focus on the driving environment. Therefore, collecting the driver's eye movement information in real time as input to the autonomous driving system is not a suitable method.

[0006] Considering this problem, the driver's eye movement information can be used as a special attention mechanism or supervisory signal to monitor the learning of spatiotemporal features in accident prediction. This mechanism effectively directs the network model's attention to important objects in the driving environment, potentially eliminating the need for object detection. However, in existing methods, different results in the neural network model reasoning process have a greater impact on the model performance, resulting in a decrease in the accuracy of the subsequent prediction process. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a spatial channel adaptive accident prediction method and system based on eye movement attention guidance. The present invention uses dashcam video data combined with driver eye movement information supervision training to develop a new model for early accident prediction. By adaptively focusing on the complex spatiotemporal interactions in the driving environment, the scene appearance features and spatiotemporal relationship features related to accidents are effectively learned. Features are extracted from dashcam video data and a heat map generated by processing the driver's eye movement data. It is worth noting that the method proposed in the present invention uses the driver's eye movement data to guide the training of the model in the training phase, and does not use eye movement data inference in the inference phase.

[0008] According to some embodiments, a first solution of the present invention provides a spatial channel adaptive accident prediction method based on eye movement attention guidance, which adopts the following technical solutions:

[0009] The spatial channel adaptive accident prediction method based on eye movement attention guidance includes:

[0010] Obtain the dashcam image, use the trained backbone network to extract features, and obtain the dashcam spatial channel feature map;

[0011] The residual blocks of different scales of the backbone network are used to generate patches of different scales, and the patches of different scales are spliced ​​to obtain multi-scale multi-view features;

[0012] Based on the customized initial historical time series feature map, the time features are extracted through the time self-attention module, the spatial cross-attention module and the feedforward neural network to obtain the current time series feature map. The spatial cross-attention module takes the multi-scale multi-view features as input to extract spatial information, and uses the current time series feature map as the historical time series feature map of the next timestamp for recursive extraction.

[0013] The current time series feature map is fused and classified with the driving recorder spatial channel feature map to obtain the accident prediction result;

[0014] Among them, during the training stage, the backbone network is trained using the eye movement attention heat map and the dashcam image respectively, and the feature cosine similarity obtained from the processing of the eye movement attention heat map and the dashcam image is used to guide the training process using the dashcam image to obtain a trained backbone network.

[0015] According to some embodiments, a second solution of the present invention provides a spatial channel adaptive accident prediction system based on eye movement attention guidance, which adopts the following technical solutions:

[0016] The spatial channel adaptive accident prediction system based on eye movement attention guidance includes:

[0017] The spatial feature extraction module is configured to obtain a driving recorder image, perform feature extraction using a trained backbone network, and obtain a driving recorder spatial channel feature map;

[0018] The backbone network is trained using the eye movement attention heat map and the dashcam image respectively during the training phase, and the feature cosine similarity obtained from the processing of the eye movement attention heat map and the dashcam image guides the training process using the dashcam image to obtain a trained backbone network;

[0019] The patch embedding module is configured to generate patches of different scales using the residual block outputs of different scales of the backbone network, and to concatenate the patches of different scales to obtain multi-scale multi-view features;

[0020] A temporal feature extraction module is configured to extract temporal features based on a customized initial historical temporal feature map through a temporal self-attention module, a spatial cross-attention module, and a feed-forward neural network to obtain a current temporal feature map, wherein the spatial cross-attention module uses multi-scale multi-view features as input to extract spatial information, and uses the current temporal feature map as a historical temporal feature map for the next timestamp for recursive extraction;

[0021] The accident prediction module is configured to fuse and classify the current time series feature map with the driving recorder spatial channel feature map to obtain an accident prediction result.

[0022] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the spatial channel adaptive accident prediction method based on eye movement attention guidance as described in the first aspect above.

[0024] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0025] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the spatial channel adaptive accident prediction method based on eye movement attention guidance as described in the first aspect above are implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] This paper proposes a novel traffic accident prediction method, the highlight of which is to use the driver's gaze information as a supervisory auxiliary means to train the traffic accident prediction model. A new model called Eye-SCAN is proposed. This model uses the driver's gaze information to guide the feature extraction of dashcam video data and maps through a backbone network, aiming to achieve early prediction of traffic accidents. Eye SCAN includes two key submodules: spatial adaptive module (SAM) and channel adaptive module (CAM), which are used to extract spatial semantic information and channel semantic information from the driver's line of sight, respectively. In addition, a new temporal information recursive transmission strategy is introduced to mitigate the impact of different early results in the reasoning process on model performance. Through experiments on two benchmark datasets, the model of the present invention outperforms the current state-of-the-art models in terms of performance, and the contribution of each component is analyzed in detail. These results provide an effective solution for improving the safety of smart cars. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 This is a general framework diagram of a spatial channel adaptive accident prediction method based on eye movement attention guidance in an embodiment of the present invention;

[0030] Figure 2 is a diagram of a spatial channel adaptive traffic accident prediction network based on driver eye movement information in an embodiment of the present invention;

[0031] Figure 3 1 is a structural diagram of a multi-scale spatiotemporal transformer according to an embodiment of the present invention;

[0032] Figure 4 1 is a structural diagram of a space adaptation module SAM and a channel adaptation module CAM in an embodiment of the present invention;

[0033] Figure 5 The driver's eye movement heat map in the embodiment of the present invention;

[0034] Figure 6Visualization of the driver's eye movement heat map on the dashcam in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0039] Embodiment 1

[0040] The present embodiment provides a spatial channel adaptive accident prediction method based on eye movement attention guidance. The present embodiment takes the application of this method to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal, a server, and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0041] Obtain the dashcam image, use the trained backbone network to extract features, and obtain the dashcam spatial channel feature map;

[0042] The residual blocks of different scales of the backbone network are used to generate patches of different scales, and the patches of different scales are spliced ​​to obtain multi-scale multi-view features;

[0043] Based on the customized initial historical time series feature map, the time features are extracted through the time self-attention module, the spatial cross-attention module and the feedforward neural network to obtain the current time series feature map. The spatial cross-attention module takes the multi-scale multi-view features as input to extract spatial information, and uses the current time series feature map as the historical time series feature map of the next timestamp for recursive extraction.

[0044] The current time series feature map is fused and classified with the driving recorder spatial channel feature map to obtain the accident prediction result;

[0045] Among them, during the training stage, the backbone network is trained using the eye movement attention heat map and the dashcam image respectively, and the feature cosine similarity obtained from the processing of the eye movement attention heat map and the dashcam image is used to guide the training process using the dashcam image to obtain a trained backbone network.

[0046] like Figure 1 As shown, the method described in this embodiment can be combined with time series information to perform efficient driver cognitive distraction detection. Specifically, it includes:

[0047] First, the method proposed in this embodiment uses the driver's eye movement information and the dashcam image (DCI) as the input of the model during training, and generates a driver's attention heat map (Map) through preprocessing of the driver's eye movement information. The pixel values ​​in the Map represent the most attractive areas for experienced drivers during driving. Specifically, this embodiment uses a preprocessing method based on gaze points to construct a driver's attention heat map. The preprocessing method first generates an attention heat map Map through gaze points, and then performs a weighted combination of the current heat map frame and the historical heat map frame at a certain moment, so that the driver's attention heat map finally generated has long-time gaze point information.

[0048] Because the Map itself has the characteristics of the driver's attention during driving, in order to enable the driving safety model to better learn the semantic information in the Map, this embodiment proposes a spatial channel adaptive traffic accident prediction network (Eye-SCAN) based on the driver's eye movement information. Two sub-modules are proposed in the Eye-SCAN network: a spatial adaptive module (SAM) for low-dimensional driver eye movement spatial semantic information adaptation and a channel adaptive module (CAM) for high-dimensional semantic information adaptation. In addition, a new temporal information recursive transmission strategy is introduced to reduce the impact of previous result changes on the current model reasoning.

[0049] As shown Figure 2 as well as Figure 4As shown, Map is first preprocessed through DCI and a parameter-sharing backbone network for feature extraction. The selected backbone network is ResNet50, where stage0 represents the initial convolution pool operation in ResNet, and stage1, Stage2, Stage3, and Stage4 represent residual blocks of different scales of ResNet, respectively. In this process, two modules, SAM and CAM, are introduced to learn the deep semantics of the driver's eye movement information at different scales. It is worth noting that in the reasoning stage, this embodiment does not use the driver's eye movement information. In the training stage, Map and DCI are processed simultaneously, and in the reasoning stage, only DCI is processed; this is equivalent to Map guiding the training of the DCI backbone network. (Both SAM and CAM will output a feature vector for similarity calculation, so that DCI can perceive the guidance of Map during the reasoning process).

[0050] Specifically, when extracting the map and dashcam images in the parameter-sharing backbone network, Eye SCAN deeply learns the semantics of the driver's eye movement information at different scales through the SAM and CAM modules. To handle temporal information, a multi-scale spatiotemporal transformer (MSTFormer) is used to recursively extract historical features and build a long-term sequence model for accident prediction. The cross-attention module in MSTFormer is responsible for extracting dashcam features using the map information obtained from Eye SCAN, while the self-attention module of MSTFormer is responsible for recursively extracting information from previous video frame features.

[0051] The key technologies are embodied in three aspects: the spatial adaptation module (SAM) for low-dimensional spatial semantic information adaptation of the driver's eye movement, the channel adaptation module (CAM) for high-dimensional semantic information adaptation, and the recursive transmission strategy of temporal information that can alleviate the impact of previous result changes on the current model reasoning.

[0052] (1) Data preparation. The present invention selected the DAD and CCD datasets, which contain accident videos and non-accident videos, among which the accident videos also contain the time mark of the start of the accident. The present invention collects the driver's eye movement information based on these two datasets. Specifically, the present invention recruited 8 volunteers, each of whom had at least 3 years of driving experience, and used a 90Hz eye tracking device (Tobii Eye Tracking 4C) to collect eye tracking motion data. The collected eye tracking data generated a Map at a sampling rate of 30Hz. The volunteers were asked to focus on the key areas in the DAD and CCD driving scenes based on their driving experience. Processing of driver motion information: The annotation coordinates x captured by the eye tracking device are converted into t and tTo enhance the prominence of the eye movement data features, x t and t Represented as a solid circle with a radius of 6. In order to incorporate temporal information, the long-term driver gaze data is modeled by weighting the Map of the current frame with the Map of the historical frames.

[0053]

[0054] Among them, F t , and F t-1 They represent the newly generated current frame Map, the original current frame Map, and the historical frame Map, respectively. In addition, threshold processing is implemented to clear the pixels in the Map that are below a certain threshold. The weighting parameter α is set to 0.991, while the threshold is set to 55. This process enhances the understanding and analysis of eye tracking data and paves the way for future investigations. The maps generated by this processing method are shown in Figure 5 The visualization of these maps projected onto the corresponding Dashcam is shown in Figure 6 shown.

[0055] (2) Training phase. In order to process the CCD and DAD datasets, the Segformer model pre-trained on the urban landscape dataset is used to pre-process the video frames of the dataset. Assume that the CCD and DAD video data consist of 50 frames and 100 frames, respectively, which consumes a lot of computing resources. To solve this problem, gradient backpropagation and parameter update are only performed on the last 25 frames of video data. For the previous frames, only recursive feature propagation is performed without gradient calculation. During the training of Eye SCAN, the AdamW optimizer was used, and the learning rate and weight decay rate were set to 0.00001 and 0.01, respectively. The batch size and epochs used to train the model of this embodiment are set to 4 and 20, respectively. Eye SCAN is implemented using PyTorch. Training and testing are performed on a server equipped with an Intel(R) Xeon(R) Gold 6226R central processing unit (CPU) and an NVIDIA GeForce RTX 3090GPU.

[0056] The technical solution adopted in the training stage of this embodiment is as follows:

[0057] Specifically, during the training phase, the backbone network is trained using the eye movement attention heat map, specifically:

[0058] Based on the eye movement attention heat map, the spatial adaptive module is used to learn and output the driver's eye movement spatial adaptive semantic information, and the residual block is used to extract the features of the output of the spatial adaptive module to obtain the attention space feature map;

[0059] Repeat the above process three times and output the attention space feature maps of different scales;

[0060] Based on the attention space feature map outputted last time, the channel adaptation module is used to learn the driver's eye movement channel adaptive semantic information and output the attention channel feature map, and the residual block is used to extract features from the output of the channel adaptation module;

[0061] Repeat the above process once more and output the attention channel feature maps of different scales;

[0062] The attention channel feature map is processed by the fully connected layer to obtain the eye movement attention heat map feature map;

[0063] The sum of category cross entropy loss, frame-level loss, spatial adaptation loss, and channel adaptation loss is used as the training loss of the backbone network until the model converges to obtain a trained backbone network.

[0064] The training process of the multi-scale spatiotemporal transformer (MSTFormer) is similar to the application process, specifically:

[0065] Based on the customized initial historical time series feature graph, feature extraction is performed through the time self-attention module to obtain historical time features;

[0066] Input the historical time features and multi-scale multi-view features into the spatial cross attention module for feature extraction to obtain the historical time patch features;

[0067] The historical time patch features are extracted using a feedforward neural network and then normalized to obtain the current time series feature map;

[0068] The obtained current time series feature graph is used as the historical time series feature graph of the next timestamp to recursively perform time series extraction.

[0069] That is to say, the SAM module is used in the low dimension of the deep learning model to learn the adaptive semantic information of the driver's eye movement space, and the attention space feature map and the driving recorder's attention space feature map are obtained.

[0070] The CAM module is used in the high-dimensional deep learning model to learn the adaptive semantic information of the driver's eye movement channel to obtain the attention channel feature map and the driving recorder attention channel feature map.

[0071] For the multi-scale space-time transformer (MSTFormer), a temporal information recursive transmission strategy is used to transmit recursive information.

[0072] like Figure 4 As shown, the specific implementation steps of the SAM module are as follows:

[0073] Patch embedding is performed on both DCI and Map features of size B*C*H*W to generate features of size B*(hw)*(Cs1s1), where B, C, H, W represent the batch size, number of channels, height and width of the feature, s1, s2 represent the size of the divided tokens, and H, W represent the number of tokens in the H and W dimensions. In other words, the process is that the features of size B*C*H*W entering SAM will first undergo the patch embedding process, specifically, the features are sliced ​​evenly, and the flattened patches are fused after slicing to output features of size B*(hw)*(Cs1s1).

[0074] The features obtained from the patch embedding are average pooled to obtain features of dimension B*(hw)*1. The second dimension of the feature represents the weight ratio of each token in the entire feature.

[0075] The dimension after pooling is adjusted to B*(hw) through squeezing operation. In order to reflect the guidance of Map feature on DCI feature, the feature cosine similarity is calculated by the feature vectors output by DCI feature and Map feature after squeezing operation.

[0076] At the same time, the output feature after the squeeze operation is activated by the tanh activation function and then multiplied by the unpooled feature of dimension B*(hw)*(Cs1s1) to achieve adaptive weighting for each token.

[0077] The features adjusted by adaptive weights are reshaped back to the original scale of B*C*H*W for further adaptive learning at different scales. This process ensures that the SAM module can dynamically adjust the weights of different labels of input features to better adapt to information at different scales.

[0078] like Figure 4 As shown, the specific implementation steps of the CAM module are as follows:

[0079] The DCI and Map features with dimensions of B*C*H*W are both average pooled and squeezed to obtain features with dimensions of B*C.

[0080] In order to reduce the computational cost and the relationship between learning channels, a linear layer is used to reduce the dimension of features with dimension B*C to obtain features with dimension B*(C / 2).

[0081] Next, in order to reflect the guidance of Map features on DCI features, the B*(C / 2) features output by DCI features and Map features after squeezing operation are compared with the cosine similarity of the output B*(C / 2) features with reduced dimension, and the cosine similarity loss of the features output by the two at the same scale is calculated.

[0082] At the same time, the features of dimension B*(C / 2) output after the squeezing operation are processed by softmax and linear layers to obtain features of size B*C.

[0083] After activation by the tanh activation function, the feature is multiplied by the unprocessed feature of size B*C*H*W to achieve an output with adaptive weighting.

[0084] This output is further passed to the residual blocks of different scales of ResNet for feature extraction. This process ensures that the input features of the CAM module establish the relationship between channels and adaptively adjust the weights, so that the features are better adapted to the feature extraction process of the residual blocks of different scales of ResNet.

[0085] like Figure 1 As shown, it can be understood that SAM and CAM are not directly connected, and the output of SAM will not be directly given to CAM. Instead, there is a residual block of the backbone network ResNet50 between SAM and CAM, that is, the output of SAM will be processed by ResNet, and then sent to CAM for further processing.

[0086] like Figure 3 As shown, the specific implementation steps of the MSTFormer are as follows:

[0087] The DCI features after adaptive learning of ResNet, SAM and ResNet, CAM modules are divided into fixed-size patches. Considering the minimum size feature output by the residual block, the patch size is set to 7*7. Each patch has a dimension of 7*7*channel. Different residual blocks have different feature channels. Therefore, a fully connected layer is applied to adjust the patch dimension to 768 for subsequent processing.

[0088] The features of the same dimension obtained after splitting the output features of residual blocks of different scales are concatenated, and a learnable class embedding and a learnable position embedding are prepared for the embedded patch sequence and added to the patch embedding to preserve the position information.

[0089] The obtained spatial features are input into the cross-attention module in Tansformer to capture spatial information. For temporal information, a historical feature is defined and input into the self-attention mechanism module in Transformer as a BEV feature to recursively capture temporal features. Moreover, the historical feature will be processed by Transformer to obtain the current feature, which will be used as the historical feature of the next timestamp process.

[0090] To solve the problem that the class label corresponding to the early timestamp feature information may affect the result of the current timestamp feature, the present invention stipulates that when the class label corresponding to each timestamp feature information changes, recursive historical feature initialization should be performed to eliminate the influence of different results of early timestamps on the inference of the current timestamp feature.

[0091] The loss of the model in the training stage consists of four parts. First is the categorical cross-entropy loss, which aims to enable the model to identify videos with accidents and videos without accidents. The categorical loss for T-frame videos is specifically expressed as follows:

[0092]

[0093] Where α is the class label of the video. When α = 1, it means there is an accident in the video; when α = 0, it means there is no accident in the video. p represents the probability of an accident in the video.

[0094] The present invention also uses a frame-level loss function, which aims to enable the model to predict accidents in the video as early as possible. It is specifically expressed as follows:

[0095]

[0096] Where τ(<T) represents the frame number at which the accident starts in the video, t represents the current frame number during the training process, and f is the frame rate of the dashcam video.

[0097] To enable the model to better learn driver eye movement information, the present invention designs a spatial adaptive loss and a channel adaptive loss, which are specifically expressed as follows:

[0098]

[0099] Where respectively represent the similarity features of the dashcam image and the similarity features of the driver eye movement heatmap output by the SAM module. Where i represents the scale of the spatial adaptive module, i = 0, 1, 2, 3, and cosin_sim() represents the cosine similarity; respectively represent the similarity features of the dashcam image and the similarity features of the driver eye movement heatmap output by the CAM module. Where j represents the scale of the spatial adaptive module, j = 0, 1, and cosin_sim() represents the cosine similarity.

[0100] L = L ce + 0.1L time + 0.001L SAM + 0.001L CAM (6);

[0101] (3) Inference stage. In the inference stage, the present invention no longer uses Map as the input of the model, but relies on the Map information adaptively learned in the training stage to act on the model's inference in DCI. In order to solve the problem that the category label corresponding to the early timestamp feature information may affect the result of the current timestamp feature, the present invention stipulates that when the category label corresponding to each timestamp feature information changes, recursive historical feature initialization should be performed to eliminate the impact of different results of early timestamps on the inference of the current timestamp feature.

[0102] (4) Timing modeling stage. Figure 3 As shown, this embodiment uses the MSTFormer network and uses a method similar to RNN to capture long-term information. First, the DCI features of the residual blocks of different scales output after Map adaptive learning are patch-embedded:

[0103]

[0104] The concatenated input features are divided into patches of fixed size. Considering the minimum size feature of the residual block output, the patch size is set to 7*7. Each patch has a dimension of 7*7*channel. Different residual blocks have different number of feature channels. Therefore, a fully connected layer is applied to adjust the patch dimension to 256 for subsequent processing.

[0105] The features of the same latitude obtained by splitting features of different scales are spliced ​​together.

[0106]

[0107] Get multi-scale multi-view features O∈R B×85×256 , at this time, the feature has a total of 85 tokens, and the dimension of each token is 256.

[0108] For the extraction of temporal information, the invention defines a set of grid-shaped learnable parameters as the historical features of the initial first time frame, that is, the initial historical time series feature map. Specifically, the initial historical time series feature map is first input into the TemporalSelfAttention multi-head attention layer, then performs a ResNet-like addition operation, and finally inputs into the LayerNorm layer again.

[0109] Q cur =LayerNorm(TemporalSelfAttention(Q his )+Q his ) (9);

[0110] Among them, the attention calculation of each head in the TemporalSelfAttention multi-head attention layer is performed independently, and each head has its own query, key, and value. Then, the outputs of multiple heads are merged through linear transformation and concatenation operations to form the final multi-head attention output.

[0111] TemporalSelfAttention(Q his )=Concat(temphead1(Q his ),…,temphead L (Q his ))W O (10);

[0112] In each attention head, the input initial historical time series feature map undergoes three linear transformations to produce three different representations: query, key, and value. The purpose of these three transformations is to map the input to different representation spaces, so that each head can focus on different information. For each head, the dot product between the query and the key is calculated and then normalized to obtain the attention weights. These weights are used to weight the sum values ​​to obtain the attention output for each position.

[0113]

[0114] Next, the historical time features and multi-view multi-scale features obtained by the temporal self-attention mechanism are input into the spatial cross-attention mechanism for feature extraction. Specifically, the historical time features and multi-view multi-scale features are first input into the Multi-Head Attention layer, then added in a similar ResNet-like operation, and finally input into the Layer Norm layer again.

[0115] Q cur =LayerNorm(SpacialCrossAttention(Q cur ,O)+Q cur ) (12);

[0116] Similarly, the attention calculation of each head of the SpacialCrossAttention multi-head attention layer is performed independently, and each head has its own query, key, and value. Then, the outputs of multiple heads are merged through linear transformation and concatenation operations to form the final multi-head attention output.

[0117] SpacialCrossAttention(Q cur ,O)=Concat(spacehead1(Q cur ,O),…,spacehead h (Qcur ,O))W O (13);

[0118] Similarly, in each attention head, unlike the temporal self-attention head, the spatial cross-attention head key and value are obtained by linear transformation of historical temporal features, and the query is obtained by linear transformation of multi-view multi-scale features. The purpose of these three transformations is also to map the input to different representation spaces, so that each head can focus on different information. For each head, the dot product between the query and the key is calculated and then normalized to obtain the attention weight. These weights are used to weight the sum value to obtain the attention output at each position.

[0119]

[0120] Similarly, the historical time patch features obtained by the spatial cross attention mechanism are input into the feedforward neural network, and a similar ResNet addition operation is performed, and finally sent to the Layer Norm layer

[0121] Q cur =LayerNorm(FeedForward(Q cur )+Q cur ) (15);

[0122] Among them, the Feed Forward layer of the feedforward neural network is composed of a two-layer fully connected layer. The first layer uses the Relu function as the activation function, and the second layer does not use the activation function. The corresponding formula is as follows.

[0123] FeedForward(Q cur )=max(0,Q cur W1+b1)W2+b2 (16).

[0124] This embodiment proposes a novel traffic accident prediction method, the highlight of which is to use the driver's gaze information as a supervisory auxiliary means for training traffic accident prediction models. The present invention proposes a new model called Eye-SCAN, which uses the driver's gaze information to guide the feature extraction of dashcam video data and maps through a backbone network, aiming to achieve early prediction of traffic accidents. Eye SCAN includes two key sub-modules: a spatial adaptive module (SAM) and a channel adaptive module (CAM), which are used to extract spatial semantic information and channel semantic information from the driver's line of sight, respectively. In addition, a new recursive transmission strategy of temporal information is introduced to mitigate the impact of different early results in the reasoning process on model performance. Through experiments on two benchmark datasets, our model outperforms the current state-of-the-art models in terms of performance, and the contribution of each component is analyzed in detail. These results provide an effective solution to improve the safety of smart cars.

[0125] Embodiment 2

[0126] This embodiment provides a spatial channel adaptive accident prediction system based on eye movement attention guidance, including:

[0127] According to some embodiments, a second solution of the present invention provides a spatial channel adaptive accident prediction system based on eye movement attention guidance, which adopts the following technical solutions:

[0128] The spatial channel adaptive accident prediction system based on eye movement attention guidance includes:

[0129] The spatial feature extraction module is configured to obtain a driving recorder image, perform feature extraction using a trained backbone network, and obtain a driving recorder spatial channel feature map;

[0130] The backbone network is trained using the eye movement attention heat map and the dashcam image respectively during the training phase, and the feature cosine similarity obtained from the processing of the eye movement attention heat map and the dashcam image guides the training process using the dashcam image to obtain a trained backbone network;

[0131] The patch embedding module is configured to generate patches of different scales using the residual block outputs of different scales of the backbone network, and to concatenate the patches of different scales to obtain multi-scale multi-view features;

[0132] A temporal feature extraction module is configured to extract temporal features based on a customized initial historical temporal feature map through a temporal self-attention module, a spatial cross-attention module, and a feed-forward neural network to obtain a current temporal feature map, wherein the spatial cross-attention module uses multi-scale multi-view features as input to extract spatial information, and uses the current temporal feature map as a historical temporal feature map for the next timestamp for recursive extraction;

[0133] The accident prediction module is configured to fuse and classify the current time series feature map with the driving recorder spatial channel feature map to obtain an accident prediction result.

[0134] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0135] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0136] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0137] Embodiment 3

[0138] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the spatial channel adaptive accident prediction method based on eye movement attention guidance as described in the first embodiment above are implemented.

[0139] Embodiment 4

[0140] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the spatial channel adaptive accident prediction method based on eye movement attention guidance as described in the first embodiment are implemented.

[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0146] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A spatial channel adaptive accident prediction method based on eye movement attention guidance, characterized in that: include: Obtain the dashcam image, use the trained backbone network to extract features, and obtain the dashcam spatial channel feature map; The residual blocks of different scales of the backbone network are used to generate patches of different scales, and the patches of different scales are spliced ​​to obtain multi-scale multi-view features; Based on the customized initial historical time series feature map, the time features are extracted through the time self-attention module, the spatial cross-attention module and the feedforward neural network to obtain the current time series feature map. The spatial cross-attention module takes the multi-scale multi-view features as input to extract spatial information, and uses the current time series feature map as the historical time series feature map of the next timestamp for recursive extraction. The current time series feature map is fused and classified with the driving recorder spatial channel feature map to obtain the accident prediction result; Among them, in the training stage, the backbone network is trained using the eye movement attention heat map and the dashcam image respectively, and the feature cosine similarity obtained based on the processing of the eye movement attention heat map and the dashcam image guides the process of training using the dashcam image to obtain a trained backbone network, specifically: Based on the eye movement attention heat map and the dashcam image, the spatial adaptive module is used to learn and output the driver's eye movement spatial adaptive semantic information, and the residual block is used to extract features from the output of the spatial adaptive module to obtain the attention space feature map and the dashcam attention space feature map; Repeat the above process three times to output the attention space feature maps of different scales and the attention space feature map of the driving recorder; Based on the attention space feature map outputted last time and the attention space feature map of the driving recorder, the channel adaptation module is used to learn and output the driver's eye movement channel adaptation semantic information, and the residual block is used to extract features from the output of the channel adaptation module; Repeat the above process once more, and output the attention channel feature maps of different scales and the attention channel feature map of the driving recorder; The attention channel feature map and the driving recorder attention channel feature map are processed by the fully connected layer to obtain the eye movement attention heat map feature map and the driving recorder attention feature map; The sum of the category cross entropy loss, frame-level loss, spatial adaptive loss, and channel adaptive loss is used as the training loss of the backbone network until the model converges to obtain a trained backbone network. The spatial adaptive loss is specifically expressed as follows: in, They represent the similarity features of the dashcam image and the driver's eye movement heat map output by the spatial adaptive module, respectively. Where i represents the scale of the spatial adaptive module, i=0,1,2,3, and cosin_sim() represents cosine similarity; The channel adaptive loss is specifically expressed as follows: in, They represent the similarity features of the dashcam image and the driver's eye movement heat map output by the channel adaptive module respectively, where j represents the scale of the spatial adaptive module, j=0,1, and cosin_sim() represents cosine similarity.

2. The spatial channel adaptive accident prediction method based on eye movement attention guidance according to claim 1, characterized in that: Based on the eye movement attention heat map and the driving recorder image, the spatial adaptive modules are used to learn and output the driver's eye movement spatial adaptive semantic information, specifically: Perform patch embedding on the eye movement attention heat map features and dashcam images of size B*C*H*W respectively, and generate features of size B*(hw)*(Cs1s1), where B, C, H, W represent the batch size, number of channels, height and width of the features, s1 and s2 represent the size of the divided tokens, and h and w represent the number of tokens in the H and W dimensions; The features obtained from the patch embedding are average pooled separately to obtain features with dimension B*(hw)*1; The second dimension of the feature represents the weight ratio of each token in the entire feature; The dimensions after pooling are adjusted to B*(hw) through squeezing operations, and the feature cosine similarity is calculated for the two feature vectors output after squeezing operations. At the same time, the features output after pooling through the squeeze operation are activated by the tanh activation function, and then multiplied by the unpooled features of dimension B*(hw)*(Cs1s1) to achieve adaptive weighting of each token; The features adjusted by adaptive weights are reshaped back to the original scale of B*C*H*W to obtain the output of the spatial adaptive module.

3. The spatial channel adaptive accident prediction method based on eye movement attention guidance according to claim 1, characterized in that: The channel adaptation modules are used to learn and output the driver's eye movement channel adaptive semantic information respectively, specifically: The last output attention space feature map with dimensions of B*C*H*W and the attention space feature map of the driving recorder are subjected to average pooling and squeezing respectively to obtain features with dimensions of B*C; Use linear layers to reduce the dimension of features with dimension B*C to obtain features with dimension B*(C / 2); Compare the cosine similarity of the output B*(C / 2) features with dimension reduction between the two; At the same time, the features after dimension reduction by the linear layer are processed by softmax and linear layer respectively to obtain features of size B*C; After being activated by the tanh activation function, the feature is multiplied by the attention space feature map of the last output with the size of B*C*H*W and the attention space feature map of the driving recorder that has not been processed to obtain the output of the channel adaptation module.

4. The spatial channel adaptive accident prediction method based on eye movement attention guidance according to claim 1, characterized in that: Based on the customized initial historical time series feature graph, the time features are extracted through the time self-attention module, the space cross-attention module and the feedforward neural network to obtain the current time series feature graph, which is specifically: Based on the customized initial historical time series feature graph, feature extraction is performed through the time self-attention module to obtain historical time features; Input the historical time features and multi-scale multi-view features into the spatial cross attention module for feature extraction to obtain the historical time patch features; The historical time patch features are extracted using a feedforward neural network and then normalized to obtain the current time series feature map; The obtained current time series feature graph is used as the historical time series feature graph of the next timestamp to recursively perform time series extraction.

5. Spatial channel adaptive accident prediction system based on eye movement attention guidance, characterized in that: include: The spatial feature extraction module is configured to obtain a driving recorder image, perform feature extraction using a trained backbone network, and obtain a driving recorder spatial channel feature map; Among them, in the training stage, the backbone network is trained using the eye movement attention heat map and the dashcam image respectively, and the feature cosine similarity obtained based on the processing of the eye movement attention heat map and the dashcam image guides the process of training using the dashcam image to obtain a trained backbone network, specifically: Based on the eye movement attention heat map and the dashcam image, the spatial adaptive module is used to learn and output the driver's eye movement spatial adaptive semantic information, and the residual block is used to extract features from the output of the spatial adaptive module to obtain the attention space feature map and the dashcam attention space feature map; Repeat the above process three times to output the attention space feature maps of different scales and the attention space feature map of the driving recorder; Based on the attention space feature map outputted last time and the attention space feature map of the driving recorder, the channel adaptation module is used to learn and output the driver's eye movement channel adaptation semantic information, and the residual block is used to extract features from the output of the channel adaptation module; Repeat the above process once more, and output the attention channel feature maps of different scales and the attention channel feature map of the driving recorder; The attention channel feature map and the driving recorder attention channel feature map are processed by the fully connected layer to obtain the eye movement attention heat map feature map and the driving recorder attention feature map; The sum of the category cross entropy loss, frame-level loss, spatial adaptive loss, and channel adaptive loss is used as the training loss of the backbone network until the model converges to obtain a trained backbone network. The spatial adaptive loss is specifically expressed as follows: in, They represent the similarity features of the dashcam image and the driver's eye movement heat map output by the spatial adaptive module, respectively, where i represents the scale of the spatial adaptive module, i=0,1,2,3, and cosin_sim() represents cosine similarity; The channel adaptive loss is specifically expressed as follows: in, They represent the similarity features of the dashcam image and the driver's eye movement heat map output by the channel adaptive module, respectively. j represents the scale of the spatial adaptive module, j = 0, 1, and cosin_sim() represents the cosine similarity. The patch embedding module is configured to generate patches of different scales using the residual block outputs of different scales of the backbone network, and concatenate the patches of different scales to obtain multi-scale multi-view features; A temporal feature extraction module is configured to extract temporal features based on a customized initial historical temporal feature map through a temporal self-attention module, a spatial cross-attention module, and a feed-forward neural network to obtain a current temporal feature map, wherein the spatial cross-attention module uses multi-scale multi-view features as input to extract spatial information, and uses the current temporal feature map as a historical temporal feature map for the next timestamp for recursive extraction; The accident prediction module is configured to fuse and classify the current time series feature map with the driving recorder spatial channel feature map to obtain an accident prediction result.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the spatial channel adaptive accident prediction method based on eye movement attention guidance as described in any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the spatial channel adaptive accident prediction method based on eye movement attention guidance as described in any one of claims 1-4 are implemented.