Traffic accident prediction method and device, electronic equipment and storage medium

Abstract

The application discloses a traffic accident prediction method, comprising: acquiring a vehicle-mounted video and preprocessing to obtain continuous image frames, extracting time features and space features from the continuous image frames, and performing mid-term fusion on the time-space features; performing position weighting on the object-level features based on a graph structure; performing spatial attention on the weighted object-level features; splicing the object-level features and full-frame features, and performing frame-level aggregation and time period-level aggregation to sufficiently learn traffic scene context information; performing space-time relationship reasoning on the obtained double-layer aggregated information to obtain a hidden state representation of a current time step; inputting the hidden state representation of the current time step into a full connection network to obtain an accident score, and giving a warning signal when the accident score exceeds a predetermined threshold. The method can successfully predict the occurrence of a traffic accident 2-4 seconds in advance, thereby giving a driving system or a driver sufficient reaction time, and thus greatly reducing the possibility of traffic accidents.

Description

Technical Field

[0001] This invention relates to the field of transportation, and specifically to a method, apparatus, electronic device, and storage medium for predicting traffic accidents. Background Technology

[0002] Traffic accidents cause incalculable losses to people's lives and property, making safe driving a pressing need for the public. One of the biggest challenges to safe driving is accurately predicting accidents so that necessary measures can be taken to prevent them from occurring. Accurate accident prediction provides drivers and autonomous vehicles with sufficient reaction time to take evasive action, thus enhancing the safe driving capabilities of both human and autonomous vehicles.

[0003] Most vehicles are now equipped with dashcams, which can record video images during the vehicle's journey. In-vehicle video can provide a stable source of predictive data for traffic accident prediction methods. If it is possible to predict potential traffic accidents ahead of the vehicle in real time based on the video captured by the dashcam, drivers will have enough time to avoid traffic accidents while driving.

[0004] Existing methods for predicting traffic accidents mainly include the following: Reference [Chan FH, Chen YT, Xiang Y, et al. Anticipating accidents in dashcam videos[C]. Asian Conference on Computer Vision. Springer, Cham, 2016:136-153] uses recurrent neural networks for traffic accident prediction; Reference [Zeng KH, Chou SH, Chan FH, et al. Agent-centric risk assessment: Accident anticipation and risky region localization[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:2222-2230] proposes using a soft attention recurrent neural network to model the nonlinear interactions between candidate targets and between candidate targets and risk regions; Reference [Suzuki T, Kataoka H, ​​Aoki Y, et al. Anticipating traffic accidents with adaptive loss and large-scale incident db[C]. Proceedings of the IEEE conference on computer vision and pattern [recognition. 2018:3521-3529] A quasi-recurrent neural network containing temporal convolution is selected to model the traffic accident prediction task; [Bao W, Yu Q, Kong Y. Uncertainty-based traffic accident anticipation with spatio-temporalrelational learning[C]. Proceedings of the 28th ACM International Conference on Multimedia. 2020:2682-2690] Graph convolutional recurrent neural network and Bayesian neural network are used to predict traffic accidents; [Yu SY, Malawade AV, Muthirayan D, et al.][Scene-graph augmented data-driven risk assessment of autonomous vehicle decisions[J].IEEE Transactions on Intelligent Transportation Systems, 2021, 23(7):7941-7951] proposes using scene graphs as intermediate representations to build models for predicting the subjective risks of driving operations; [Malawade AV, Yu SY, Hsu B, et al. Spatiotemporal scene-graph embedding for autonomous vehicle collision prediction[J].IEEE Internet of Things Journal, 2022, 9(12):9379-9388] proposes a spatiotemporal scene graph embedding method, using graph neural networks and long short-term memory layers to predict future collisions through visual scene perception. However, these models all consider irrelevant objects in video frames and irrelevant frames in vehicle video in their predictions, which reduces the accuracy of traffic accident prediction. Summary of the Invention

[0005] This invention addresses the problem of numerous irrelevant traffic objects and frames in vehicle-mounted video frames being present, hindering traffic accident prediction. It focuses on reducing irrelevant information and enhancing key information within traffic scenes. By analyzing the properties of irrelevant information in traffic scenes, a deep learning method is proposed to reduce the influence of irrelevant objects and frames in vehicle-mounted videos, and applied to traffic accident prediction using vehicle-mounted videos. The invention uses RGB images to extract optical flow information to weaken stationary targets in the video. Mid-term fusion via a two-stream network integrates spatiotemporal information, enabling interaction between these elements. Furthermore, a graph-based positional weighting process is applied to nearby targets in the traffic scene, giving them greater weight. Finally, a spatial attention module, through neural network learning, automatically guides the traffic accident prediction model to focus on targets more likely to cause accidents, thus achieving high accuracy in traffic accident prediction. Finally, to improve the scene understanding capability of the traffic prediction model, a combination of gated recurrent units, frame-level aggregation, and time-segment aggregation is proposed. Deep learning is used to perform temporal attention operations on video frames and video time segments, enabling accident reasoning to fully learn the traffic scene context information in the in-vehicle video. This traffic scene context information reduces irrelevant target information in the traffic scene and irrelevant video frame information in the in-vehicle video, improving the accuracy of traffic accident prediction and greatly enhancing the practicality of the traffic accident prediction method of this invention.

[0006] This invention is achieved through the following technical solution:

[0007] Firstly, a method for predicting traffic accidents includes the following steps:

[0008] Step S1: Acquire the vehicle-mounted video and perform preprocessing to obtain continuous image frames;

[0009] Step S2: Extract temporal and spatial features from consecutive image frames;

[0010] Step S3: Perform mid-term fusion of the extracted temporal and spatial features;

[0011] Step S4: Perform positional weighting on object-level features based on graph structure;

[0012] Step S5: After spatial attention is applied to the position-weighted object-level features, the object-level features are concatenated with the full-frame features and then frame-level and time-level aggregations are performed to fully learn the traffic scene context information in the vehicle video. Spatiotemporal relationship reasoning is performed on the obtained two-layer aggregation information to obtain the hidden state representation of the current time step.

[0013] Step S6: Input the hidden state representation of the current time step obtained by aggregation into the fully connected network to obtain the final accident score;

[0014] Step S7: For the obtained accident score, the system issues an early warning signal when the accident score exceeds a predetermined threshold.

[0015] Secondly, a traffic accident prediction device includes the following modules:

[0016] The acquisition and preprocessing module is used to acquire vehicle-mounted video and perform preprocessing to obtain continuous image frames.

[0017] The feature extraction module is used to extract temporal and spatial features from consecutive image frames;

[0018] The spatiotemporal fusion module is used to perform intermediate-term fusion of extracted temporal and spatial features;

[0019] The position weighting module is used to perform position weighting on object features based on a graph structure.

[0020] The two-layer aggregation module is used to perform spatial attention on the position-weighted object-level features, then concatenate the object features with the full-frame features and perform frame-level and time-level aggregation to fully learn the traffic scene context information in the vehicle video. The obtained two-layer aggregation information is used to perform spatiotemporal relationship reasoning to obtain the hidden state representation of the current time step.

[0021] The incident scoring module is used to input the aggregated hidden state representation of the current time step into the fully connected network to obtain the final incident score.

[0022] The accident early warning module is used to issue an early warning signal when the obtained accident score exceeds a predetermined threshold.

[0023] Thirdly, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the traffic accident prediction method.

[0024] Fourthly, a storage medium storing a computer program that, when executed by a processor, implements the steps of the traffic accident prediction method.

[0025] Compared with the prior art, the present invention has the following beneficial technical effects:

[0026] This invention provides a traffic accident prediction method, device, electronic device, and storage medium. It involves segmenting in-vehicle video into frames, extracting optical flow image sequences from the video frame sequences, and extracting spatial and temporal features from the video frames and optical flow images, respectively. Using graph-based positional weighting, the influence of closely spaced traffic targets is highlighted. Then, through a spatial attention module and neural network learning, the traffic accident prediction model automatically focuses on targets more likely to cause traffic accidents, thus achieving high accuracy in traffic accident prediction. Finally, to improve the scene understanding capability of the traffic prediction model, a combination of gated recurrent units, frame-level aggregation, and time-segment aggregation is proposed. Deep learning is used to perform temporal attention operations on video frames and video time segments, enabling accident reasoning to fully learn the traffic scene context information in the in-vehicle video. By reducing the dimensionality of the obtained context information, a traffic accident prediction score can be obtained. When the traffic accident prediction score exceeds a threshold, a danger warning can be issued to the driver, giving them time to take emergency evasive action and reducing the occurrence of traffic accidents. On the DAD dataset, the average accuracy of this invention is 21.7% higher than existing methods, and the average accident prediction time is 0.48 seconds higher. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0028] Figure 1 This is a flowchart of a traffic accident prediction and hazard warning method according to the present invention;

[0029] Figure 2 This is a flowchart illustrating a traffic accident prediction method according to the present invention.

[0030] Figure 3 This is a schematic diagram of the optical flow extraction method for adjacent frame images according to the present invention;

[0031] Figure 4 This is a probability curve of an accident occurring in scenario one (with an accident) in a specific embodiment of the present invention;

[0032] Figure 5 This is a probability curve of an accident occurring in scenario two (with an accident) in a specific embodiment of the present invention;

[0033] Figure 6 This is a probability curve of an accident occurring in scenario three (with an accident) in a specific embodiment of the present invention;

[0034] Figure 7 This is a probability curve of an accident occurring in scenario four (no accident) in a specific embodiment of the present invention;

[0035] Figure 8 This is a schematic diagram of the structure of a traffic accident prediction device according to the present invention;

[0036] Figure 9 This is a schematic diagram of the structure of an electronic device according to the present invention. Detailed Implementation

[0037] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0038] This invention provides a method for predicting traffic accidents, such as... Figure 1 , Figure 2 As shown, it includes the following steps:

[0039] Step S1: Acquire the vehicle-mounted video and perform preprocessing to obtain continuous image frames;

[0040] Step S2: Extract temporal and spatial features from consecutive image frames;

[0041] Step S3: Perform mid-term fusion of the extracted temporal and spatial features;

[0042] Step S4: Perform positional weighting on object-level features based on graph structure;

[0043] Step S5: After spatial attention is applied to the position-weighted object-level features, the object-level features are concatenated with the full-frame features and then frame-level and time-level aggregations are performed to fully learn the traffic scene context information in the vehicle video. Spatiotemporal relationship reasoning is performed on the obtained two-layer aggregation information to obtain the hidden state representation of the current time step.

[0044] Step S6: Input the hidden state representation of the current time step obtained by aggregation into the fully connected network to obtain the final accident score;

[0045] Step S7: For the obtained accident score, the system issues an early warning signal when the accident score exceeds a predetermined threshold.

[0046] The traffic accident prediction method described in this embodiment of the invention is executed by an electronic device. The electronic device can be any type of electronic device; for example, the electronic device can be, but is not limited to, at least one of the following: a server, a computer, a tablet computer, or other electronic devices.

[0047] Based on the above method, step S1 specifically includes the following steps:

[0048] Step S11: Obtain the in-vehicle video captured by the dashcam, perform frame segmentation to obtain T consecutive image frames, and scale the image resolution to 1080*720.

[0049] Based on the above method, step S2 includes the following steps:

[0050] Step S21: Perform traffic target detection on the obtained T consecutive image frames. The detected traffic targets include people, bicycles, motorcycles, cars and buses, and obtain bounding boxes of N traffic targets in each image frame.

[0051] This embodiment uses a detection model based on Faster R-CNN for traffic target detection. The traffic target detection model is trained on the COCO dataset, and its output is the bounding boxes of traffic targets in the image, with each bounding box represented by a 4-dimensional vector. The image size in the video is uniformly scaled to 1024*1024 before being input into Faster R-CNN. The number of traffic target detection boxes is N. If there are fewer than N traffic targets in the image, a 4-dimensional zero vector is used; if there are more than N traffic targets, only the N bounding boxes with the highest detection scores are retained. In the experiment, N = 19 was set.

[0052] It should be understood that any object detection algorithm can be used for traffic object detection, such as SSD, YOLO, etc. This embodiment takes into account that Faster R-CNN has higher accuracy and can achieve good detection of small objects, so it is chosen as a preferred implementation method.

[0053] Step S22: Extract optical flow from the obtained T consecutive image frames.

[0054] The optical flow extraction operation is calculated using the following steps: For video frame images at consecutive time points T, the optical flow of each frame is calculated relative to the previous frame, resulting in T optical flow images, for a total of T-1 optical flow images; this embodiment preferably uses the FlowNet2 algorithm for optical flow calculation of adjacent frames; the two-dimensional optical flow vector of the j-th pixel in the optical flow image is: I j =(u j ,v j ), u j v j These are the vertical and horizontal components of the optical flow vector, respectively; for example... Figure 3 As shown.

[0055] Step S23: Insert blank frames into the obtained T-1 optical flow images;

[0056] The blank frame insertion operation involves inserting a white image with a resolution of 1080*720 at the beginning of the optical flow image. The RGB value of the white image is (255,255,255), thereby obtaining a total of T frames of optical flow image sequence.

[0057] Step S24: Extract the corresponding spatial features from consecutive image frames of sequence length T;

[0058] The spatial feature extraction operation is as follows: An RGB image is input, and a convolutional neural network is used to extract the spatial visual representation of the RGB image. The RGB image is a red-green-blue color pattern image, and the convolutional neural network used is a VGG16 network. The feature extraction range includes N bounding boxes in each frame and the entire frame. The spatial features of each frame's RGB image are divided into object-level spatial features and full-frame spatial features, with feature dimensions of D*N and D*1, respectively. In the experiment, D = 4096 was set.

[0059] It should be noted that the VGG16 network is only a preferred convolutional neural network in this embodiment. In other embodiments, any other feature extraction network can be selected for the extraction of temporal and spatial features.

[0060] Step S25: Extract the corresponding temporal features from the optical flow image with a sequence length of T;

[0061] The operation of extracting temporal features involves inputting an optical flow image and using a convolutional neural network to extract temporal visual representations. The optical flow image is a red-green-blue color pattern image, and the convolutional neural network used is a VGG16 network. The feature extraction range is the N bounding boxes in each frame of the image and the entire frame. The temporal features of each frame of the optical flow image are divided into object temporal features and full-time features, with feature dimensions of D*N and D*1, respectively.

[0062] Based on the above method, step S3 includes the following steps:

[0063] Step S31: Add the extracted spatial and temporal features; specifically, add the object-level spatial features to the full-frame spatial features, and the object-level temporal features to the full-frame temporal features, respectively, to obtain the mid-term fusion object features of the t-th frame image. Mid-term fusion of full-frame features The feature dimension of the mid-term fusion object features is D*N, and the feature dimension of the mid-term fusion full-frame features is D*1.

[0064] Step S4 includes the following steps:

[0065] Step S41: Based on the target bounding box obtained in step S21, the center point coordinates C of each bounding box in the t-th frame image can be obtained. t C t Given a set of two-dimensional vectors, the Euclidean distance between target i and target j in the t-th frame of the video sequence is obtained using the following formula.

[0066]

[0067] Step S42: Obtain the sum of Euclidean distances between target i in frame t and all other traffic targets in the image according to the following formula:

[0068]

[0069] Step S43: Obtain the position weight of target i in the t-th frame image according to the following formula.

[0070]

[0071] Where b is a constant. In the experiment, b = 1.675 was set.

[0072] Step S44: Based on the position weight of target i in the t-th frame image obtained in step S43. The mid-term fusion object features of target i in frame t obtained in step S31 Multiplying them together yields the position-weighted feature of target i in frame t.

[0073]

[0074] The dot (·) represents element-wise multiplication.

[0075] Step S45: The mid-frame fusion full-frame features of the t-th frame image obtained in step S31 and the position-weighted features of the t-th frame image obtained in step S44 are respectively passed through a fully connected layer to obtain corresponding low-dimensional features: namely, the low-dimensional position-weighted object-level features of the t-th frame image. Low-dimensional frame-level features of the t-th frame image in The feature dimension is d*N. The feature dimension is d*1. In the experiment, d=512 was set.

[0076] Based on the above method, step S5 includes the following steps:

[0077] Step S51: Obtain the spatial attention weight α of the t-th frame image. t :

[0078]

[0079] in Here, is the SoftMax activation function, tanh is the tanh activation function, and w sa w b w c w d h′ is a learnable parameter. t-1 This represents the hidden state obtained in frame t-1. For the low-dimensional location-weighted object-level features of the t-th frame image, where

[0080] Step S52: Obtain spatial attention target-level features of the t-th frame image.

[0081]

[0082] Step S53: Spatial attention target-level features of the t-th frame image. Low-dimensional frame-level features of the t-th frame image By concatenating the phases, the traffic scene features X of the t-th frame image are obtained. t :

[0083]

[0084] Where; represents splicing;

[0085] Step S54: Obtain the hidden state representation H for a single time period. t-i :

[0086] H t-i =[h t-1 ,...,h t-i ], i∈(1,...,z)

[0087] Where h t-i Z is the hidden state representation obtained for frame ti, and Z is a constant. In the experiment, Z = 10 was set.

[0088] Step S55: In the frame-level aggregation layer, the temporal attention operation is applied to the hidden state representation of a single frame. The frame-level weighted hidden state representation is obtained through frame-level aggregation and is represented as follows:

[0089]

[0090] in For the SoftMax activation function, w ta For learnable parameters, tanh represents the hyperbolic tangent activation function;

[0091] Step S56: Obtain hidden state representations A for multiple time periods. t-Z :

[0092] A t-Z =[H′ t-1 ,H′ t-2 ,...,H′ t-Z ]

[0093] Step S57: In the time-level aggregation layer, apply the temporal attention operation to multiple time-level hidden state representations, and obtain the time-level hidden state representation h′ through time-level aggregation. t-1 This process can be represented as:

[0094]

[0095] in For the SoftMax activation function, w tsa For learnable parameters, tanh represents the hyperbolic tangent activation function;

[0096] Step S58: Use a gated loop unit to perform spatiotemporal relationship reasoning;

[0097] The gated loop unit has two gates: a reset gate and a reset gate. and Update Gate Used to retain the most relevant information about traffic accidents in the video sequence, the overall structure can be described by the following equation:

[0098]

[0099]

[0100]

[0101]

[0102] Where σ represents the sigmoid activation function, h′ t-1 X represents the time-level hidden state representation of frame t-1 obtained after two-layer aggregation. t This represents the traffic scene characteristics input at the current moment. Represents the element-wise multiplication operator, r t h represents the candidate hidden state at time step t. t Let represent the hidden state representation of the image at frame t, and tanh represent the hyperbolic tangent activation function. All of these are learnable parameters, and their dimensions are all d*d.

[0103] Based on the above method, step S6 includes the following steps:

[0104] The dimension of the hidden state representation of the t-th frame image obtained by S5 is reduced to 1 through a fully connected layer, thus obtaining the traffic accident score of the t-th frame image. This process can be expressed by the formula:

[0105]

[0106] FC stands for fully connected layer, which is used to reduce the feature dimension from d to 1. The fully connected layer contains 512 neurons, with an input dimension of 512 and an output dimension of 1.

[0107] Based on the above method, in step S7, when the traffic accident score exceeds a predetermined threshold, the system issues an alarm signal. In this embodiment, the predetermined threshold is preferably 0.5.

[0108] Loss function:

[0109] During experimental training, the goal of the model prediction is to make the traffic accident prediction results as close as possible to the actual traffic accident situation, achieving a balance between prediction time and prediction accuracy. In other words, the purpose of the loss function is to minimize the error between the two. Therefore, the equation for the model loss function can be expressed as:

[0110]

[0111] The first term applies to accident frames, and the second term applies to normal frames. Loss represents the loss function, a t Let y represent the traffic accident prediction score of the t-th frame image, y represent the number of frames in which the actual accident occurred, f represent the frame rate of the in-vehicle video, and T represent the total number of frames in the in-vehicle video. Specific implementation examples:

[0113] In this embodiment of the invention, the entire system structure adopts a multi-input single-output model. The method constructs the network model using the PyTorch deep learning framework and optimizes the parameters of the entire network using the ADMA optimizer, with a learning rate of 0.0001. Training and testing are performed on the DAD dataset.

[0114] The DAD dataset contains 620 accident video sequences and 1130 normal video sequences, totaling 175,000 frames. Two-thirds of the dataset was used as the training set and one-third as the test set. The dataset was trained for 80 epochs and compared with seven existing algorithms: DSA (from the paper Chan FH, Chen YT, Xiang Y, et al. Anticipating accidents in dashcam videos[C]. Asian Conference on Computer Vision. Springer, Cham, 2016:136-153), SP (from the paper Alahi A, Goel K, Ramanathan V, et al. Social lstm: Human trajectory prediction in crowded spaces[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:961-971), and L-RAI (from the paper Zeng KH, Chou SH, Chan FH, et al. Agent-centric risk assessment: Accident anticipation and risky region localization[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition). Recognition.2017:2222-2230), AdaLEA (from the paper Suzuki T, Kataoka H, ​​Aoki Y, et al. Anticipating traffic accidents with adaptive loss and large-scale incident db[C]. Proceedings of the IEEEconference on computer vision and pattern recognition. 2018:3521-3529), GCRNN (from the paper Bao W, Yu Q,Kong Y.Uncertainty-based traffic accident anticipation with spatial-temporal relational learning[C].The performance of the following algorithms was compared: Proceedings of the 28th ACM International Conference on Multimedia. 2020:2682-2690; FA (from the paper Fatima M, Khan MUK, Kyung C M. Global feature aggregation for accident anticipation[C]. 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021:2809-2816); and DSTA (from the paper Karim MM, Li Y, Qin R, et al. A dynamic spatial-temporalattention network for early anticipation of traffic accidents[J]. IEEE Transactions on Intelligent Transportation Systems, 2022). The comparison results are shown in Table 1.

[0115] Table 1: Performance Comparison with Existing Algorithms

[0116]

[0117] AP (%): Average accuracy;

[0118] mTTA (seconds): The time interval for prediction advance.

[0119] Furthermore, the method of this invention was compared with three algorithms: DSA, GCRNN, and DSTA, through simulation. The accident occurrence probability curves obtained from the simulation are shown below. Figure 4 , Figure 5 , Figure 6 , Figure 7 As shown, these correspond to scenarios one through four. Traffic accidents occurred in scenarios one, two, and three, with corresponding TTAs (Time-to-Accident) of 2.05 seconds, 1.6 seconds, and 3.8 seconds, respectively. No traffic accident occurred in scenario four. (From Table 1 and...) Figure 4-7 As can be seen, the traffic accident prediction method proposed in this invention has achieved better results than the current best method in all indicators on the DAD dataset. The average accuracy of the method proposed in this invention is 21.7% higher than that of the existing method, and the average accident prediction time is 0.48 seconds higher than that of the existing method.

[0120] The following describes a traffic accident prediction device provided by the present invention. The traffic accident prediction device described below can be referred to in correspondence with the traffic accident prediction method described above.

[0121] like Figure 8 As shown, a traffic accident prediction device includes the following modules:

[0122] The acquisition and preprocessing module 010 is used to acquire vehicle video and perform preprocessing to obtain continuous image frames.

[0123] Feature extraction module 020 is used to extract temporal and spatial features from consecutive image frames;

[0124] The spatiotemporal fusion module 030 is used to perform intermediate-term fusion of extracted temporal and spatial features;

[0125] Position weighting module 040 is used to perform position weighting on object features based on graph structure.

[0126] The dual-layer aggregation module 050 is used to perform spatial attention on the position-weighted object-level features, then concatenate the object features with the full-frame features and perform frame-level and time-level aggregation to fully learn the traffic scene context information in the vehicle video. It then performs spatiotemporal relationship reasoning on the obtained dual-layer aggregation information to obtain the hidden state representation of the current time step.

[0127] The incident scoring module 060 is used to input the hidden state representation of the current time step obtained by aggregation into the fully connected network to obtain the final incident score;

[0128] The accident early warning module 070 is used to issue an early warning signal when the obtained accident score exceeds a predetermined threshold.

[0129] like Figure 9As shown, a schematic diagram of the physical structure of an electronic device is illustrated. The electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the steps of the traffic accident prediction method described above, specifically including: acquiring vehicle-mounted video and preprocessing it to obtain continuous image frames; extracting temporal and spatial features from the continuous image frames; performing intermediate fusion on the extracted temporal and spatial features; performing positional weighting on object-level features based on graph structure; performing spatial attention on the position-weighted object-level features, concatenating the object-level features with the full-frame features, and performing frame-level and time-level aggregation to fully learn the traffic scene context information in the vehicle-mounted video; performing spatiotemporal relationship reasoning on the obtained two-layer aggregation information to obtain the hidden state representation of the current time step; inputting the aggregated hidden state representation of the current time step into a fully connected network to obtain the final accident score; and issuing a warning signal when the accident score exceeds a predetermined threshold.

[0130] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] In another aspect, embodiments of the present invention also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described traffic accident prediction method, specifically including:

[0132] The system acquires and preprocesses in-vehicle video to obtain continuous image frames. Temporal and spatial features are extracted from these frames. Intermediate fusion of the extracted temporal and spatial features is performed. Object-level features are weighted based on graph structure. Spatial attention is applied to the weighted object-level features, which are then concatenated with the full-frame features and aggregated at both the frame and time-segment levels to fully learn the traffic scene context information in the in-vehicle video. Spatiotemporal relationship reasoning is performed on the obtained two-layer aggregated information to obtain the hidden state representation for the current time step. This aggregated hidden state representation is input into a fully connected network to obtain the final accident score. For each accident score exceeding a predetermined threshold, the system issues a warning signal.

[0133] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0134] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several devices, several of these devices may be embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as identifiers.

[0135] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for predicting traffic accidents, characterized in that, Includes the following steps: Step S1: Acquire the vehicle-mounted video and perform preprocessing to obtain continuous image frames; Step S2: Extract temporal and spatial features from consecutive image frames; Step S3: Perform mid-term fusion of the extracted temporal and spatial features; Step S4: Perform positional weighting on object-level features based on graph structure; Step S5: After spatial attention is applied to the position-weighted object-level features, the object-level features are concatenated with the full-frame features and then frame-level and time-level aggregations are performed to fully learn the traffic scene context information in the vehicle video. Spatiotemporal relationship reasoning is performed on the obtained two-layer aggregation information to obtain the hidden state representation of the current time step. Step S6: Input the hidden state representation of the current time step obtained by aggregation into the fully connected network to obtain the final accident score; Step S7: For the obtained accident score, the system issues an early warning signal when the accident score exceeds a predetermined threshold. Step S1 includes the following steps: Acquire the in-vehicle video captured by the dashcam, perform frame segmentation on the in-vehicle video to obtain T consecutive image frames, and scale the image resolution to 1080*720. Step S2 includes the following steps: Step S21: Perform traffic target detection on the obtained T consecutive image frames. The detected traffic targets include people, bicycles, motorcycles, cars and buses, and obtain bounding boxes of N traffic targets in each image frame. Step S22: Extract optical flow from the obtained T consecutive image frames; The optical flow extraction operation is calculated using the following steps: For video frame images at consecutive times T, calculate the optical flow of each frame to the previous frame to obtain the optical flow images corresponding to T images, for a total of T-1 optical flow images; Step S23: Insert blank frames into the obtained T-1 optical flow images; The blank frame insertion operation is as follows: a white image with a resolution of 1080*720 is inserted at the beginning of the optical flow image. The RGB value of the white image is (255,255,255), thereby obtaining a total of T frames of optical flow image sequence. Step S24: Extract the corresponding spatial features from consecutive image frames of sequence length T; The spatial feature extraction operation is as follows: input an RGB image, use a convolutional neural network to extract the spatial visual representation of the RGB image, where the RGB image is a red-green-blue color mode image, the convolutional neural network used is a VGG16 network, the feature extraction range is N bounding boxes in each frame image and the whole frame image, the spatial features of each frame RGB image are divided into object-level spatial features and whole-frame spatial features, and their feature dimensions are D*N and D*1 respectively; Step S25: Extract the corresponding temporal features from the optical flow image with a sequence length of T; The operation of extracting temporal features is as follows: input optical flow image, use convolutional neural network to extract temporal visual representation respectively, wherein the optical flow image is a red-green-blue color mode image, the convolutional neural network used is VGG16 network, the feature extraction range is N bounding boxes in each frame image and the whole frame image, the temporal features of each frame optical flow image are divided into object-level temporal features and whole-frame temporal features, and their feature dimensions are D*N and D*1 respectively; Step S3 includes the following steps: Step S31: Add the extracted spatial and temporal features; specifically, add the object-level spatial features to the full-frame spatial features, and the object-level temporal features to the full-frame temporal features, respectively, to obtain the mid-term fusion object features of the t-th frame image. Mid-term fusion of full-frame features The feature dimension of the mid-term fusion object features is D*N, and the feature dimension of the mid-term fusion full-frame features is D*1. Step S4 includes the following steps: Step S41: Obtain the center point coordinates C of each bounding box in the t-th frame image based on the target bounding box obtained in step S21. t C t Given a set of two-dimensional vectors, the Euclidean distance between target i and target j in the t-th frame of the video sequence is obtained using the following formula. Step S42: Obtain the sum of Euclidean distances between target i in frame t and all other traffic targets in the image according to the following formula. Step S43: Obtain the position weight of target i in the t-th frame image according to the following formula. Where b is a constant; Step S44: Based on the position weight of target i in the t-th frame image obtained in step S43. The mid-term fusion object features of target i in frame t obtained in step S31 Multiply to obtain the position-weighted feature of target i in frame t. Where · represents element-wise multiplication; Step S45: The mid-frame fusion full-frame features of the t-th frame image obtained in step S31 and the position-weighted features of the t-th frame image obtained in step S44 are respectively passed through a fully connected layer to obtain corresponding low-dimensional features: namely, the low-dimensional position-weighted object-level features of the t-th frame image. Low-dimensional frame-level features of the t-th frame image in The feature dimension is d*N. The feature dimension is d*1; Step S5 includes the following steps: Step S51: Obtain the spatial attention weight α of the t-th frame image. t : in Here, is the SoftMax activation function, tanh is the tanh activation function, and w sa w b w c w d h′ is a learnable parameter. t-1 This represents the hidden state obtained in frame t-1. For the low-dimensional location-weighted object-level features of the t-th frame image, where Step S52: Obtain spatial attention target-level features of the t-th frame image. Step S53: Spatial attention target-level features of the t-th frame image. Low-dimensional frame-level features of the t-th frame image By concatenating the phases, the traffic scene features X of the t-th frame image are obtained. t : Where; represents splicing; Step S54: Obtain the hidden state representation H for a single time period. t-i : H t-i =[h t-1 ,...,h t-i ],i∈(1,...,z) Where h t-i Z is the hidden state representation obtained for frame ti, where Z is a constant; Step S55: In the frame-level aggregation layer, the temporal attention operation is applied to the hidden state representation of a single frame. The frame-level weighted hidden state representation is obtained through frame-level aggregation and is represented as follows: in For the SoftMax activation function, w ta For learnable parameters, tanh represents the hyperbolic tangent activation function; Step S56: Obtain hidden state representations A for multiple time periods t-Z : A t-Z =[H′ t-1 ,H′ t-2 ,…,H′ t-Z ] Step S57: In the time-level aggregation layer, apply the temporal attention operation to multiple time-level hidden state representations, and obtain the time-level hidden state representation h′ through time-level aggregation. t-1 This process can be represented as: in For the SoftMax activation function, w tsa For learnable parameters, tanh represents the hyperbolic tangent activation function; Step S58: Use a gated loop unit to perform spatiotemporal relationship reasoning; The gated loop unit has two gates: a reset gate and a reset gate. and Update Gate Used to retain the most relevant information about traffic accidents in the video sequence, the overall structure can be described by the following equation: Where σ represents the sigmoid activation function, h′ t-1 X represents the time-level hidden state representation of frame t-1 obtained after two-layer aggregation. t This represents the traffic scene characteristics input at the current moment. Represents the element-wise multiplication operator, r t h represents the candidate hidden state at time step t. t Let represent the hidden state representation of the image at frame t, and tanh represent the hyperbolic tangent activation function. W r , All of these are learnable parameters, and their dimensions are all d*d.

2. The traffic accident prediction method according to claim 1, characterized in that, Step S6 includes the following steps: The dimension of the hidden state representation of the t-th frame image obtained by S5 is reduced to 1 through a fully connected layer, thus obtaining the traffic accident score of the t-th frame image. This process can be expressed by the formula: FC stands for fully connected layer, used to reduce the feature dimension from d to 1.

3. A traffic accident prediction device, characterized in that, A method for implementing any one of claims 1 to 2 includes the following modules: The acquisition and preprocessing module is used to acquire vehicle-mounted video and perform preprocessing to obtain continuous image frames. The feature extraction module is used to extract temporal and spatial features from consecutive image frames; The spatiotemporal fusion module is used to perform intermediate-term fusion of extracted temporal and spatial features; The position weighting module is used to perform position weighting on object features based on a graph structure. The two-layer aggregation module is used to perform spatial attention on the position-weighted object-level features, then concatenate the object features with the full-frame features and perform frame-level and time-level aggregation to fully learn the traffic scene context information in the vehicle video. The obtained two-layer aggregation information is used to perform spatiotemporal relationship reasoning to obtain the hidden state representation of the current time step. The incident scoring module is used to input the aggregated hidden state representation of the current time step into the fully connected network to obtain the final incident score. The accident early warning module is used to issue an early warning signal when the obtained accident score exceeds a predetermined threshold.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the traffic accident prediction method as described in any one of claims 1-2.

5. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the traffic accident prediction method as described in any one of claims 1-2.

Images