Traffic accident prediction method, system and device and storage medium

Abstract

The invention relates to the field of computer technology, and relates to a traffic accident prediction method, which comprises the following steps: acquiring an image frame acquired by an automobile data recorder; target detection is carried out on the image frame, feature extraction is carried out on a detected target, target-level features of the image frame are obtained, and the target is a preset traffic participant; performing global feature extraction on the image frames to obtain frame-level features of the image frames; performing feature fusion on the target-level features and the frame-level features of the image frames to obtain fusion features; and determining a risk score of a future traffic accident of the image frame according to the fusion feature, and obtaining a traffic accident prediction result. The invention further provides a traffic accident prediction system and device and a storage medium. The image frame feature utilization rate can be improved, and the accuracy of a traffic accident prediction result is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a traffic accident prediction method, system, device and storage medium. Background Art

[0002] Dashcam videos provide real-world data and diverse scenarios, providing valuable information for analysis. Researchers often use dashcam videos to study traffic accident prediction. At present, common methods for traffic accident prediction include track prediction and spatial relationship analysis between targets in image frames. Track prediction predicts traffic accidents by analyzing the movement trajectory of the vehicle itself and the moving targets in the dashcam video; spatial relationship analysis between targets in image frames uses graph neural networks (GNN for short) to capture the spatial relationship between targets in the frame, and then models the temporal relationship between different frames; and predicts accidents by studying the interaction between targets.

[0003] However, when using the track prediction method to predict traffic accidents, the dashcam video is different from the surveillance video, the dashcam has a limited field of view, and the recording device is also very mobile. Therefore, the running trajectory of the vehicle itself and the moving target in the dashcam is not accurate enough, which makes the track prediction method poor in predicting traffic accidents; and most of the spatial relationship analysis methods based on graph neural networks only regard the targets in the image frame as nodes in the graph network, and use target-level features to represent the entire frame. Although the spatial relationship analysis method based on graph neural networks can help solve the complexity of dashcam video scenes and help the model better identify traffic participants related to accidents, it also leads to information omissions and ignores other content in the image frame, resulting in unsatisfactory feature utilization. Summary of the invention

[0004] In order to make a more accurate prediction of traffic accidents, a traffic accident prediction method, system, device and storage medium are provided.

[0005] In order to achieve the above-mentioned object of the present invention, according to a first aspect of the present invention, the present invention provides a traffic accident prediction method, comprising:

[0006] Get the image frames captured by the driving recorder;

[0007] Performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame, wherein the target is a preset traffic participant;

[0008] Perform global feature extraction on the image frame to obtain frame-level features of the image frame;

[0009] Perform feature fusion on the object-level features and frame-level features of the image frame to obtain fused features;

[0010] The risk score of a traffic accident occurring in the future of the image frame is determined based on the fusion features, and the traffic accident prediction result is obtained.

[0011] Optionally, the frame-level feature is a global feature of the image frame, and the frame-level feature is used to represent the features of the entire image frame;

[0012] The target-level feature is a local feature of the image frame. The target-level feature is used to represent the features of the target contained in the image frame. The target is the traffic participant related to the traffic accident in the image frame.

[0013] The extracting features of each image frame in the video clip and outputting the object-level features and frame-level features of each image frame includes:

[0014] Performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame;

[0015] Perform global feature extraction on the image frame to obtain the frame-level features of the image frame.

[0016] Optionally, the step of fusing the object-level features and the frame-level features of the image frame to obtain the fused features includes:

[0017] The preset attention mechanism is used to weight the target-level features and frame-level features to obtain the attention-weighted features;

[0018] The fused features are obtained based on attention-weighted features, object-level features and frame-level features.

[0019] Optionally, the attention mechanism is:

[0020] Input the target-level features into the linear layer to obtain the query vector Q of the target-level features;

[0021] Input the frame-level features into the first fully connected layer to obtain the key vector K of the frame-level features;

[0022] Input the frame-level features into the second fully connected layer to obtain a numerical vector V of the frame-level features;

[0023] The key vector K of the frame-level feature, the numerical vector V of the frame-level feature, and the query vector Q of the target-level feature are used to calculate the attention mechanism to obtain the first attention weighted feature;

[0024] Input the frame-level features into the linear layer to obtain the query vector Q of the frame-level features;

[0025] Input the target-level features into the first fully connected layer to obtain the key vector K of the target-level features;

[0026] Input the target-level features into the second fully connected layer to obtain a numerical vector V of the target-level features;

[0027] The key vector K of the target-level feature, the numerical vector V of the target-level feature, and the query vector Q of the frame-level feature are used to calculate the attention mechanism to obtain the second attention weighted feature;

[0028] The calculation formula of attention weighted feature is

[0029] Among them, K T represents the transpose of the key vector K, d2 represents the feature dimension, and Softmax() is the normalized exponential function.

[0030] Optionally, obtaining fused features based on attention weighted features, object-level features, and frame-level features includes:

[0031] Add the target-level feature to the first attention-weighted feature to obtain the first global-local feature;

[0032] Add the frame-level feature to the second attention-weighted feature to obtain the second global-local feature;

[0033] The first global-local feature and the second global-local feature are connected to obtain a fused feature.

[0034] Optionally, the performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame includes:

[0035] The image frame is input into the target detector, which analyzes the image frame and outputs the recognition result of the image frame, wherein the recognition result includes the target and the bounding box of the target.

[0036] The recognition result is input into the feature extractor, which extracts features from the image within the bounding box of the target and outputs target-level features.

[0037] Optionally, the inputting the image frame into the target detector, the target detector parsing the image frame, and outputting the recognition result of the image frame includes:

[0038] Input the image frame into the object detector;

[0039] Receiving an object in the image frame identified by the object detector after parsing the image frame;

[0040] Compare the number of targets identified by the target detector with the target number reference value N, where N is a positive integer:

[0041] If the number of targets identified by the target detector is equal to the target number reference value N, a bounding box is added to the location of the target in the image frame and the recognition result is output;

[0042] If the number of targets recognized by the target detector is greater than the target number reference value N, the first N targets with the highest confidence scores are selected as the targets to be output, a bounding box is added to the position of the target to be output in the image frame, and the recognition result is output;

[0043] If the number of targets recognized by the target detector is less than the target number reference value N, the missing number of targets in the image frame is filled with zeros, a bounding box is added at the location of the target in the image frame, and the recognition result is output.

[0044] Optionally, after performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame, and before performing feature fusion on the target-level features and frame-level features of the image frame to obtain fused features, the method further includes:

[0045] Obtain the location information of N targets in the image frame;

[0046] Determine the relationship information between objects in the frame according to the position information;

[0047] The object-level features extracted by the feature extractor are processed using a graph convolutional network, and the object-level features of N objects are updated according to the relationship information between objects in the frame to obtain updated object-level features.

[0048] In order to solve the above problem, according to a second aspect of the present invention, the present invention further provides a traffic accident prediction system, the system comprising:

[0049] An acquisition module, used to acquire image frames collected by a driving recorder;

[0050] A first feature extraction module is used to perform target detection on the image frame and extract features of the detected target to obtain target-level features of the image frame, wherein the target is a preset traffic participant;

[0051] The second feature extraction module is used to extract global features of the image frame to obtain frame-level features of the image frame;

[0052] A feature fusion module is used to fuse the target-level features and frame-level features of the image frame to obtain fused features;

[0053] The output module is used to determine the risk score of traffic accidents in the image frame in the future based on the fusion features and obtain the traffic accident prediction result.

[0054] In order to solve the above problem, according to a third aspect of the present invention, the present invention further provides an electronic device, the electronic device comprising:

[0055] at least one processor; and,

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the traffic accident prediction method described above.

[0058] In order to solve the above-mentioned problem, the fourth aspect of the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned traffic accident prediction method.

[0059] In summary, this application includes the following beneficial technical effects:

[0060] When processing video clips, different types of information are captured from the same image frame, and the global features of the entire image frame (i.e., frame-level features) and the local features of each traffic participant in the image frame (i.e., object-level features) are extracted respectively; by integrating frame-level features and object-level features, the processing efficiency of driving record video clips with complex scenes can be improved, and the recognition rate of traffic participants related to accidents can be improved; in addition, the object-level features and frame-level features are combined to provide a more comprehensive frame representation, thereby improving the accuracy of the prediction results of traffic accidents;

[0061] The preset attention mechanism is used to perform weighted processing on the target-level features and frame-level features, and then the attention-weighted features, target-level features and frame-level features obtained by weighted processing are connected to obtain fused features. The fused features after attention-weighted processing can facilitate the model to capture the similarities between different features in the same image frame and improve the utilization rate of features. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of the system structure of an accident prediction model corresponding to the traffic accident prediction method provided by an embodiment of the present invention;

[0063] Figure 2 A flow chart of the execution steps of a traffic accident prediction method provided by an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of the steps of a method for acquiring target-level features provided by an embodiment of the present invention;

[0065] Figure 4 A schematic diagram of the steps of a method for object recognition and object-level feature extraction in an image frame provided by an embodiment of the present invention;

[0066] Figure 5 A schematic diagram of the steps of a method for object recognition and object-level feature extraction in an image frame provided in a preferred implementation manner of an embodiment of the present invention;

[0067] Figure 6 A flowchart of the steps of fusing target-level features and frame-level features of an image frame to obtain fused features provided by an embodiment of the present invention;

[0068] Figure 7 A schematic diagram of a module of a feature fusion part in an accident prediction model provided by an embodiment of the present invention;

[0069] Figure 8 A flowchart of the steps of obtaining fused features based on attention weighted features, object-level features and frame-level features provided in one embodiment of the present invention;

[0070] Fig. 9 A functional module diagram of a traffic accident prediction system provided by an embodiment of the present invention;

[0071] Fig.10 A schematic diagram of the structure of an electronic device for implementing the traffic accident prediction method provided by one embodiment of the present invention.

[0072] Figure numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface; 91, acquisition module; 92, first feature extraction module; 93, second feature extraction module; 94, feature fusion module; 95, output module.

[0073] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0074] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0075] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0076] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0077] Reference Figure 1 and Figure 2 , is a flow chart of a traffic accident prediction method provided by an embodiment of the present invention. In this embodiment, the traffic accident prediction method includes:

[0078] S1. Obtain image frames captured by a driving recorder.

[0079] During the driving process of the vehicle, the video clips in the driving recorder are extracted in real time. For example, the image frames of the driving recorder video can be extracted with a time period of T. The user can set the extraction period T according to actual needs, and this implementation does not impose any restrictions.

[0080] S2. Perform target detection on the image frame and extract features of the detected target to obtain target-level features of the image frame.

[0081] The target-level feature is a local feature of the image frame. The target-level feature is used to represent the features of the target contained in the image frame. The target is the traffic participant related to the traffic accident in the image frame.

[0082] Specifically, in this embodiment, traffic participants refer to moving objects such as people, vehicles and animals in motion in a vehicle, as well as objects that are about to move, such as people, vehicles or animals that are about to enter a dynamic state; traffic participants also include stationary objects on or near the road (such as green plants, guardrails, street lights, buildings, temporarily parked vehicles and some temporarily placed objects, etc.); traffic participants can also be other objects that affect the operation of the vehicle, which will not be listed here one by one for the sake of brevity of the specification.

[0083] In this embodiment, the target level feature is denoted as o t , Among them, d1 represents the target level feature o t The dimension of is, N represents the number of targets detected in the image frame.

[0084] S3. Perform global feature extraction on the image frame to obtain frame-level features of the image frame.

[0085] The frame-level feature is the global feature of the image frame, and the frame-level feature is used to represent the features of the entire image frame.

[0086] Reference Figure 1 ,In the feature extraction stage, a dual-branch feature extraction module is used to extract ,frame-level features and object-level features respectively; one branch performs object ,detection and extracts object-level features, and the other branch extracts frame-level features.

[0087] The frame-level feature is denoted as f t , Among them, d2 represents the frame-level feature denoted by f t The dimension of is, l represents the length of the frame-level feature.

[0088] I3D (Two-Stream Inflated 3D ConvNets) is used to extract frame-level features, and the feature dimension is d2 = 2048. Frame-level features extracted by I3D have time knowledge and can improve the accuracy of traffic accident recognition results.

[0089] Reference Figure 3 , performing target detection on the image frame and extracting features of the detected target, the specific steps of obtaining the target-level features of the image frame include:

[0090] S211, input the image frame into the target detector, the target detector analyzes the image frame, and outputs the recognition result of the image frame.

[0091] The recognition result includes the target and the bounding box of the target. In this embodiment, the recognition result includes N targets, where N is a positive integer. The specific value of N can be set according to the actual situation and the accuracy required by the prediction result, and this embodiment does not impose any limitation.

[0092] After detecting the target, we number the targets in the image frame, and the bounding box of the target is represented by four coordinates, such as (x1, y1, x2, y2). The area enclosed by the four coordinates is the area corresponding to the target.

[0093] Reference Figure 4 The specific operation of step S211 is as follows:

[0094] S201, input the image frame into the target detector.

[0095] In this embodiment, we can use Faster R-CNN (Faster Region-based Convolutional Neural Networks) and Cascade R-CNN (Cascade Region-based Convolutional Neural Networks) as target detectors to detect targets in image frames; Faster R-CNN is a two-stage target detection algorithm, and Cascade R-CNN is an algorithm improved from Faster R-CNN. Cascade R-CNN improves the performance of target detection by cascading multiple R-CNN networks; Cascade R-CNN continuously optimizes the detection results by cascading multiple R-CNN networks based on different IOU (Intersection over Union) thresholds.

[0096] S202, receiving a target in the image frame identified by the target detector after parsing the image frame.

[0097] S203, comparing the number of targets identified by the target detector with a target number reference value N, and processing the identified targets according to the comparison result to obtain a recognition result.

[0098] The specific processing method is:

[0099] If the number of targets recognized by the target detector is equal to the target number reference value N, a bounding box is added to the position of the target in the image frame and the recognition result is output; in this embodiment, the number of targets output per image frame is set to 19.

[0100] If the number of targets recognized by the target detector is greater than the target number reference value N, the first N targets with the highest confidence scores are selected as the targets to be output, a bounding box is added to the position of the target to be output in the image frame, and the recognition result is output;

[0101] If the number of targets recognized by the target detector is less than the target number reference value N, the missing number of targets in the image frame is filled with zeros, a bounding box is added at the location of the target in the image frame, and the recognition result is output.

[0102] When the traffic scene is complex, it usually involves many traffic participants and changing backgrounds, so it is easy for the number of identified targets to be greater than N; however, only a small number of these participants are related to potential traffic accidents. Since computers usually take a long time to extract and fuse features from image frames; in this embodiment, we limit the number of targets, and when the number of targets in the image frame is large, targets with little relationship to traffic accidents are eliminated; in target detection, the model assigns a confidence score to each detected target, which reflects the model's confidence in the correctness of the detection results; only the top N targets with the highest confidence scores are selected for feature extraction, so that the model can judge the probability of an accident based on the target-level features of the N targets most relevant to the traffic accident, while being able to achieve the effect of saving computing power while realizing the prediction of traffic accidents, reducing the time for traffic accident prediction, and improving the prediction efficiency of traffic accidents.

[0103] When the number of recognized targets in an image frame is less than N, the missing parts are filled with 0, and the number of output targets is unified (all N), which can improve the convenience of subsequent processing of the targets.

[0104] S212: Input the recognition result into a feature extractor, and the feature extractor extracts features from the image within the bounding box of the target and outputs target-level features.

[0105] In this embodiment, VGG-16 is used to extract target-level features, and the feature dimension is d1=4096; wherein, VGG-16 is the full name of Visual Geometry Group 16, which is a deep convolutional neural network model.

[0106] Reference Figure 5 As a preferred implementation of this embodiment, after performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame, and before performing feature fusion on the target-level features and frame-level features of the image frame to obtain fused features, the traffic accident prediction method further includes:

[0107] S213: Obtain position information of N targets in the image frame.

[0108] S214. Determine the relationship information between objects in the frame according to the position information.

[0109] In this embodiment, the relationship information between objects in the frame includes the adjacency matrix of the image frame object relationship graph. In this embodiment, each object identified in the image frame is regarded as a node, and edges are constructed between each pair of nodes to generate the image frame object relationship graph; we use the two-dimensional relative positions of the objects in the image frame to define the adjacency matrix.

[0110] The expression of the adjacency matrix is:

[0111]

[0112] Where i is the index of the first target, j is the index of the second target, and d(o i , o j ) is the Euclidean distance between the first target i and the second target j; i is the two-dimensional coordinate of the first target i, o j is the two-dimensional coordinate of the second target j;

[0113] S215. Use a graph convolutional network to process the object-level features extracted by the feature extractor, and update the object-level features of N objects according to the relationship information between objects in the frame to obtain updated object-level features.

[0114] Specifically, we use a graph convolutional network (GCN) for message passing to update node features, and then use a pooling layer to obtain the updated target-level frame features.

[0115] The expression of the updated target-level frame feature is: t =Pool(GCN(o t ,A o (i,j))),

[0116] Among them, Pool represents the maximum pooling operation, GCN represents the graph convolution layer processing, t Represents the target-level frame features extracted by the feature extractor.

[0117] S4. Fusing the object-level features and frame-level features of the image frame to obtain fused features.

[0118] Reference Figure 6 and Figure 7 , the target-level features and frame-level features of the image frame are fused to obtain fused features, including:

[0119] S41, using a preset attention mechanism to perform weighted processing on the target-level features and the frame-level features to obtain attention-weighted features;

[0120] In this embodiment, the target-level features and the frame-level features are processed using a global-local feature fusion module; the attention-weighted features include a first attention-weighted feature and a second attention-weighted feature; Figure 7 , the attention mechanism is:

[0121] Input the target-level features into the linear layer to obtain the query vector Q of the target-level features;

[0122] Input the frame-level features into the first fully connected layer to obtain the key vector K of the frame-level features;

[0123] Input the frame-level features into the second fully connected layer to obtain a numerical vector V of the frame-level features;

[0124] The key vector K of the frame-level feature, the numerical vector V of the frame-level feature, and the query vector Q of the target-level feature are used to calculate the attention mechanism to obtain the first attention weighted feature;

[0125] Input the frame-level features into the linear layer to obtain the query vector Q of the frame-level features;

[0126] Input the target-level features into the first fully connected layer to obtain the key vector K of the target-level features;

[0127] Input the target-level features into the second fully connected layer to obtain a numerical vector V of the target-level features;

[0128] The key vector K of the target-level feature, the numerical vector V of the target-level feature, and the query vector Q of the frame-level feature are used to calculate the attention mechanism to obtain the second attention weighted feature.

[0129] The calculation formula of attention weighted feature is

[0130] Among them, K T represents the transpose of the key vector K, d2 represents the feature dimension, and Softmax() is the normalized exponential function.

[0131] S42. Obtain fused features based on attention-weighted features, object-level features, and frame-level features.

[0132] Reference Figure 7 and Figure 8 The specific execution steps of step S32 include:

[0133] S421, adding the target-level feature to the first attention-weighted feature to obtain a first global-local feature;

[0134] S3422, adding the frame-level feature to the second attention weighted feature to obtain a second global-local feature;

[0135] S423: Connect the first global-local feature and the second global-local feature to obtain a fusion feature.

[0136] based on Figure 7 The fusion module shown processes the object-level features and the frame-level features, and can better fuse the object-level features and the frame-level features.

[0137] In this embodiment, the performance of the global-local feature fusion module is verified by two sets of ablation experiments. One set uses a direct connection method to combine object-level features and frame-level features. The other set of experiments uses a single attention module: in one experiment, only object-level features are used to apply attention weighting to frame-level features (Frame-weighted), and in another experiment, only frame-level features are used to apply attention weighting to object-level features (Object-weighted). In each case, the attention-weighted features are then fused with another type of original unweighted features, and the results are shown in Table 1:

[0138] Table 1 compares different feature fusion methods on the DAD dataset.

[0139]

[0140]

[0141] When concatenation is chosen as the feature fusion method, both mAP and mTTA drop, with mAP dropping by 7.5% and mTTA dropping by 0.16s. For the method using single attention, mAP and mTTA are improved compared to direct concatenation. However, these results are still inferior to those achieved using global-local feature fusion methods.

[0142] It should be noted that the DAD dataset is a dataset collected by VSLab of National Tsing Hua University in Taiwan and is widely used for dashcam accident prediction. mAP stands for Mean Average Precision, which means the mean average precision. mAP is a key indicator for evaluating target detection models. mAP is used to measure the detection performance of the model for multiple categories.

[0143] mTTA stands for Mean Time To Analysis, which is an indicator for measuring the real-time performance of a model. mTTA indicates the average time required for a model to complete an analysis. The shorter the mTTA, the better the real-time performance of the model, and the faster it can process images or data.

[0144] S5. Determine the risk score of a traffic accident occurring in the image frame in the future according to the fused features, and obtain a traffic accident prediction result.

[0145] In this embodiment, a directed graph is constructed based on the fusion features to represent the temporal relationship between frames, and then a directed graph network and a fully connected layer are used to obtain the final risk score; when the obtained risk score is lower than the preset score reference value, it indicates that the probability of a traffic accident is low; when the obtained risk score is higher than the preset score reference value, it indicates that the probability of a traffic accident is high, there is a risk of a traffic accident, and a traffic accident prediction result is obtained.

[0146] Reference Fig. 9 Based on the same inventive concept, an embodiment of the present invention provides a traffic accident prediction system.

[0147] The traffic accident prediction system of the present invention can be installed in an electronic device. According to the functions to be implemented, the traffic accident prediction system includes an acquisition module 91 , a first feature extraction module 92 , a second feature extraction module 93 , a feature fusion module 94 and an output module 95 .

[0148] The acquisition module 91 can acquire the video clip to be predicted, the first feature extraction module 92 can perform target detection on the image frame, and extract features of the detected target to obtain target-level features of the image frame, wherein the target is a preset traffic participant; the second feature extraction module 93 performs global feature extraction on the image frame to obtain frame-level features of the image frame; the feature fusion module 94 can fuse the target-level features and frame-level features of the image frame to obtain fused features; the output module 95 can determine the risk score of a traffic accident in the future of the image frame based on the fused features to obtain a traffic accident prediction result.

[0149] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0150] The various variations and specific examples of the traffic accident prediction method provided in the above embodiment are also applicable to the traffic accident prediction system of this embodiment. Through the above detailed description of the traffic accident prediction method, those skilled in the art can clearly know the implementation method of the traffic accident prediction system in this embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0151] The present application also discloses an electronic device, such as Fig.10 , which is a schematic diagram of the structure of an electronic device of a traffic accident prediction method provided by an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 connected to the at least one processor, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a traffic accident prediction method program.

[0152] It should be noted that in some examples of the present application, the electronic device for processing the image frames of the driving recorder can be a microcomputer device installed on the vehicle, and the image frames extracted from the video captured by the driving recorder are directly sent to the microcomputer device installed on the vehicle for processing; in another example of this embodiment, the extracted image frames can also be uploaded to the cloud platform, and the image frames captured by the driving recorder are processed by a computing device with greater computing power. It is only necessary to feed back the prediction results to the microcomputer device mounted on the vehicle, and this embodiment is not limited.

[0153] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a traffic accident prediction method, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0154] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the traffic accident prediction method program, but also be used to temporarily store data that has been output or is to be output.

[0155] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0156] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0157] Fig.10 Only an electronic device with components is shown, and those skilled in the art will understand that Fig.10 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0158] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0159] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0160] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0161] The embodiment of the present application provides a computer-readable storage medium, for example, including: any entity or device, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM) that can carry the computer program code. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the traffic accident prediction method of the above embodiment.

[0162] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0163] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A traffic accident prediction method, characterized in that: include: Get the image frames captured by the driving recorder; Performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame, wherein the target is a preset traffic participant; Perform global feature extraction on the image frame to obtain frame-level features of the image frame; Perform feature fusion on the object-level features and frame-level features of the image frame to obtain fused features; The risk score of a traffic accident occurring in the future of the image frame is determined based on the fusion features, and the traffic accident prediction result is obtained.

2. The traffic accident prediction method according to claim 1, characterized in that: The step of fusing the target-level features and the frame-level features of the image frame to obtain the fused features includes: The preset attention mechanism is used to weight the target-level features and frame-level features to obtain the attention-weighted features; The fused features are obtained based on attention-weighted features, object-level features and frame-level features.

3. The traffic accident prediction method according to claim 2, characterized in that: The attention mechanism is: Input the target-level features into the linear layer to obtain the query vector Q of the target-level features; Input the frame-level features into the first fully connected layer to obtain the key vector K of the frame-level features; Input the frame-level features into the second fully connected layer to obtain the numerical vector V of the frame-level features; The key vector K of the frame-level feature, the numerical vector V of the frame-level feature, and the query vector Q of the target-level feature are used to calculate the attention mechanism to obtain the first attention weighted feature; Input the frame-level features into the linear layer to obtain the query vector Q of the frame-level features; Input the target-level features into the first fully connected layer to obtain the key vector K of the target-level features; Input the target-level features into the second fully connected layer to obtain a numerical vector V of the target-level features; The key vector K of the target-level feature, the numerical vector V of the target-level feature, and the query vector Q of the frame-level feature are used to calculate the attention mechanism to obtain the second attention weighted feature; The calculation formula of attention weighted feature is Among them, K T represents the transpose of the key vector K, d2 represents the feature dimension, and Softmax() is the normalized exponential function.

4. The traffic accident prediction method according to claim 3, characterized in that: The fusion feature obtained based on the attention weighted feature, the target level feature and the frame level feature includes: Add the target-level feature to the first attention-weighted feature to obtain the first global-local feature; Add the frame-level feature to the second attention-weighted feature to obtain the second global-local feature; The first global-local feature and the second global-local feature are connected to obtain a fused feature.

5. The traffic accident prediction method according to any one of claims 1 to 4, characterized in that: The performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame includes: The image frame is input into the target detector, which parses the image frame and outputs the recognition result of the image frame, wherein the recognition result includes the target and the bounding box of the target. The recognition result is input into the feature extractor, which extracts features from the image within the bounding box of the target and outputs target-level features.

6. The traffic accident prediction method according to claim 5, characterized in that: The image frame is input into the target detector, the target detector analyzes the image frame, and outputs the recognition result of the image frame, including: Input the image frame into the object detector; Receiving an object in the image frame identified by the object detector after parsing the image frame; Compare the number of targets identified by the target detector with the target number reference value N, where N is a positive integer: If the number of targets identified by the target detector is equal to the target number reference value N, a bounding box is added to the location of the target in the image frame and the recognition result is output; If the number of targets recognized by the target detector is greater than the target number reference value N, the first N targets with the highest confidence scores are selected as the targets to be output, a bounding box is added to the position of the target to be output in the image frame, and the recognition result is output; If the number of targets recognized by the target detector is less than the target number reference value N, the missing number of targets in the image frame is filled with zeros, a bounding box is added at the location of the target in the image frame, and the recognition result is output.

7. The traffic accident prediction method according to claim 5 or 6, characterized in that: After performing target detection on the image frame and extracting features of the detected target to obtain target-level features of the image frame, and before performing feature fusion on the target-level features and frame-level features of the image frame to obtain fused features, the method further includes: Obtain the position information of each target in the image frame; Determine the relationship information between objects in the frame according to the position information; The object-level features extracted by the feature extractor are processed using a graph convolutional network, and the object-level features of each object are updated according to the relationship information between objects in the frame to obtain updated object-level features.

8. A traffic accident prediction system, used to implement the traffic accident prediction method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire image frames collected by a driving recorder; A first feature extraction module is used to perform target detection on the image frame and extract features of the detected target to obtain target-level features of the image frame, wherein the target is a preset traffic participant; The second feature extraction module is used to extract global features of the image frame to obtain frame-level features of the image frame; A feature fusion module is used to fuse the target-level features and frame-level features of the image frame to obtain fused features; The output module is used to determine the risk score of traffic accidents in the image frame in the future based on the fusion features and obtain the traffic accident prediction result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the traffic accident prediction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed by a processor, the traffic accident prediction method according to any one of claims 1 to 7 is implemented.

Images