Traffic accident prediction method and device based on deep learning
By performing feature extraction and synergistic attention mechanism fusion on-board videos, combined with GRU network model, the problem of insufficient understanding of traffic accident prediction methods in the existing technology is solved, and a more accurate and comprehensive traffic accident prediction is achieved.
Patent Information
- Application Number
- CN202411208544.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-10
AI Technical Summary
The existing traffic accident prediction method for on-board video captures the relationship between traffic targets and scenes in two-dimensional pictures. The prediction model has poor understanding of traffic scenes, resulting in a lack of accuracy and comprehensiveness in the prediction.
Using a deep learning-based method, the RGB graph global features, RGB graph local features, depth graph global features and depth graph local features are obtained by extracting the monitoring video, and the coordinated attention mechanism is used to fuse these features and input them to the GRU network. Model to predict traffic accident probability.
Through the complementary perception fusion of multiple features, the accuracy and comprehensiveness of traffic accident prediction are improved, and the occurrence of traffic accidents is reduced.
Smart Images
Figure CN119107614A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a traffic accident prediction method and device based on deep learning. Background Art
[0002] Traffic accidents have caused immeasurable losses to people's lives and property, so safe driving has become an urgent need for the public. One of the biggest challenges facing safe driving is how to accurately predict the occurrence of accidents, so that necessary measures can be taken to avoid traffic accidents when they are about to happen. If accurate traffic accident prediction results can be obtained, drivers and self-driving cars will have enough reaction time to take emergency avoidance measures, so traffic accident prediction can enhance the safe driving capabilities of human driving and self-driving cars. At present, most vehicles are equipped with driving recorders, which can record video images during vehicle driving. On-board videos can provide a stable source of prediction data for traffic accident prediction methods. If it is possible to predict the possible future traffic accidents ahead of the vehicle in real time based on the video captured by the driving recorder, that is, the on-board video, it will allow the driver to have enough time to avoid traffic accidents during driving.
[0003] However, current traffic accident prediction methods for vehicle-mounted videos are mainly limited to capturing the relationship between various traffic targets and between traffic targets and traffic scenes in two-dimensional images. The prediction model's ability to understand traffic scenes is poor, which limits the accuracy and comprehensiveness of traffic accident predictions. Summary of the invention
[0004] The purpose of the present invention is to solve the problem that the current traffic accident prediction method for vehicle-mounted video lacks accuracy and comprehensiveness in predicting traffic accidents. The present invention provides a traffic accident prediction method and device based on deep learning.
[0005] The technical solution of the embodiment of the present application is implemented as follows: A first aspect of an embodiment of the present application provides a traffic accident prediction method based on deep learning, comprising: Obtain surveillance video corresponding to the target traffic scene; Extracting features from the surveillance video to obtain target image features; the target image features include RGB image global features, RGB image local features, depth image global features, and depth image local features; Using a collaborative attention mechanism to obtain correlation information between different features in the target image features, and fusing each feature in the target image features based on the correlation information to obtain a target fused feature; The target fusion feature is input into the trained GRU network model, and the predicted probability of a traffic accident corresponding to the target traffic scenario is obtained as output.
[0006] Optionally, the extracting features from the surveillance video to obtain target image features includes: Processing the surveillance video by frame division to obtain a target image frame; The target image frame is input into the VGG-16 network for feature extraction to obtain the target image features.
[0007] Optionally, the using of the collaborative attention mechanism to obtain correlation information between different features in the target image features, and fusing each feature in the target image features based on the correlation information to obtain a target fused feature, includes: Any two features of the target image features are fused, and the fused features are fused with the remaining features one by one to obtain the target fused features.
[0008] Optionally, the using of the collaborative attention mechanism to obtain correlation information between different features in the target image features, and fusing each feature in the target image features based on the correlation information to obtain a target fused feature, includes: Fusing the RGB image global feature and the RGB image local feature of the target image frame to obtain a first target fusion feature; Fusing the first target fusion feature with the global feature of the depth map of the target image frame to obtain a second target fusion feature; The second target fusion feature is fused with the local feature of the depth map of the target image frame to obtain the target fusion feature.
[0009] Optionally, the collaborative attention mechanism is: ; in represents the computation process of the collaborative attention block, Represents a splicing operation, are the parameters of the collaborative embedding layer, and Given two features.
[0010] Optionally, the step of inputting the target fusion feature into a trained GRU network model and outputting a traffic accident prediction probability corresponding to the target traffic scene includes: The target fusion feature and the output information of the GRU network model at the previous moment are input into the trained GRU network model, and the prediction information is output.
[0011] Optionally, before inputting the target fusion feature into a trained GRU network model to output a traffic accident prediction probability corresponding to the target traffic scene, the method further includes: The original network model is trained using historical surveillance videos and the binary cross entropy loss function to obtain a trained GRU network model.
[0012] A second aspect of the embodiment of the present application provides a traffic accident prediction device based on deep learning, including: a video acquisition module, a feature extraction module, a feature fusion module and a prediction output module, wherein: The video acquisition module is configured to acquire the surveillance video corresponding to the target traffic scene; The feature extraction module is configured to extract features from the surveillance video to obtain target image features; the target image features include RGB image global features, RGB image local features, depth image global features, and depth image local features; The feature fusion module is configured to obtain association information between different features in the target image features by using a collaborative attention mechanism, and fuse each feature in the target image features based on the association information to obtain a target fusion feature; The prediction output module is configured to input the target fusion feature into the trained GRU network model, and output the predicted probability of a traffic accident corresponding to the target traffic scene.
[0013] A third aspect of an embodiment of the present application provides an electronic device, comprising a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the deep learning-based traffic accident prediction method described in the first aspect.
[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0015] Compared with the prior art, the technical solution provided by this application has the following beneficial effects: The present invention provides a traffic accident prediction method and device based on deep learning, which extracts features from the surveillance video corresponding to the target traffic scene to obtain target image features, including RGB image global features, RGB image local features, depth image global features and depth image local features, and then uses a collaborative attention mechanism to obtain the correlation information between different features in the target image features, and fuses each feature in the target image features based on the correlation information to obtain a target fusion feature; the target fusion feature is input into a trained GRU network model, and the output is the prediction probability of a traffic accident corresponding to the target traffic scene. Through the cascade combination of collaborative attention layers, the complementary perception fusion of multiple features is realized, and then the GRU module is used to predict whether a traffic accident will occur in the future, thereby reducing the occurrence of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A flowchart of a traffic accident prediction method based on deep learning provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a traffic accident prediction device based on deep learning provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0017] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application. The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components. All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.
[0018] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions, when executed by the processor, may create a device for implementing the functions / operations described in these block diagrams and / or flow charts.
[0019] In some embodiments, see Figure 1 , Figure 1 A flow chart of a traffic accident prediction method based on deep learning provided in an embodiment of the present application; the traffic accident prediction method based on deep learning provided in an embodiment of the present application includes: S110, obtaining a surveillance video corresponding to the target traffic scene.
[0020] The surveillance video here can be a vehicle-mounted video, which is acquired in real time through the Internet of Vehicles combined with cameras on the road. The target traffic scene can be a certain road section or a specific area.
[0021] S120, extracting features from the surveillance video to obtain target image features; the target image features include RGB image global features, RGB image local features, depth image global features, and depth image local features.
[0022] The global features of the RGB image are the statistical characteristics or overall properties of the entire image area, which are independent of the specific location in the image and the global color distribution of the image. The local features of the RGB image focus on small areas or specific points of interest in the image, and are robust to rotation, scale transformation and illumination changes. The global features of the depth map include the distribution of each depth value in the entire depth map. The local features of the depth map are the features of detecting significant points in the areas with large changes in the depth map. These points often correspond to the edges of objects or depth discontinuities in the scene, so as to extract edge features on the depth map through edge detection algorithms (such as Canny, Sobel, etc.).
[0023] In one example, Lite-Mono is used to process each frame of a car video, and Magma pseudo-color mapping is used to represent depth information. Lighter colors (such as yellow) represent closer distance values, while darker colors (such as dark purple) represent farther distance values.
[0024] In some embodiments, S120, extracting features from the surveillance video to obtain target image features includes: Process the surveillance video by dividing it into frames to obtain the target image frame; The target image frame is input into the VGG-16 network for feature extraction to obtain the target image features.
[0025] In this embodiment, the vehicle video is divided into multiple video frames, and one or more frames are selected and input into the VGG-16 network for feature extraction. The VGG-16 network has 16 layers with parameters (excluding the pooling layer and the Flatten layer before the fully connected layer). The entire network uses a small 3x3 convolution kernel and a 2x2 maximum pooling layer.
[0026] In one example, the tth frame of the video frame sequence is input into the VGG-16 network, and the output of the penultimate fully connected layer of the VGG-16 network is taken as the extracted image features. After the penultimate fully connected layer of the VGG-16, this embodiment adds a fully connected layer with a ReLU activation function. Here, "i-fc" is used to represent the fully connected layer. i-fc outputs the global features of the RGB image of the tth frame of the vehicle video. ,and . In another example, a target detector (YOLO, etc.) is used to detect each frame of the vehicle video to identify traffic targets such as pedestrians, cars, trucks, buses, and motorcycles. If there are more than N traffic targets in the frame, the N targets with the highest detection scores are selected. If there are not enough, the N target position coordinates are padded with 0 vectors. Next, the image is cropped using the bounding box obtained by the target detector and scaled to 224×224 size through bilinear interpolation. The VGG-16 network is then used to extract features from the target area image. In order to keep the dimension consistent with the global features of the RGB image extracted above, the output of the second to last fully connected layer of the VGG-16 network is still used as the image feature. After obtaining the features of the N regions, a fully connected layer with a ReLU activation function is used to reduce the feature dimension to , and then through a fully connected layer, the feature dimension is reduced to In this embodiment, “o-fc” is used to represent these two fully connected layers. o-fc outputs the vehicle video. Local features of frame RGB images ,and .
[0027] In another example, the dimension is kept consistent with the global features of the RGB image extracted above. Here, the tth frame of the depth image sequence processed by pseudo color mapping is input to the VGG-16 network. The output of the second to last fully connected layer of the VGG-16 network is still taken as the depth image feature and input to the fully connected layer with the ReLU activation function. This embodiment uses "di-fc" to represent the fully connected layer. The di-fc outputs the global features of the tth frame depth image of the vehicle video. ,and .
[0028] In another example, the depth image processed by the pseudo-color mapping obtained above and the obtained target bounding box are used to extract local features of the depth image of the vehicle video. The pseudo-color image in each target bounding box is scaled to 224×224 using the bilinear interpolation method, and the feature extraction is performed using the VGG-16 network. In order to keep the dimension consistent with the image features extracted above, the output of the second to last fully connected layer of the VGG-16 network is still taken as the extracted depth image feature. After obtaining the feature representation of N regions, the dimension is reduced to , and then another fully connected layer reduces the dimension to In this embodiment, “do-fc” is used to represent these two fully connected layers. do-fc outputs the local features of the depth image of the tth frame of the vehicle video. ,and .
[0029] S130, using the collaborative attention mechanism to obtain the correlation information between different features in the target image features, and fuse the various features in the target image features based on the correlation information to obtain the target fused features.
[0030] Co-attention is a method for modeling dense interactions between two features. For example, the co-attention layer is given two features and , the collaborative attention block computes the query matrix , key matrix Sum Matrix Then, a key matrix of features Sum Matrix As input, it is passed to another collaborative attention block. Finally, the outputs of the two collaborative attention blocks are sent to the collaborative embedding layer, which concatenates the two outputs and passes them through a fully connected layer to reduce the concatenation dimension to the original feature dimension. The input and output dimensions of the collaborative attention layer are both , which is convenient for subsequent multi-layer connections. By exchanging information between input features, the collaborative attention layer can effectively model the dense interaction relationship between input features.
[0031] In some embodiments, S130, using a collaborative attention mechanism to obtain correlation information between different features in the target image features, and fusing each feature in the target image features based on the correlation information to obtain a target fused feature, including: Any two features of the target image are fused, and the fused features are fused with the remaining features one by one to obtain the target fused features.
[0032] In this embodiment, three collaborative attention layers can be used and combined in a cascade manner. These collaborative attention layers are used to fuse the previously extracted RGB image global features, RGB image local features, depth image global features, and depth image local features. The entire fusion process is gradual, and the output of each collaborative attention layer is used as one of the inputs of the next layer to enhance the interaction and information transfer between features. In this way, the neural network can make more full use of the information between different features and improve the effect of feature fusion.
[0033] In some embodiments, S130, using a collaborative attention mechanism to obtain correlation information between different features in the target image features, and fusing each feature in the target image features based on the correlation information to obtain a target fused feature, including: The RGB global feature and the RGB local feature of the target image frame are fused to obtain a first target fusion feature; The first target fusion feature is fused with the global feature of the depth map of the target image frame to obtain a second target fusion feature; The second target fusion feature is fused with the local feature of the depth map of the target image frame to obtain the target fusion feature.
[0034] In this embodiment, the global features of the RGB image of the tth frame of the vehicle video can be fused in the first collaborative attention layer Local features of RGB images , and then fuse the output of the first collaborative attention layer in the second collaborative attention layer The global features of the depth image of the tth frame of the vehicle video , and finally the output of the second collaborative attention layer is fused in the third collaborative attention layer The first Frame depth image local features To output traffic scene features The entire complementary perceptual feature fusion process based on collaborative attention can be expressed as: ; in, , , and They are the global features of RGB image, local features of RGB image, global features of depth image, and local features of depth image of the tth frame of the vehicle video. , and The output of the three cascaded collaborative attention layers is the traffic scene feature of the tth frame of the vehicle video. .
[0035] In some embodiments, the collaborative attention mechanism is: ; in, represents the computation process of the collaborative attention block, Represents a splicing operation, are the parameters of the collaborative embedding layer, and Given two features.
[0036] here, , , .
[0037] S140, input the target fusion features into the trained GRU network model, and output the predicted probability of a traffic accident corresponding to the target traffic scenario.
[0038] In some embodiments, S140, the target fusion feature is input into the trained GRU network model, and the output is the traffic accident prediction probability corresponding to the target traffic scene, including: The target fusion features and the output information of the GRU network model at the previous moment are input into the trained GRU network model, and the prediction information is output.
[0039] The GRU network model is a recurrent neural network (RNN) architecture that controls the flow of information by introducing gating mechanisms (including update gates and reset gates) to more effectively capture long-term dependencies. The GRU network model contains two main gating mechanisms: Update Gate and Reset Gate. The Update Gate determines how much past information (i.e., the hidden state of the previous time step) needs to be retained in the hidden state of the current time step. The Reset Gate determines how much past information (i.e., the hidden state of the previous time step) needs to be ignored in order to calculate the current candidate hidden state.
[0040] The principle is: GRU receives the traffic scene features at the current moment And the hidden state representation of the previous moment (the output of the previous moment of GRU) as input. On this basis, GRU first calculates the reset gate through a Sigmoid activation function. The reset gate determines whether to combine the hidden state of the previous moment with the current traffic scene characteristics. Mix and control the degree of discarding of hidden states. Its calculation process can be expressed as follows:
[0041] in, is the weight matrix of the reset gate, It is a Sigmoid activation function, which is used to map the input value to a probability value ranging from 0 to 1. It means connecting the hidden state vector of the previous moment and the input vector of the current moment.
[0042] After this, the GRU uses another Sigmoid activation function to calculate the update gate. The update gate determines how much information is derived from the current traffic scene features. The output range of the update gate is also 0 to 1. The calculation process can be expressed as follows:
[0043] in, is the weight matrix of the update gate.
[0044] GRU is based on the output of the reset gate and the current traffic scene characteristics Compute candidate hidden states The reset gate controls the degree of discarding the hidden state of the previous moment, while the current input provides new information. This process can be expressed as follows: ; in, is the weight matrix of the candidate hidden states, Represents element-wise multiplication. is the hyperbolic tangent activation function, which is used to map the input value to the range between -1 and 1.
[0045] GRU calculates the current hidden state by linearly combining the candidate hidden state with the output of the update gate The calculation process of updating the hidden state can be expressed as follows:
[0046] The updated hidden state vector As output, it is used for subsequent processing or GRU unit calculation at the next moment.
[0047] The above steps are the workflow of the GRU network model, which controls the flow of information and the update of hidden states through the gating mechanism, thereby realizing the modeling and learning of sequence data. This embodiment uses the following formula to represent the calculation process of the entire GRU network model:
[0048] On this basis, a fully connected layer is used, labeled "p-fc". p-fc receives the current hidden state representation obtained by the GRU network as input and outputs the probability of a traffic accident in the future. The fully connected layer maps the input to the probability space by learning appropriate weights and biases, thereby predicting traffic accidents. This process can be expressed by the following formula: ; in, and are the weights and biases of the fully connected layer p-fc, is the Softmax function for normalization.
[0049] In some embodiments, at S140, before inputting the target fusion feature into the trained GRU network model and outputting the predicted probability of a traffic accident corresponding to the target traffic scene, the method further includes: The original network model is trained using historical surveillance videos and the binary cross entropy loss function to obtain a trained GRU network model.
[0050] In this embodiment, the model training loss includes frame-level loss and loss of negative class video frames.
[0051] The calculation method of the frame-level loss function is as follows: ; in, is the start time of the traffic accident provided by the dataset, is the accident score obtained by the complementary perception fusion prediction network at time t (frame t), is the frame rate of the vehicle video, is the total length of the vehicle video. In this equation: The first term is the loss of the positive video frame, whose frame-level loss coefficient increases exponentially as it approaches the event to encourage early prediction of traffic events and reaches the same level of loss coefficient as the negative video frame after the event occurs. The loss of negative class video frames is calculated as follows: .
[0052] In an optional embodiment, the network model is constructed by the pytorch deep learning framework, and the parameters of the entire network are optimized using the Adma optimizer, with a learning rate of 0.0001. Training and testing are performed on the DAD dataset. The DAD dataset contains 620 accident video sequences and 1130 normal video sequences, totaling 175,000 frames of images. Two-thirds are taken as the training set and one-third as the test set, and 80 cycles of training are performed on the dataset. On the DAD dataset, the method we proposed has achieved results that are better than the current best method in all indicators.
[0053] The embodiment of the present application extracts features from the surveillance video corresponding to the target traffic scene to obtain target image features, which include RGB global features, RGB local features, depth global features, and depth local features, and then uses a collaborative attention mechanism to obtain the correlation information between different features in the target image features, and fuses each feature in the target image features based on the correlation information to obtain target fusion features; the target fusion features are input into a trained GRU network model, and the output is the predicted probability of a traffic accident corresponding to the target traffic scene. The cascade combination of collaborative attention layers is used to achieve complementary perception fusion of multiple features, and then the GRU module is used to predict whether a traffic accident will occur in the future, thereby reducing the occurrence of traffic accidents.
[0054] In some embodiments, see Figure 2 , Figure 2The structure diagram of a traffic accident prediction device based on deep learning provided in the embodiment of the present application. The traffic accident prediction device 200 based on deep learning provided in the embodiment of the present application includes: a video acquisition module 210, a feature extraction module 220, a feature fusion module 230 and a prediction output module 240, wherein: The video acquisition module 210 is configured to acquire a surveillance video corresponding to a target traffic scene; The feature extraction module 220 is configured to extract features from the surveillance video to obtain target image features; the target image features include RGB image global features, RGB image local features, depth image global features, and depth image local features; The feature fusion module 230 is configured to obtain the correlation information between different features in the target image features by using the collaborative attention mechanism, and fuse the features in the target image features based on the correlation information to obtain the target fusion feature; The prediction output module 240 is configured to input the target fusion feature into the trained GRU network model, and output the predicted probability of a traffic accident corresponding to the target traffic scene.
[0055] In some embodiments, the feature extraction module 220 is specifically configured as follows: Process the surveillance video by dividing it into frames to obtain the target image frame; The target image frame is input into the VGG-16 network for feature extraction to obtain the target image features.
[0056] In some embodiments, the feature fusion module 230 is specifically configured as follows: Any two features of the target image are fused, and the fused features are fused with the remaining features one by one to obtain the target fused features.
[0057] In some embodiments, the feature fusion module 230 is specifically configured as follows: The RGB global feature and the RGB local feature of the target image frame are fused to obtain a first target fusion feature; The first target fusion feature is fused with the global feature of the depth map of the target image frame to obtain a second target fusion feature; The second target fusion feature is fused with the local feature of the depth map of the target image frame to obtain the target fusion feature.
[0058] In some embodiments, the collaborative attention mechanism is: ; in represents the computation process of the collaborative attention block, Represents a splicing operation, are the parameters of the collaborative embedding layer, and Given two features.
[0059] In some embodiments, the prediction output module 240 is specifically configured as follows: The target fusion features and the output information of the GRU network model at the previous moment are input into the trained GRU network model, and the prediction information is output.
[0060] In some embodiments, the traffic accident prediction device 200 based on deep learning further includes a training module, and the training module is specifically configured as follows: The original network model is trained using historical surveillance videos and the binary cross entropy loss function to obtain a trained GRU network model.
[0061] The deep learning-based traffic accident prediction device provided in the embodiment of the present application can implement each process in the embodiment corresponding to the above-mentioned deep learning-based traffic accident prediction method. To avoid repetition, it will not be repeated here.
[0062] It should be noted that the deep learning-based traffic accident prediction device provided in the embodiment of the present application and the deep learning-based traffic accident prediction method provided in the embodiment of the present application are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned deep learning-based traffic accident prediction method, and the repetitive parts will not be repeated.
[0063] In some embodiments, see Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. An electronic device 300 provided in an embodiment of the present application includes a processor 310 and a memory 320; the memory 320 stores a computer program, wherein the computer program implements the above-mentioned traffic accident prediction method based on deep learning when executed by the processor. Specifically, the processor 310 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 310 may also include an onboard memory for cache purposes. The processor 310 may be a single processing unit or multiple processing units for executing different actions of the method flow according to the embodiment of the present application. The memory 320 may be any medium capable of containing, storing, conveying, propagating or transmitting instructions. For example, the memory 320 may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device, component or propagation medium. Specific examples of the memory 320 include: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a random access memory (RAM) or a flash memory; and / or a wired / wireless communication link. The present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned traffic accident prediction method based on deep learning is implemented. The computer-readable medium may be included in the device / apparatus / system described in the above-mentioned embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed, the method according to the embodiment of the present application is implemented. According to an embodiment of the present application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, optical cable, radio frequency signal, etc., or any suitable combination of the above. Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application. Therefore, the scope of the present application should not be limited to the above-described embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.
Claims
1. A traffic accident prediction method based on deep learning, It is characterized in that include: Obtain surveillance video corresponding to the target traffic scene; Extracting features from the surveillance video to obtain target image features; the target image features include RGB image global features, RGB image local features, depth image global features, and depth image local features; Using a collaborative attention mechanism to obtain correlation information between different features in the target image features, and fusing each feature in the target image features based on the correlation information to obtain a target fused feature; The target fusion feature is input into the trained GRU network model, and the predicted probability of a traffic accident corresponding to the target traffic scenario is obtained as output.
2. The traffic accident prediction method based on deep learning according to claim 1, It is characterized in that The step of extracting features from the surveillance video to obtain target image features includes: Processing the surveillance video by frame division to obtain a target image frame; The target image frame is input into the VGG-16 network for feature extraction to obtain the target image features.
3. The traffic accident prediction method based on deep learning according to claim 1, It is characterized in that The method of using the collaborative attention mechanism to obtain the correlation information between different features in the target image features, and fusing the various features in the target image features based on the correlation information to obtain the target fused features, includes: Any two features of the target image features are fused, and the fused features are fused with the remaining features one by one to obtain the target fused features.
4. The traffic accident prediction method based on deep learning according to claim 3, It is characterized in that The method of using the collaborative attention mechanism to obtain the correlation information between different features in the target image features, and fusing the various features in the target image features based on the correlation information to obtain the target fused features, includes: Fusing the RGB image global feature and the RGB image local feature of the target image frame to obtain a first target fusion feature; Fusing the first target fusion feature with the global feature of the depth map of the target image frame to obtain a second target fusion feature; The second target fusion feature is fused with the local feature of the depth map of the target image frame to obtain the target fusion feature.
5. According to the traffic accident prediction method based on deep learning as claimed in claim 1, It is characterized in that The collaborative attention mechanism is: ； in represents the computation process of the collaborative attention block, Represents a splicing operation, are the parameters of the collaborative embedding layer, and Given two features.
6. The traffic accident prediction method based on deep learning according to claim 1, It is characterized in that The target fusion feature is input into the trained GRU network model, and the predicted probability of a traffic accident corresponding to the target traffic scene is obtained as an output, including: The target fusion feature and the output information of the GRU network model at the previous moment are input into the trained GRU network model, and the prediction information is output.
7. The traffic accident prediction method based on deep learning according to claim 1, It is characterized in that Before inputting the target fusion feature into the trained GRU network model and outputting the predicted probability of a traffic accident corresponding to the target traffic scene, the method further includes: The original network model is trained using historical surveillance videos and the binary cross entropy loss function to obtain a trained GRU network model.
8. A traffic accident prediction device based on deep learning, It is characterized in that include: Video acquisition module, feature extraction module, feature fusion module and prediction output module, among which, The video acquisition module is configured to acquire the surveillance video corresponding to the target traffic scene; The feature extraction module is configured to extract features from the surveillance video to obtain target image features; the target image features include RGB image global features, RGB image local features, depth image global features, and depth image local features; The feature fusion module is configured to obtain association information between different features in the target image features by using a collaborative attention mechanism, and fuse each feature in the target image features based on the association information to obtain a target fusion feature; The prediction output module is configured to input the target fusion feature into the trained GRU network model, and output the predicted probability of a traffic accident corresponding to the target traffic scene.
9. An electronic device comprising a processor and a memory; the memory stores a computer program, in, When the computer program is executed by the processor, the computer program implements the traffic accident prediction method based on deep learning according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.