Traffic accident detection method, storage medium and device

Abstract

The present invention discloses a traffic accident detection method, storage medium and device, the method comprising the steps of: pre-processing the filtered traffic data to construct a traffic accident data set; treating each original video sample in the traffic accident data set as a packet, performing space-time domain segmentation on each packet, and obtaining a number of instances corresponding to each packet; constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on the several instances corresponding to each packet to obtain a trained accident detection model; performing end-to-end traffic accident detection on the test video according to the trained accident detection model. The method provided by the present invention can effectively improve the traffic accident detection rate and reduce the false alarm rate.

Description

Technical Field

[0001] The present invention relates to the field of traffic accident detection, and in particular to a traffic accident detection method, storage medium and equipment. Background Art

[0002] In recent years, with the substantial increase in the number of motor vehicles owned by residents, the urban traffic environment has become increasingly complex, and the incidence of traffic accidents has remained high. Traffic accidents not only cause traffic congestion, personal and public property losses, but also endanger life safety and cause serious physical and psychological trauma to people. With the development and application of video surveillance technology, many cities are currently installing more and more road surveillance cameras to monitor road traffic conditions and parameters in order to strengthen the supervision of urban traffic and deal with traffic accidents in a timely manner.

[0003] The traditional method of investigating traffic accidents is manual monitoring, that is, personnel are on duty around the clock, watching surveillance videos and discovering accidents or anomalies from them. Since traffic accidents are random and unpredictable, this method not only requires a lot of manpower, but is also affected by uncontrollable factors such as human eye resolution and fatigue. It may lead to low stability and cannot meet the efficiency and cost requirements of large-scale traffic monitoring network applications.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a traffic accident detection method, storage medium and device in view of the shortcomings of the existing technology, aiming to solve the problems that the traditional traffic monitoring accident investigation method has poor stability and cannot meet the efficiency and cost requirements of large-scale traffic monitoring network applications.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] A traffic accident detection method, comprising the steps of:

[0008] Preprocess the screened traffic data and construct a traffic accident dataset;

[0009] Each original video sample in the traffic accident data set is regarded as a packet, and each packet is segmented in the space-time domain to obtain a number of instances corresponding to each packet;

[0010] Constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each package to obtain a trained accident detection model;

[0011] End-to-end traffic accident detection is performed on the test video according to the trained accident detection model.

[0012] The traffic accident detection method, wherein the step of preprocessing the screened traffic data to construct a traffic accident data set comprises:

[0013] Get UCF Crimes traffic data and CADP traffic data;

[0014] All normal videos in traffic scenes are screened out from the UCF Crimes traffic data to form a basic positive sample set;

[0015] All accident videos in traffic scenes are screened out from the UCF Crimes traffic data, and merged with the CADP traffic data to form a basic negative sample set;

[0016] Data cleaning is performed on the basic positive sample set and the basic negative sample set to obtain corresponding positive sample sets and negative sample sets, and the positive sample sets and the negative sample sets are divided according to a predetermined ratio to obtain a training set and a test set.

[0017] The traffic accident detection method, wherein after dividing the positive sample set and the negative sample set according to a predetermined ratio to obtain a training set and a test set, further comprises:

[0018] Crop the long-duration normal videos in the positive sample set in the training set to obtain an equal number of normal videos and accident videos in the training set, wherein the normal videos in the training set include normal video-level labels, and the accident videos in the training set include accident video-level labels;

[0019] The long-duration normal videos in the positive sample set in the test set are cropped to obtain normal videos and accident videos with a quantity ratio of 3-10:1 in the test set. The normal videos in the test set include normal video-level labels, and the accident videos in the test set include accident video-level labels and the start frame and end frame of the accident.

[0020] In the traffic accident detection method, the steps of performing space-time segmentation on the original video samples in the traffic accident data set to obtain a plurality of instances include:

[0021] Each video in the traffic accident dataset is regarded as a packet, and the normal video is a negative packet, denoted as B n , the accident video is the positive package, denoted as B a ;

[0022] Each packet is evenly divided into several time-continuous but non-overlapping video segments in the time domain, the number of which is recorded as N T ;

[0023] Each video clip obtained by segmentation in the time domain is further segmented or sampled in the spatial domain to obtain a number of instances, the number of which is recorded as N s .

[0024] The traffic accident detection method, wherein the accident detection model is constructed according to the weak label attributes of the traffic accident data set, and the accident detection model is trained based on a number of instances corresponding to each packet, and the steps of obtaining the trained accident detection model include:

[0025] A 3D feature extractor is used to perform multimodal feature extraction on each instance in the package to capture the timing information and motion features related to the accident process in the video;

[0026] Traffic accident detection is regarded as a regression problem to design an accident detection model, wherein the accident detection model is a deep neural network composed of several fully connected layers;

[0027] Based on the assumptions of short duration and continuity of traffic accidents, the loss function of the accident detection model is constructed by combining multi-instance learning with the sorting idea. The expression is as follows:

[0028]

[0029] In the formula, and Respectively represent the feature vectors extracted from the instances of the i-th time period and the j-th video screen area in the accident video and the normal video, and represents the corresponding accident detection model prediction score, λ1 and λ2 are hyperparameters, and W represents the network weight parameter;

[0030] The loss function is further simplified as: L = L(B a ,B n )+||W||2, where L(B a ,B n ) is the sum of the first three terms in the loss function and is directly related to the input sample data;

[0031] During training, a certain batch of samples is selected in each iteration process. Assume that the number of samples is N bs , where normal video and accident video samples each account for half, and are randomly selected from the positive sample set and negative sample set in the training set, respectively. According to the definition of the loss function, a pair of normal video and accident video combination can be calculated to obtain a loss value, so the loss value of batch sample training can be expressed as:

[0032]

[0033] The traffic accident detection method, wherein the step of constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each package, and obtaining the trained accident detection model further comprises:

[0034] In the accident detection model training process, L i (B a ,B n )(i=1,2,K,N bs / 2) Arrange in descending order, assuming that the proportion of difficult samples is P h , before selecting The largest value L i (B a ,B n ) to calculate the final loss value;

[0035] The screened loss value is recorded as L s,i (B a ,B n ), i=1,2,K, The final loss value calculation formula is:

[0036] The traffic accident detection method, wherein the step of constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each package, and obtaining the trained accident detection model further comprises:

[0037] The accident detection model is iteratively trained using a deep learning optimizer algorithm until the loss function converges;

[0038] The C3D pre-trained model is used to directly extract the first feature on the RGB image sequence;

[0039] Use I3D's Kinetics pre-trained model to directly extract the second feature on the RGB image sequence;

[0040] The third feature is calculated based on the original image and the ImageNet pre-trained model of I3D is used to extract the third feature on the optical flow image;

[0041] The accident detection model is trained using the first feature, the second feature and the third feature respectively to obtain three corresponding trained accident detection models.

[0042] The traffic accident detection method, wherein the step of performing end-to-end traffic accident detection on a test video according to the trained accident detection model comprises:

[0043] Each test video in the test set is divided into several instances, denoted as I i,j , where i = 1, 2, KN T , represents the time domain serial number of the instance, j = 1, 2, KN S , indicating the spatial sequence number of the instance;

[0044] For each instance, the optical flow map of the image frame is calculated, and then the multimodal feature information is extracted, including: using the C3D model to extract features on the RGB image sequence, denoted as The I3D Kinetics pre-trained model is used to extract features on the RGB image sequence, denoted as The I3D ImageNet pre-trained model is used to extract features on the optical flow image, denoted as

[0045] The three types of features are input into the corresponding trained accident detection model to predict the anomaly scores, which are recorded as follows:

[0046] For each type of feature, the prediction scores of several instances obtained by spatial segmentation and enhancement in the same time period are fused as follows:

[0047]

[0048]

[0049]

[0050] The score vector of the input video in the time domain is obtained as follows:

[0051]

[0052] For S I3D_FLOW Perform Min-Max normalization processing, the formula is:

[0053]

[0054] In each time period, the scores of multi-modal features are combined by soft voting, and the formula is: in, is a sequence, representing the final anomaly score of the input video in each time period. When the anomaly score exceeds a certain threshold S Th When , it is determined that a traffic accident occurred during this period of time.

[0055] A storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the traffic accident detection method of the present invention.

[0056] A traffic accident detection device includes a processor suitable for implementing various instructions; and a storage medium suitable for storing multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the steps in the traffic accident detection method described in the present invention.

[0057] Beneficial effects: Compared with the prior art, the traffic accident detection method proposed in the present invention has the following advantages: based on the existing large-scale, systematic and widely used traffic data set, the data is screened, cleaned and recombined to construct a new traffic accident data set, eliminating data interference factors such as scene conversion and non-monitoring images, which is conducive to the algorithm learning the real accident feature information; a multi-instance learning method is used to establish an accident detection model, the original video is segmented in time and space, and data enhancement is performed to obtain several instances, so that finer-grained and more accurate features can be obtained for learning; an online difficult sample learning mechanism is added during training, focusing on The invention is used to learn samples that are more difficult to detect and identify, thereby improving the algorithm's ability to identify normal videos and accident videos; multiple 3D feature extractors are used to extract multimodal visual feature information in the RGB image domain and the optical flow domain respectively, and then the multi-model and multimodal feature prediction scores are integrated, which can effectively improve the traffic accident detection performance, increase the traffic accident detection rate, and reduce the false alarm rate; when a traffic accident occurs, the traffic accident detection method provided by the invention can automatically, quickly and accurately detect it from the traffic monitoring video, thereby sending an alarm signal in time after the accident occurs, activating the accident response plan, better carrying out rescue and dredging work, and reducing the loss of people's lives and property. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The figure is a flow chart of a preferred embodiment of a traffic accident detection method of the present invention.

[0059] Figure 2 This is a schematic diagram of the present invention performing space-time domain segmentation on each packet.

[0060] Figure 3 The present invention is a flow chart of performing end-to-end traffic accident detection on a test video according to the trained accident detection model.

[0061] Figure 4 The figure is a schematic diagram of a traffic accident detection device according to the present invention. DETAILED DESCRIPTION

[0062] The present invention provides a traffic accident detection method, storage medium and device. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0064] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0065] The invention is further described below by describing embodiments in conjunction with the accompanying drawings.

[0066] The traditional method of investigating traffic accidents is manual monitoring, that is, personnel are on duty around the clock, watching surveillance videos and discovering accidents or anomalies from them. Since traffic accidents are random and unpredictable, this method not only requires a lot of manpower, but is also affected by uncontrollable factors such as human eye resolution and fatigue. It may lead to low stability and cannot meet the efficiency and cost requirements of large-scale traffic monitoring network applications.

[0067] Therefore, when a traffic accident occurs, how to automatically, quickly and accurately detect it from the traffic monitoring video, so as to send out an alarm signal in time after the accident, activate the accident response plan, better carry out rescue and dredging work, and reduce the loss of life and property of the people, is a key problem to be solved in the field of traffic monitoring. With the development of computer vision technology and artificial intelligence technology, intelligent transportation systems have gradually emerged, applying intelligent video analysis technology, historical data, etc. to the establishment of accident detection models and parameter fitting, providing an intelligent solution for traffic accident detection. One method is to use supervised learning, including non-end-to-end and end-to-end processing methods. The non-end-to-end method is to connect a series of steps such as background subtraction, detection, and tracking in series, extract manually designed features, and detect accidents based on a pre-defined set of rules. This method requires the use of target bounding box annotation information to supervise the training of the target detection network. When the scene changes, there is also the problem of invalidation of rules or judgment conditions. The end-to-end method usually establishes a deep learning model to extract more refined semantic features, and finally regresses to obtain anomaly scores. The data used in this method is generally short videos of a few seconds in length, or there is precise label information (for example, frame level) about the time period of the accident in the video to complete model training so that it has the ability to identify and locate accidents. This method often requires a large amount of training data and consumes a lot of manpower to perform data preprocessing such as video annotation and cropping. There is currently no large-scale dataset available for traffic accident detection.

[0068] Another method is to use unsupervised learning to reconstruct the image frame based on the autoencoder. This method uses the encoder to represent the features of the image frame, and then uses the decoder to reconstruct the image frame and calculate the reconstruction error. When the reconstruction error is greater than the threshold, it is judged as abnormal. This type of method generally models normal behavior patterns. There are only normal videos in the training set, and no labeling is required. However, it is sensitive to changes in the video screen. When the test video has sudden changes in lighting, large foreground targets move quickly, etc., it may cause large reconstruction errors. In addition, since it is difficult to cover all normal behaviors in the training set, some atypical normal behaviors may also be misjudged as abnormal accidents.

[0069] In fact, there are often periods of normal traffic in the original traffic accident videos, which makes the video clips naturally become weakly labeled data, that is, their coarse-grained label information is known. In order to fully mine and utilize this data, weakly supervised learning is a very effective way.

[0070] Based on this, the present invention provides a traffic accident detection method, such as Figure 1 As shown, it includes the steps of:

[0071] S10, preprocessing the screened traffic data to construct a traffic accident data set;

[0072] S20, treating each original video sample in the traffic accident data set as a packet, performing space-time segmentation on each packet, and obtaining a number of instances corresponding to each packet;

[0073] S30, constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each package to obtain a trained accident detection model;

[0074] S40: Perform end-to-end traffic accident detection on the test video according to the trained accident detection model.

[0075] This embodiment provides a traffic accident detection method based on weakly supervised learning. Based on an in-depth analysis and characterization of the characteristics and scope of traffic accidents, combined with the current data status, an accident detection model is designed in a targeted manner. This method is suitable for cross-scenario applications and can improve the traffic accident detection rate. Specifically, this embodiment is based on the existing large-scale, systematic and widely used traffic data set, and screens, cleans and recombines the data to construct a new traffic accident data set, eliminating data interference factors such as scene conversion and non-monitoring images, which is conducive to the algorithm learning the real accident feature information; the original video is segmented in time and space dimensions, and data enhancement is performed to obtain several instances, so that finer-grained and more accurate features can be obtained for learning; a multi-instance learning method is used to establish an accident detection model, and an online difficult sample learning mechanism is added, focusing on learning samples that are more difficult to detect and identify, and improving the algorithm's ability to identify normal videos and accident videos; multiple 3D feature extractors are used to extract multimodal visual feature information in the RGB image domain and the optical flow domain respectively, and then the multi-model and multimodal feature prediction scores are integrated, which can effectively improve the traffic accident detection performance, that is, improve the traffic accident detection rate and reduce the false alarm rate.

[0076] In some embodiments, the scope of traffic accidents is defined and feature analyzed, video data is screened and processed, and a traffic accident data set is constructed. Specifically, since there is no public and authoritative traffic accident data set in the academic community for research in the field of traffic accident detection, this embodiment first needs to construct a data set for traffic accident detection. The traffic accident referred to in this embodiment refers to a special type of traffic abnormal event that occurs instantly, violently, and has strong visual significance. It is relatively easy to judge by the apparent characteristics such as the motion characteristics or appearance changes of the accident subject, such as the sudden stop of the vehicle, the sudden change of the driving direction, the deformation of the vehicle body, etc. Traffic accidents have scene decoupling. Traffic videos or surveillance videos in the real world usually come from cameras deployed in different locations, and there are great differences in scene, angle, lighting and other picture factors. Due to the sporadic and low frequency of traffic accidents, it is relatively difficult to obtain normal videos and accident videos in the same scene. At the same time, considering the feature significance and scene decoupling characteristics of traffic accidents, it is possible for the algorithm to learn deep semantic information that is not related to the scene, thereby ensuring a certain cross-scene generalization ability. Therefore, this embodiment uses videos from different domains to construct a traffic accident data set.

[0077] In this embodiment, the traffic accident dataset is used to train the accident detection model established based on deep learning theory. In order to ensure that the model can learn truly valuable accident feature information and enhance the scene adaptability of the model, the quality and completeness of the dataset should be ensured as much as possible. Therefore, the construction criteria of the traffic accident dataset include: first, the sample size in the dataset must be large enough; second, the scenes, lighting conditions, and accident types covered by the samples must be rich enough; third, the sample quality must be high enough, including video clarity, the proportion of the accident area to the screen, etc., and try to remove some interference factors that may affect the learning of accident features.

[0078] Based on the above criteria, this embodiment comprehensively utilizes the currently public, large-scale and widely used UCF Crimes traffic data and CADP traffic data to construct a traffic accident dataset. Among them, UCF Crimes traffic data is an abnormal behavior detection dataset containing 1900 videos and rich scenes, including surveillance videos and traffic videos, with a total of 13 types of abnormal events, of which traffic accidents are one; CADP traffic data is a traffic accident dataset, containing 1416 videos of various types of traffic accidents in multiple scenes.

[0079] In some embodiments, normal videos in all traffic scenes are screened out from the UCF Crimes traffic data to form a basic positive sample set; accident videos in all traffic scenes are screened out from the UCF Crimes traffic data and merged with the CADP traffic data to form a basic negative sample set. In this embodiment, since many videos in the UCF Crimes traffic data have data quality problems such as lens movement, switching, zooming, fast forwarding, and non-monitoring images, the basic positive sample set and the basic negative sample set obtained in the previous step are cleaned, and the parts with scene conversion, zooming, non-monitoring images, and image quality problems are deleted to obtain the cleaned positive sample set and negative sample set, and then the cleaned positive sample set and negative sample set are divided according to a predetermined ratio to obtain a training set and a test set, and a traffic accident data set is constructed based on the training set and the test set.

[0080] In some specific implementations, a normal video with a long duration is cut into several short videos, for example, 1 minute in length, and the video-level labels are retained as samples for final use. In this embodiment, the number of normal videos and the number of accident videos in the training set in the traffic accident data set constructed are substantially equal, each with 1500 segments. The ratio of the number of normal videos and the number of accident videos in the test set is about 5:1, 500 segments and 100 segments respectively, which not only avoids a large amount of fine annotation work, but also reflects to a certain extent the fact that normal traffic conditions in the real world are far more than accidents. In the video data in the training set, each video has only one normal or abnormal video-level label, that is, the normal videos in the training set include normal video-level labels, and the accident videos in the training set include accident video-level labels; and in addition to the video-level labels, each video in the test set is also labeled with the start frame and end frame of the traffic accident, which is used for frame-level performance evaluation during testing, that is, the normal videos in the test set include normal video-level labels, and the accident videos in the test set include accident video-level labels and the start frame and end frame of the accident.

[0081] In some implementations, the original video samples in the traffic accident dataset are segmented in the space-time domain to obtain several instances. Specifically, Figure 2 As shown in the figure, each video in the traffic accident dataset is regarded as a packet, and the normal video is a negative packet, denoted as B n , the accident video containing the traffic accident is the positive package, denoted as B a ;Evenly divide each packet into several temporally continuous but non-overlapping video segments (also called instances) in the time domain, the number of which is recorded as N T; Each video segment obtained by segmentation in the time domain is further segmented or sampled in the spatial domain to obtain several instances. The specific method is to cut out five video segments of size w×h (width*height) with the four corners of the video screen as base points and the center of the video screen as the center. There can be overlapping areas between the video segments. Then, data enhancement is performed and a mirror operation is performed on the above five instances to offset the impact of uneven screen distribution.

[0082] After each video clip segmented in the time domain is segmented and enhanced in the spatial domain, 10 new instances are finally obtained. The number of spatial domain instances is recorded as N S It should be noted that, based on the spatial-temporal segmentation concept of the present invention, other spatial segmentation methods or data enhancement methods can also be used according to the situation to obtain different numbers of instances. After the original video is segmented in the spatial-temporal domain, the instances obtained by the positive packet are recorded as The instance obtained by the negative bag is recorded as Where i = 1, 2, KN T , represents the time domain serial number of the instance, j = 1, 2, KN S , indicating the spatial sequence number of the instance.

[0083] In some embodiments, an accident detection regression model based on multi-instance learning is established, and a loss function is designed by combining the sorting idea and the assumptions of the short-term and continuous type of traffic accidents. Specifically, a 3D feature extractor is used to perform a multimodal feature extraction operation on each instance in the package to better capture the timing information, motion features, etc. related to the accident process in the video. The feature extractor includes but is not limited to C3D, Inception I3D, Resnet I3D, etc. The present invention uses C3D and Inception I3D to extract multimodal features. Taking the I3D network as an example, the extracted feature vector is used as the input data of the accident detection model; the accident detection model is designed by regarding traffic accident detection as a regression problem, which is a deep neural network composed of several fully connected layers. The network structure adopted in this embodiment is specifically as follows: input layer-fully connected layer 1-dropout1-fully connected layer 2-dropout2-fully connected layer 3, the activation function of fully connected layer 1 is relu, the activation function of fully connected layer 3 is sigmoid, the number of neurons in the three fully connected layers is 512, 32, and 1 respectively, the output of fully connected layer 3 is the output result of the network, which represents the abnormality score of the instance, and the value range is 0 to 1. The higher the score, the greater the possibility of an accident. The number of neurons in the input layer is equal to the dimension of the feature vector extracted in the previous step. For C3D features, the number of input layer nodes is 4096, and for I3D features, the number of input layer nodes is 1024.

[0084] In some implementations, based on the assumptions of the short duration and continuity of traffic accidents, the loss function of the accident detection model is constructed by combining multi-instance learning with the idea of ​​sorting, and the expression is as follows:

[0085]

[0086] In the formula, and Respectively represent the feature vectors extracted from the instances of the i-th time period and the j-th video screen area in the accident video package and the normal video package, and Represents the corresponding accident detection model prediction score, λ1 and λ2 are hyperparameters, and W represents the network weight parameter. The first term of the loss function is to maximize the interval loss, which is used to minimize the highest score in normal videos and maximize the highest score in accident videos, so as to detect accident periods as much as possible and reduce the false alarm rate. Taking the maximum score of instances in different time periods implies a sorting process; the second and third terms are constraints added based on the assumptions of the temporal continuity of abnormal scores and the short-term nature of accidents, and the fourth term is a regularization term. The loss function is further simplified as:

[0087] L=L(B a ,B n )+||W||2

[0088] In the formula, L(B a ,B n ) is the sum of the first three items in the loss function, which is directly related to the input sample data; during training, a certain batch of samples is selected in each iteration process. Assume that the number of samples is N bs , where normal video and accident video samples each account for half, and are randomly selected from the positive sample set and negative sample set in the training set, respectively. According to the definition of the loss function, a pair of normal video and accident video combination can be calculated to obtain a loss value, so the loss value of batch sample training can be expressed as:

[0089]

[0090] In some implementations, during the model training process, an online difficult sample learning mechanism is added to focus on learning samples that are difficult to identify, thereby improving the algorithm's ability to distinguish between normal and accidental situations. i (B a ,B n )(i=1,2,K,N bs / 2) Arrange in descending order, assuming that the proportion of difficult samples is P h , before selecting The largest value L i (B a ,Bn ) to calculate the final loss value;

[0091] The screened loss value is recorded as L s,i (B a ,B n ), i=1,2,K, The final loss value calculation formula is:

[0092]

[0093] In some embodiments, the accident detection model is trained using multimodal features extracted using multiple 3D feature extractors in the RGB image domain and the optical flow domain of the instance, respectively, to obtain multiple accident detection models. Specifically, multimodal feature vectors are extracted from the video segments (instances) obtained by space-time segmentation of the original video samples for accident detection model training; the deep learning optimizer algorithm used in accident detection model training can adopt stochastic gradient descent method, momentum optimization method, Adam algorithm, AdaDelta algorithm, etc. The present invention takes the AdaDelta algorithm as an example and uses the back propagation algorithm to iteratively train the accident detection model until the loss function converges. The training data used by the accident detection model is a multimodal feature vector extracted from a video clip (instance) obtained by space-time segmentation of the original video sample. The present invention uses three types of feature data. One is the features directly extracted from the RGB image sequence using the C3D pre-trained model. The second is the features directly extracted from the RGB image sequence using the I3D Kinetics pre-trained model. The third is the features extracted from the optical flow image calculated from the original image using the I3D ImageNet pre-trained model. The three accident detection models are trained separately using the three types of feature data.

[0094] In some embodiments, Figure 3 As shown, the step of performing end-to-end traffic accident detection on the test video includes: dividing the test video into several instances, extracting multimodal feature information of all instances; for the features of each modality, predicting anomaly scores according to the corresponding accident detection model, fusing the anomaly scores of all instances obtained by spatial segmentation and data enhancement in the same time period, and normalizing the optical flow prediction scores of all time periods of a video; for each time period, using the averaging method to fuse the multimodal feature prediction scores to obtain a final score, which is then compared with a pre-set threshold to determine whether a traffic accident has occurred.

[0095] Specifically, a video is used as the test input, and the test video is divided into several instances according to the space-time domain segmentation and data enhancement method described above, denoted as I i,j , where i = 1, 2, KN T, represents the time domain serial number of the instance, j = 1, 2, KN S , indicating the spatial sequence number of the instance;

[0096] For each instance, the optical flow map of the image frame is calculated, and then the multimodal feature information is extracted, including: using the C3D model to extract features on the RGB image sequence, denoted as The I3D Kinetics pre-trained model is used to extract features on the RGB image sequence, denoted as The I3D ImageNet pre-trained model is used to extract features on the optical flow image, denoted as

[0097] The three types of features obtained in the previous step are input into the corresponding trained accident detection model to predict the anomaly scores, which are recorded as follows:

[0098] For each type of feature, the prediction scores of several instances obtained by spatial segmentation and enhancement in the same time period are fused as follows:

[0099]

[0100]

[0101]

[0102] The score vector of the input video in the time domain is obtained as follows:

[0103]

[0104] In order to avoid the problem that the I3D optical flow feature prediction score distribution is too concentrated and its score baseline is inconsistent with the RGB feature, S I3D_FLOW Perform Min-Max normalization processing, the formula is:

[0105]

[0106] In each time period (an instance of time domain segmentation), the scores of multi-modal features are fused by soft voting, and the formula is:

[0107]

[0108] in, is a sequence, representing the final anomaly score of the input video in each time period. When the anomaly score exceeds a certain threshold S Th, it is considered that a traffic accident is likely to have occurred during that period. The videos in the test set can be batch tested according to the above process to evaluate the performance indicators of the algorithm, such as AUC, detection rate, false alarm rate, etc. It should be noted that in real-time processing mode, the above process can also be executed to predict the anomaly score.

[0109] In this embodiment, a video is regarded as a packet, the accident video is a positive packet, and the normal video is a negative packet. Each packet is divided into several instances (video clips) according to the two dimensions of time and space; for each instance, the C3D network and the I3D network are used to extract the feature information of the RGB image respectively, and the feature information of the optical flow image is extracted by the I3D network, and the three types of feature information are respectively input into the accident detection network in the form of a multi-layer perceptron to obtain anomaly scores; then, for each type of feature, the scores of several instances obtained by spatial segmentation of the same time period are averaged to obtain the anomaly score within this time period, and the anomaly scores of multiple time periods (corresponding to multiple instances in the time domain) constitute a vector, and then the I3D optical flow feature anomaly score vector is normalized to the maximum and minimum; finally, in each time period, the scores of the three types of features are merged by soft voting. When the anomaly score exceeds the threshold, it can be considered that a traffic accident has occurred during this period. Due to the different information and dimensions represented by the features, during training, C3D RGB features, I3D RGB features and I3D optical flow features are used to train three independent accident detection networks respectively to obtain three accident detection models. The accident detection models are trained using multi-instance learning, and the supervision information is the video-level label, that is, whether an accident occurs in the video. In addition, an online learning mechanism for difficult samples is added to maximize the ability to distinguish traffic accidents.

[0110] In some embodiments, a storage medium is also provided, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the traffic accident detection method of the present invention.

[0111] In some embodiments, a traffic accident detection device is also provided, such as Figure 4 As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0112] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0113] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0114] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0115] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0116] In summary, the present invention screens, cleans, and recombines data to construct a new traffic accident data set, excludes data interference factors such as scene conversion and non-monitoring images, and is conducive to the algorithm learning real accident feature information; a multi-instance learning method is used to establish an accident detection model, the original video is segmented in time and space dimensions, and data enhancement is performed to obtain several instances, so that finer-grained and more accurate features can be obtained for learning; the present invention adds an online difficult sample learning mechanism to the training, focusing on learning samples that are more difficult to detect and identify, and improving the algorithm's ability to recognize normal videos and accident videos; multiple 3D feature extractors (such as C3D, Inception I3D) are used to extract multimodal visual feature information in the RGB image domain and the optical flow domain respectively, and then the multi-model and multimodal feature prediction scores are integrated, which can effectively improve the algorithm performance, that is, improve the accident detection rate and reduce the false alarm rate.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic accident detection method, characterized in that: Includes steps: Define the scope, analyze the characteristics and filter the traffic data, pre-process the filtered traffic data and build a traffic accident data set; Each video in the traffic accident dataset is regarded as a packet, and the normal video is a negative packet, denoted as B n , the accident video is the positive package, denoted as B a ; Each packet is evenly divided into several time-continuous but non-overlapping video segments in the time domain, the number of which is recorded as N T ; Each video clip obtained by segmentation in the time domain is further segmented or sampled in the spatial domain to obtain a number of instances, the number of which is recorded as N s ; Constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each packet to obtain a trained accident detection model; Performing end-to-end traffic accident detection on the test video according to the trained accident detection model; The steps of constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each package to obtain the trained accident detection model include: A 3D feature extractor is used to perform multimodal feature extraction on each instance in the package to capture the timing information and motion features related to the accident process in the video; Traffic accident detection is regarded as a regression problem to design an accident detection model, wherein the accident detection model is a deep neural network composed of several fully connected layers; Based on the assumptions of short duration and continuity of traffic accidents, the loss function of the accident detection model is constructed by combining multi-instance learning with the sorting idea. The expression is as follows: In the formula, and Respectively represent the feature vectors extracted from the instances of the i-th time period and the j-th video screen area in the accident video and the normal video, and represents the corresponding accident detection model prediction score, λ1 and λ2 are hyperparameters, and W represents the network weight parameter; The loss function is further simplified as: L = L(B a ,B n )+||W||2, where L(B a ,B n ) is the sum of the first three terms in the loss function and is directly related to the input sample data; During training, a certain batch of samples is selected in each iteration process. Assume that the number of samples is N bs , where normal video and accident video samples each account for half, and are randomly selected from the positive sample set and negative sample set in the training set, respectively. According to the definition of the loss function, a pair of normal video and accident video combination can be calculated to obtain a loss value, so the loss value of batch sample training can be expressed as:

2. The traffic accident detection method according to claim 1, characterized in that: The steps of defining the scope, analyzing the characteristics and screening the traffic data, preprocessing the screened traffic data and constructing the traffic accident data set include: Get UCF Crimes traffic data and CADP traffic data; All normal videos in traffic scenes are screened out from the UCF Crimes traffic data to form a basic positive sample set; All accident videos in traffic scenes are screened out from the UCF Crimes traffic data, and merged with the CADP traffic data to form a basic negative sample set; Data cleaning is performed on the basic positive sample set and the basic negative sample set to obtain corresponding positive sample sets and negative sample sets, and the positive sample sets and the negative sample sets are divided according to a predetermined ratio to obtain a training set and a test set.

3. The traffic accident detection method according to claim 2, characterized in that: After dividing the positive sample set and the negative sample set according to a predetermined ratio to obtain a training set and a test set, the following step further comprises: Crop the long-duration normal videos in the positive sample set in the training set to obtain an equal number of normal videos and accident videos in the training set, wherein the normal videos in the training set include normal video-level labels, and the accident videos in the training set include accident video-level labels; The long-duration normal videos in the positive sample set in the test set are cropped to obtain normal videos and accident videos with a quantity ratio of 3-10:1 in the test set. The normal videos in the test set include normal video-level labels, and the accident videos in the test set include accident video-level labels and the start frame and end frame of the accident.

4. The traffic accident detection method according to claim 1, characterized in that: The step of constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each package to obtain a trained accident detection model also includes: In the accident detection model training process, L i (B a ,B n )(i=1,2,K,N bs / 2) Arrange in descending order, assuming that the proportion of difficult samples is P h , before selecting The largest value L i (B a ,B n ) to calculate the final loss value; The screened loss value is recorded as The final loss value calculation formula is:

5. The traffic accident detection method according to claim 4, characterized in that: The step of constructing an accident detection model according to the weak label attributes of the traffic accident data set, and training the accident detection model based on a number of instances corresponding to each package to obtain a trained accident detection model also includes: The accident detection model is iteratively trained using a deep learning optimizer algorithm until the loss function converges; The C3D pre-trained model is used to directly extract the first feature on the RGB image sequence; The I3D Kinetics pre-trained model is used to directly extract the second feature on the RGB image sequence; The third feature is calculated based on the original image and the ImageNet pre-trained model of I3D is used to extract the third feature on the optical flow image; The accident detection model is trained using the first feature, the second feature and the third feature respectively to obtain three corresponding trained accident detection models.

6. The traffic accident detection method according to claim 5, characterized in that: The step of performing end-to-end traffic accident detection on the test video according to the trained accident detection model comprises: Each test video in the test set is divided into several instances, denoted as I i,j , where i = 1, 2, KN T , represents the time domain serial number of the instance, j = 1, 2, KN S , indicating the spatial sequence number of the instance; For each instance, the optical flow map of the image frame is calculated, and then the multimodal feature information is extracted, including: using the C3D model to extract features on the RGB image sequence, denoted as The I3DKinetics pre-trained model is used to extract features on the RGB image sequence, denoted as The I3DImageNet pre-trained model is used to extract features on the optical flow image, denoted as The three types of features are input into the corresponding trained accident detection model to predict the anomaly scores, which are recorded as follows: For each type of feature, the prediction scores of several instances obtained by spatial segmentation and enhancement in the same time period are fused as follows: The score vector of the input video in the time domain is obtained as follows: For S I3D_FLOW Perform Min-Max normalization processing, the formula is: In each time period, the scores of multi-modal features are combined by soft voting, and the formula is: in, is a sequence, representing the final anomaly score of the input video in each time period. When the anomaly score exceeds a certain threshold S Th When , it is determined that a traffic accident occurred during this period of time.

7. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the traffic accident detection method as described in any one of claims 1-6.

8. A traffic accident detection device, characterized in that: It comprises a processor, which is suitable for implementing each instruction; and a storage medium, which is suitable for storing multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the steps in the traffic accident detection method according to any one of claims 1-6.

Images