Traffic accident detection method and system based on multi-modal video classification model
By adopting multimodal video classification model and deep learning technology in traffic accident detection, the existing methods are solved in the problem of insufficient robustness and speed in complex traffic environments, and efficient and real-time traffic accident detection is achieved.
Patent Information
- Application Number
- CN202411838581.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-03-21
AI Technical Summary
The existing traffic accident detection methods are not robust enough in complex traffic environments and are not running fast enough, making it difficult to effectively detect traffic accidents in real time.
The traffic accident detection method based on the multimodal video classification model is adopted, and accident detection is carried out through vehicle detection and tracking, parking judgment, clustering and video classification steps. Deep learning and multimodal big model VideoCLIP are used.
It improves the robustness and operating speed of traffic accident detection, can accurately detect traffic accidents in complex traffic environments, and reduces the pressure on subsequent video classification network calculations.
Smart Images
Figure CN119672653A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of traffic management and image processing, and in particular relates to a traffic accident detection method and system based on a multimodal video classification model. Background Art
[0002] Traffic accident detection is one of the most important parts of intelligent transportation systems. Real-time and robust traffic accident detection methods can make great contributions to reducing casualties and property losses. With the rapid development of intelligent transportation systems, the research on traffic accident detection systems based on computer vision technology and image processing technology has attracted widespread attention, and many researchers have also made significant progress in this field. However, due to the complexity of the traffic environment, some of the methods currently proposed still have certain limitations in practical applications.
[0003] In order to detect traffic accidents quickly and accurately and achieve the purpose of practical application, some challenging problems need to be solved. That is, the traffic accident detection algorithm must meet the following three requirements: 1) be able to handle complex traffic environments, including poor weather, different lighting conditions, complex road conditions and different traffic participants; 2) be able to handle different traffic flow states, that is, different degrees of traffic congestion; 3) be able to run in real time.
[0004] Existing traffic accident detection algorithms either do not run fast enough or are not robust enough in complex traffic environments. Specifically, they have the following defects or deficiencies: 1) They still use traditional logical judgment methods, such as judging whether an accident has occurred by whether there are people around the vehicle, which is not very robust; 2) They directly use video classification methods. Directly using video classification will introduce too much irrelevant background information, making it difficult to focus on the area where the accident occurred, and video classification networks often have poor generalization due to their greater demand for training data.
[0005] Therefore, there is an urgent need for a traffic accident detection method and system with fast running speed and good robustness. Summary of the invention
[0006] In view of this, the purpose of the present invention is to provide a traffic accident detection method and system based on a multimodal video classification model. The present invention aims to solve the problem that the existing traffic accident detection method and system have insufficient running speed and poor robustness.
[0007] The present invention provides a traffic accident detection method based on a multimodal video classification model, comprising the following steps:
[0008] S1. Vehicle detection and tracking;
[0009] Based on traffic monitoring video images, vehicle detection is performed on each input frame in time sequence, and the detected vehicles are tracked, that is, the vehicle position and tracking ID in each frame are output;
[0010] S2. Based on the vehicle information detected and tracked in step S1, find the parked vehicle in the current frame;
[0011] If there is a parked vehicle in the current frame, the process proceeds to step S3; if there is no parked vehicle in the current frame, the process stops detecting and deems that there is no traffic accident.
[0012] S3. Clustering and outputting parking areas based on the parking vehicles output in step S2;
[0013] S4. Based on the parking area output in step S4, extract the parking area ROI image sequence, that is, extract the image sequence corresponding to the parking area from the cached traffic monitoring video frame;
[0014] S5. Based on the image ROI sequence output in step S5, use the multimodal large model VideoCLIP to perform video classification to determine whether an accident has occurred;
[0015] If an accident occurs, an alarm is sounded; if no accident occurs, subsequent image frames are analyzed and processed according to the time sequence.
[0016] Furthermore, in step S1, vehicle detection adopts deep learning detection network yolo series or Faster-rcnn series.
[0017] Furthermore, in step S1, the vehicle tracking adopts a sort / deep sort multi-target tracking method.
[0018] Furthermore, in step S2, the method for determining whether there is a parked vehicle in the current frame is: if the intersection over union (IOU) ratio of the target frames of the same ID vehicle in the two frames before and after within 10 seconds is greater than 0.95, then the ID vehicle is determined to be parked; otherwise, it is determined to be not parked.
[0019] Furthermore, the specific content of step S3 is: based on the parked vehicles output in step S2, clustering is performed according to the size of the parked vehicles and the distance between the vehicles, clustering adjacent parked vehicles together, and outputting the parking area.
[0020] The adjacent vehicle clustering method can use the Kmeans algorithm. K-means is a classic distance-based clustering algorithm that divides data points into K clusters through iterative calculations, so that the sum of the distances from each data point to the center of its cluster is minimized, thereby clustering adjacent parked vehicles into one category.
[0021] Furthermore, in step S5, the multimodal large model used is VideoCLIP.
[0022] VideoCLIP is a contrastive learning method for pre-training zero-shot video and text understanding models that does not require any labels for downstream tasks. VideoCLIP trains Transformers for videos and text by comparing temporally overlapping positive video-text pairs and negative sample pairs retrieved by nearest neighbors, which can be used for action recognition tasks.
[0023] The present invention also provides a traffic accident detection system based on a multimodal video classification model, comprising a video stream acquisition unit, a traffic accident detection unit, an alarm unit and an event retrieval and query unit;
[0024] The video stream acquisition unit is used to collect and store traffic monitoring video images and cache video frames;
[0025] The traffic accident detection unit is used to detect traffic accidents;
[0026] The traffic accident detection unit detects and tracks vehicles based on the images obtained by the video stream unit, uses the tracking information to find parked vehicles, clusters parked vehicles, and then extracts the parking area ROI image sequence from the cache frame, and then uses the multimodal large model VideoCLIP to perform video classification to determine whether an accident has occurred;
[0027] The alarm unit is used to promptly alarm when the traffic accident detection unit detects a traffic accident;
[0028] The event retrieval and query unit is used for quickly retrieving and querying the traffic accidents that have occurred.
[0029] Beneficial effects:
[0030] 1. The present invention provides a traffic accident detection method based on a multimodal video classification model. The first step is to determine whether there is a potential accident by stopping the vehicle. This has two advantages: 1). Accurately locate the potential accident area; 2). Reduce the calculation pressure of the subsequent video classification network, because under normal traffic conditions, there are no parked vehicles, so there is no need for subsequent video classification calculations.
[0031] 2. The present invention uses a multimodal video classification model (VideoCLIP) to determine whether an accident occurs. Compared with the traditional logical judgment method, it has better robustness. Compared with the single-modal video classification method, it can utilize the prior of text information to alleviate the problem of poor model generalization caused by insufficient video training samples.
[0032] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flow chart of a traffic accident detection method based on a multimodal video classification model of the present invention;
[0034] Figure 2 It is a component unit of a traffic accident detection system based on a multimodal video classification model of the present invention. DETAILED DESCRIPTION
[0035] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application.
[0036] like Figure 1 As shown, the present invention provides a traffic accident detection method based on a multimodal video classification model, comprising the following steps:
[0037] S1. Vehicle detection and tracking;
[0038] Based on traffic monitoring video images, vehicle detection is performed on each input frame in time sequence, and the detected vehicles are tracked, that is, the vehicle position and tracking ID in each frame are output;
[0039] Vehicle detection can use the common deep learning detection network yolo series or Faster-rcnn series;
[0040] Vehicle tracking can adopt the common sort / deep sort multi-target tracking method;
[0041] S2. Based on the vehicle tracking ID detected and tracked in step S1, find the parked vehicle in the current frame;
[0042] If there is a parked vehicle in the current frame, the process proceeds to step S3; if there is no parked vehicle in the current frame, the process stops detecting and deems that there is no traffic accident.
[0043] The method for determining whether there is a parked vehicle in the current frame is as follows: if the IOU (intersection over union) of the target frames of the same ID vehicle in the two frames before and after is greater than 0.95 (this value can be flexibly adjusted according to the specific scenario) within 10 seconds (this value can be flexibly adjusted according to the specific scenario), then the ID vehicle is judged to be parked; otherwise, it is judged to be not parked.
[0044] S3. Clustering and outputting parking areas based on the parking vehicles output in step S2;
[0045] Based on the parked vehicles output in step S2, clustering is performed according to the size of the parked vehicles and the distance between the vehicles, adjacent parked vehicles are clustered together, and a parking area (there may be multiple parking areas) is output; wherein the adjacent vehicle clustering method may use the Kmeans algorithm, which is a classic distance-based clustering algorithm. It divides data points into K clusters through iterative calculation, so that the sum of the distances from each data point to the center of its cluster is minimized, thereby clustering adjacent parked vehicles into one category.
[0046] S4. Based on the parking area output in step S4, extract the parking area ROI image sequence, that is, extract the image sequence corresponding to the parking area from the cached traffic monitoring video frame (such as 8 frames, 16 frames, depending on the video classification network used in step S5);
[0047] S5. Based on the parking area ROI image sequence output in step S5, use a large multimodal model (such as VideoCLIP) to perform video classification to determine whether an accident has occurred; VideoCLIP is a comparative learning method for pre-training zero-shot video and text understanding models that does not require any labels for downstream tasks. VideoCLIP trains the Transformer of video and text by comparing temporally overlapping positive video-text pairs and negative sample pairs retrieved by nearest neighbors, which can be used for behavior recognition tasks.
[0048] If an accident occurs, an alarm is sounded; if no accident occurs, subsequent image frames are analyzed and processed according to the time sequence.
[0049] like Figure 2 As shown, the present invention also provides a traffic accident detection system based on a multimodal video classification model, including a video stream acquisition unit, a traffic accident detection unit, an alarm unit and an event retrieval and query unit;
[0050] The video stream acquisition unit is used to collect and store traffic monitoring video images and cache video frames;
[0051] The traffic accident detection unit is used to detect traffic accidents;
[0052] The traffic accident detection unit detects and tracks vehicles based on the images obtained by the video stream unit, uses the tracking information to find parked vehicles, clusters parked vehicles, and then extracts the parking area ROI image sequence from the cache frame. After that, the multimodal large model VideoCLIP is used to perform video classification to determine whether an accident has occurred.
[0053] The alarm unit is used to promptly alarm when the traffic accident detection unit detects a traffic accident;
[0054] The event retrieval and query unit is used to quickly retrieve and query traffic accidents that have occurred.
[0055] It is hereby declared that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be included in the scope of the claims of the present invention.
Claims
1. A traffic accident detection method based on a multimodal video classification model, characterized in that: The following steps are involved: S1. Vehicle detection and tracking; Based on traffic monitoring video images, vehicle detection is performed on each input frame in time sequence, and the detected vehicles are tracked, that is, the vehicle position and tracking ID in each frame are output; S2. Based on the vehicle information detected and tracked in step S1, find the parked vehicle in the current frame; If there is a parked vehicle in the current frame, the process proceeds to step S3; if there is no parked vehicle in the current frame, the process stops detecting and deems that there is no traffic accident. S3. Clustering and outputting parking areas based on the parking vehicles output in step S2; S4. Based on the parking area output in step S4, extract the parking area ROI image sequence, that is, extract the image sequence corresponding to the parking area from the cached traffic monitoring video frame; S5. Based on the image ROI sequence output in step S5, use the multimodal large model VideoCLIP to perform video classification to determine whether an accident has occurred; If an accident occurs, call the police; If no accident occurs, the subsequent image frames are analyzed and processed in time sequence.
2. The traffic accident detection method based on a multimodal video classification model according to claim 1, characterized in that: In step S1, vehicle detection uses a deep learning detection network YOLO series or Faster-RCNN series.
3. The traffic accident detection method based on a multimodal video classification model according to claim 2, characterized in that: In step S1, the vehicle tracking adopts the sort / deep sort multi-target tracking method.
4. The traffic accident detection method based on a multimodal video classification model according to claim 3 is characterized in that: In step S2, the method for judging whether there is a parked vehicle in the current frame is: if the intersection-over-union ratio (IOU) of the target frames of the same ID vehicle in the two frames before and after within 10 seconds is greater than 0.95, then the ID vehicle is judged to be parked; otherwise, it is judged to be not parked.
5. The traffic accident detection method based on a multimodal video classification model according to claim 4 is characterized in that: The specific content of step S3 is: based on the parked vehicles output in step S2, clustering is performed according to the size of the parked vehicles and the distance between the vehicles, adjacent parked vehicles are clustered together, and the parking area is output.
6. The traffic accident detection method based on a multimodal video classification model according to claim 5, characterized in that: In step S5, the multimodal large model used is VideoCLIP.
7. A traffic accident detection system based on a multimodal video classification model, characterized in that: It includes a video stream acquisition unit, a traffic accident detection unit, an alarm unit and an event retrieval and query unit; The video stream acquisition unit is used to collect and store traffic monitoring video images and cache video frames; The traffic accident detection unit is used to detect traffic accidents; The traffic accident detection unit detects and tracks vehicles based on the images obtained by the video stream unit, uses the tracking information to find parked vehicles, clusters parked vehicles, and then extracts the parking area ROI image sequence from the cache frame, and then uses the multimodal large model VideoCLIP to perform video classification to determine whether an accident has occurred; The alarm unit is used to promptly alarm when the traffic accident detection unit detects a traffic accident; The event retrieval and query unit is used for quickly retrieving and querying the traffic accidents that have occurred.