Collision event detection method, medium, equipment and system
By using a collision event detection method based on a video mixing model in video stream processing, combined with a cooperative detection mechanism of small models and multimodal large models, the problem of low collision event detection accuracy in the existing technology is solved, and higher detection accuracy and lower computing costs are achieved.
Patent Information
- Application Number
- CN202411839114.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-05-02
AI Technical Summary
In the prior art, the accuracy of deep learning small models for collision event detection is low, and it is greatly affected by weather and camera quality, so accurate detection cannot be achieved.
The collision event detection method based on the video mixing model is adopted, and the driving speed of the vehicle is detected by taking the continuously acquired video frames as input to the vehicle detection model, and collision detection is performed in the approaching state between low-speed vehicles. The small model is used for preliminary detection and the multimodal large model is reviewed to generate alarm information.
It improves the accuracy of collision event detection, reduces the computing burden and training cost of small models, and enhances the anti-interference ability of the system.
Smart Images
Figure CN119920124A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video stream processing, and in particular to a collision event detection method, medium, device and system. Background Art
[0002] In the prior art, there are two main methods for detecting collision events. One is to use various sensors to collect the characteristics of road traffic flow and judge the position or collision status between vehicles in real time. However, the rule-based sensor method of collecting the characteristics of road traffic flow lacks flexibility and extensiveness, and cannot meet the detection needs under different road conditions and environments.
[0003] The other method is to use deep learning technology to detect collisions using cameras. However, the problem with this method is that the development and training process of a small deep learning model capable of visual detection is complex, with a long development cycle and high cost. At the same time, it is greatly affected by weather and front-end cameras, and cannot achieve accurate detection. Summary of the invention
[0004] The present application mainly provides a collision event detection method, medium, device and system to solve the problem of low accuracy of collision event detection for small models.
[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a collision event detection method based on a video hybrid model, comprising: using continuously acquired video frames as input of a vehicle detection model to detect the driving speed of each vehicle in the video frames; in response to the existence of low-speed vehicles in a proximity state, the vehicle detection model performs collision detection on the low-speed vehicles in the proximity state to mark a collision event, wherein the proximity state is a distance relationship mark between vehicles; wherein the driving speed of the low-speed vehicle is lower than a preset speed threshold; using a multimodal large model to review the accuracy of the collision event, and generating an alarm message of the collision event that has passed the review; wherein the model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model.
[0006] The positioning and collision judgment of vehicles in the image are completed through a small model, namely the vehicle detection model, which only relies on simple calculations, reduces the computational processing volume of the small model, simplifies the training process, and reduces training costs. The detection results of the small model are reviewed in combination with the multimodal large model to further improve the accuracy of the small model detection.
[0007] In some embodiments, in response to the existence of low-speed vehicles in a proximity state, the vehicle detection model performs collision detection on the low-speed vehicles in the proximity state to mark a collision event, and also includes: expanding the initial detection frame of the low-speed vehicle by a preset first expansion ratio to obtain an expanded detection frame of each of the low-speed vehicles, wherein the initial detection frame is used to calibrate the position and size of the vehicle in the video frame; calculating a first overlap degree between the expanded detection frames of each of the low-speed vehicles to mark the relevant low-speed vehicles whose first overlap degree is greater than a preset first threshold as being in a proximity state.
[0008] By detecting vehicles and expanding the overlapping area between detection frames, the approximate distance between vehicles can be determined, thereby distinguishing whether the vehicles are in an adjacent state.
[0009] In some embodiments, in response to the existence of low-speed vehicles in a proximity state, the vehicle detection model performs collision detection on the low-speed vehicles in the proximity state to mark a collision event, including: in response to the existence of the low-speed vehicles in a proximity state, the vehicle detection model obtains a suspected car accident frame of the low-speed vehicles in the proximity state; the vehicle detection model obtains a pedestrian detection frame from the video frame, and calculates a second overlap degree between the pedestrian detection frame and the suspected car accident frame, the pedestrian detection frame being used to calibrate the position and size of the pedestrian in the video frame; in response to the second overlap degree being greater than or equal to a preset second threshold, the low-speed vehicle associated with the suspected car accident frame is marked as a collision event.
[0010] The presence of a collision between adjacent vehicles is determined based on whether there are pedestrians getting out of the vehicle to deal with the accident in the suspected accident area.
[0011] In some embodiments, in response to the existence of low-speed vehicles in a proximity state, the vehicle detection model obtains a suspected car accident frame of the low-speed vehicles in the proximity state, including: the vehicle detection model obtains the duration of the low-speed vehicles in the proximity state; in response to the duration being greater than or equal to a preset duration threshold, taking the union of the expanded detection frames of the relevant low-speed vehicles as the union detection frame; and expanding the union detection frame with a preset second expansion ratio to obtain the suspected car accident frame.
[0012] The frame range is expanded with the union of the two vehicles' highly overlapping positions as the center, which is used as the approximate area where the accident occurred in the video frame.
[0013] In some embodiments, the use of a multimodal large model to review the accuracy of the collision event and generate alarm information for the collision event that has passed the review includes: in response to the generation of the collision event, the multimodal large model obtains real-time video frames in which the collision event occurs multiple times at time intervals, and reviews whether the collision event in the real-time video frames is established; in response to the collision events being identified as established, generating alarm information for the collision event.
[0014] In some embodiments, the continuously acquired video frames are used as input to a vehicle detection model to obtain the position of each vehicle in a first video frame and a second video frame, including: using the continuously acquired video frames as input to the vehicle detection model to obtain historical bounding box coordinates of each vehicle and a current image of each vehicle in the first video frame or the second video frame; constructing a motion model of each vehicle based on the historical bounding box coordinates of each vehicle to predict the predicted coordinates of each vehicle in the first video frame or the second video frame; matching the predicted coordinates with the current image of each vehicle to obtain the current bounding box coordinates of each vehicle in the current image as the position of each vehicle in the first video frame or the second video frame.
[0015] In some embodiments, the calculation of the moving speed of each vehicle from the position in the first video frame to the position in the second video frame includes: respectively calculating the coordinate offset in the first direction and the coordinate offset in the second direction of each vehicle in the process of moving from the position in the first video frame to the position in the second video frame; converting the coordinate offset of each vehicle in the first direction to the actual offset in the first direction, and converting the coordinate offset of each vehicle in the second direction to the actual offset in the second direction; calculating the actual offset distance of each vehicle based on the actual offset in the first direction and the actual offset in the second direction; dividing the actual offset distance of each vehicle by the time difference between the first video frame and the second video frame to obtain the moving speed of the vehicle from the position in the first video frame to the position in the second video frame.
[0016] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the collision event detection method as described above are implemented.
[0017] The beneficial effects of the storage medium refer to the beneficial effects of the above-mentioned collision event detection method, which will not be repeated here.
[0018] The present application also provides a computer device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the collision event detection method as described above are implemented.
[0019] The beneficial effects of the computer device refer to the beneficial effects of the above-mentioned collision event detection method, which will not be repeated here.
[0020] The present application also provides a collision event detection system, comprising a camera and a computer device as described above; the camera is installed above the road, and is used to continuously acquire video frames and send them to the computer device; the computer device receives the video frames sent by the camera to execute the collision event detection method as described above.
[0021] The beneficial effects of the collision event detection system refer to the beneficial effects of the collision event detection method described above, which will not be repeated here.
[0022] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a collision event detection method, medium, device and system, which uses the continuously acquired video frames as the input of the vehicle detection model to detect the driving speed of each vehicle in the video frame, and uses the small model to perform simple operations on a small number of parameters to quickly process the video frame. In response to the existence of low-speed vehicles in a close state, the vehicle detection model performs collision detection on the low-speed vehicles in the close state to mark the collision event. The small model vehicle detection model simply identifies the position and speed of the vehicle in the video frame, and can make a preliminary judgment on the occurrence of the collision event. Fewer parameters and operations make the processing speed of the small model faster, and the simple judgment mechanism makes the training cost of the small model lower. The multimodal large model is used to review the feature consistency of the vehicle in the collision event, and generate warning information of the collision event that has passed the review, wherein the model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model, and the multimodal large model is used to simply review the detection results of the small model to eliminate the situation of misdetection by the small model, thereby improving the detection accuracy of the collision event. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0024] Figure 1 It is a flow chart of an embodiment of a collision event detection method based on a video hybrid model provided by the present application;
[0025] Figure 2 Yes Figure 1 A schematic flow chart of an embodiment of step 10 in the method shown;
[0026] Figure 3Yes Figure 2 A schematic flow chart of an embodiment of step 11 in the method shown;
[0027] Figure 4 Yes Figure 2 A schematic flow chart of an embodiment of step 12 in the method shown;
[0028] Figure 5 Schematic diagram of the detection frame when the road vehicle is traveling horizontally and longitudinally;
[0029] Figure 6 Yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 20 in the method shown;
[0030] Figure 7 Yes Figure 1 A schematic flow chart of another embodiment of step 20 in the method shown;
[0031] Figure 8 Yes Figure 7 A schematic diagram of a flow chart of an embodiment of step 23 in the method shown;
[0032] Fig. 9 Yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 30 in the method shown;
[0033] Fig.10 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;
[0034] Fig.11 is a structural schematic diagram of an embodiment of a computer device provided by the present application;
[0035] Fig.12 It is a structural schematic diagram of an embodiment of a collision event detection system provided by the present application. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0037] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0038] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0039] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a collision event detection method based on a video hybrid model provided by the present application. The collision event detection method based on a video hybrid model includes:
[0040] Step 10: Use the continuously acquired video frames as input to the vehicle detection model to detect the driving speed of each vehicle in the video frames.
[0041] The key image frames are extracted from the video stream data sent back by the surveillance camera at a preset frame rate to obtain continuous video frames as the input of the vehicle detection model. The vehicle detection model outputs the boundary of each vehicle in the video frame image and forms the corresponding initial detection frame. The initial detection frame information includes the coordinates of the initial detection frame anchor point, such as the upper left corner coordinates, the lower left corner coordinates or the center coordinates.
[0042] The vehicle detection model marks the initial detection frame coordinates and category information for the vehicle according to the object category of the target in the initial detection frame, continuously locates and tracks the vehicle, and thus calculates the moving speed of each vehicle.
[0043] The vehicle detection model is a small model that has been parameterized and trained. Compared with the large deep learning model, it has lower memory, parameter types and computing power, but the small model has the advantages of simple construction, low cost, fast processing speed, etc., which makes it easy to use.
[0044] See also Figure 2 , step 10 further comprises:
[0045] Step 11: Use the continuously acquired video frames as input to the vehicle detection model to obtain the position of each vehicle in the first video frame and the second video frame.
[0046] According to the vehicle detection results of the vehicle detection model, the information corresponding to each vehicle ID in the vehicle tracking program is initialized, and the initial detection frame information and motion status of each vehicle such as direction, speed, etc. are recorded at this time, and the above information is continuously updated based on the image obtained in each frame.
[0047] Further, see Figure 3 , step 11 further includes:
[0048] Step 111: using the continuously acquired video frames as inputs of the vehicle detection model to obtain historical bounding box coordinates of each vehicle and a vehicle image of each vehicle in the first video frame or the second video frame.
[0049] Each extracted video frame is input into the vehicle detection model, which performs image feature extraction through a convolutional neural network to extract feature information about the vehicle in the video frame, output the boundary of each vehicle in the video frame image, and form the corresponding initial detection frame. The initial detection frame information includes the coordinates of the initial detection frame anchor point, such as the upper left corner coordinates, the lower left corner coordinates, or the center coordinates.
[0050] Step 112: constructing a motion model of each vehicle based on the historical bounding box coordinates of each vehicle to predict the predicted coordinates of each vehicle in the first video frame or the second video frame.
[0051] The vehicle detection model marks the initial detection frame coordinates and category information for the vehicle according to the object category of the target in the initial detection frame, such as target ID: 123456, coordinates (12, 35), category: small motor vehicle. The vehicle's marking information can be used to continuously locate and track the vehicle.
[0052] After obtaining video frames within a certain period of time (such as all video frames extracted within 2 seconds), a motion model corresponding to each vehicle is constructed based on the position, direction and other information of each vehicle in each video frame to predict the vehicle position in the next frame.
[0053] Step 113: Match the predicted coordinates with the current image of each vehicle to obtain the current bounding box coordinates of each vehicle in the current image as the position of each vehicle in the first video frame or the second video frame.
[0054] The vehicle position in the next frame predicted by the vehicle detection model is matched with the actual next frame image, so that each vehicle in the image corresponds to the initial detection frame, and the motion model of each vehicle is updated according to the detection results.
[0055] Two video frames, a first video frame and a second video frame, separated by a certain time are captured, and the vehicles in the first video frame and the second video frame are matched with the initial detection frame in the above manner to determine the position coordinates of each vehicle in the first video frame and the second video frame, such as (x, y).
[0056] Among them, x is the first direction coordinate corresponding to the position of the vehicle in the first video frame or the second video frame, and y is the second direction coordinate corresponding to the position of the vehicle in the first video frame or the second video frame.
[0057] Step 12: Calculate the moving speed of each vehicle from the position in the first video frame to the position in the second video frame.
[0058] The moving speed of each vehicle from the position in the first video frame to the position in the second video frame is calculated by the time interval from the first video frame to the second video frame and the distance traveled by the vehicle from the position in the first video frame to the position in the second video frame.
[0059] Further, see Figure 4 , step 12 further includes:
[0060] Step 121: Calculate the coordinate offset in the first direction and the coordinate offset in the second direction of each vehicle in the process of moving from the position in the first video frame to the position in the second video frame.
[0061] Select a point in the video frame image as the original location to establish an image coordinate system, which uses the first direction and the second direction as the positive directions of the two coordinate axes to obtain the position coordinates (x1, y1) of the vehicle in the first video frame and the position coordinates (x2, y2) in the second video frame, and calculate the coordinate offset Δx of the vehicle in the first direction respectively. p and the coordinate offset Δy in the second direction p .
[0062] Specifically, the coordinate offset Δx of the vehicle in the first direction is calculated p The formula is:
[0063] Δx p =|x2-x1|.
[0064] Among them, x1 is the first direction coordinate corresponding to the position of the vehicle in the first video frame, and x2 is the first direction coordinate corresponding to the position of the vehicle in the second video frame.
[0065] Calculate the coordinate offset Δy of the vehicle in the second direction p The formula is:
[0066] Δy p =|y2-y1|.
[0067] Among them, y1 is the second direction coordinate corresponding to the position of the vehicle in the first video frame, and y2 is the second direction coordinate corresponding to the position of the vehicle in the second video frame.
[0068] For example, the current video surveillance includes vehicle A, and the position coordinates of vehicle A in the first video frame are (23, 46), and the position coordinates in the second video frame are (25, 52). In the process of vehicle A moving from the position in the first video frame to the position in the second video frame, the displacement in the first direction is |25-23|=2, and the displacement in the second direction is |52-46|=6.
[0069] Step 122: Convert the coordinate offset of each vehicle in the first direction into an actual offset in the first direction, and convert the coordinate offset of each vehicle in the second direction into an actual offset in the second direction.
[0070] The size of the data in the image is restored to the size of the data in the real space through the ratio of the image to the actual space. The ratio of the image to the actual space can be calculated based on the length and width data of the initial detection frame of the vehicle in the video frame and the actual length, width and height data of the vehicle.
[0071] See also Figure 5 When the vehicle is traveling longitudinally, the vehicle in the video frame image, such as vehicle S, mainly includes the roof, that is, the top view angle. The width of the initial detection frame is the width of the vehicle in the video frame, and the height of the initial detection frame is the length of the vehicle in the video frame. At this time, the width of the initial detection frame is smaller than the height of the initial detection frame.
[0072] When the vehicle is traveling sideways, the vehicle in the video frame image, such as vehicle R, mainly includes the side of the door, that is, the side view angle, and the width of the initial detection frame is the length of the vehicle in the video frame, and the height of the initial detection frame is the height of the vehicle in the video frame. At this time, the width of the initial detection frame is greater than the height of the initial detection frame.
[0073] On this basis, the ratio between the image and the actual size is calculated, and the moving distance on the image can be converted into the actual moving distance according to the calculated ratio.
[0074] Specifically, the formula for converting the coordinate offset of each vehicle in the first direction into the actual offset in the first direction is:
[0075]
[0076] or:
[0077] Among them, d x1 is the actual offset in the first direction when the vehicle is traveling longitudinally, d x2 is the actual offset in the first direction when the vehicle is traveling laterally, Δx p is the coordinate offset of the vehicle in the first direction, Δx r is the conversion ratio of the coordinate offset in the first direction to the actual offset in the first direction, x1 is the first direction coordinate corresponding to the position of the vehicle in the first video frame, x2 is the first direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, W car is the width of the vehicle, and w is the width of the initial detection frame of the vehicle.
[0078] When the vehicle is traveling longitudinally, the coordinate offset of the vehicle in the first direction is the difference between the first direction coordinate x2 of the vehicle in the second video frame and the first direction coordinate x1 in the first video frame, that is, Δx p =|x2-x1|.
[0079] The ratio between the image and the actual size is the ratio of the initial detection frame size, i.e., the size of the vehicle in the image, to the actual size of the vehicle. When the vehicle is traveling longitudinally, the vehicle in the video frame image, such as vehicle S, mainly includes the roof, i.e., the top view. At this time, the width w of the initial detection frame is the width of the vehicle in the video frame, corresponding to the actual width W of the vehicle. car , the ratio between the image and the actual size is
[0080] When the vehicle is traveling sideways, the vehicle in the video frame, such as vehicle R, mainly includes the side of the door, that is, the side view angle. At this time, the width w of the initial detection frame is the length of the vehicle in the video frame, which corresponds to the actual length H of the vehicle. car , the ratio between the image and the actual size is
[0081]
[0082] Finally, multiply the coordinate offset of the vehicle in the video frame by the ratio between the image and the actual size to get:
[0083]
[0084] or,
[0085] For example, there is a vehicle B that is traveling longitudinally. Its position in the first video frame is (32, 11), and its position in the second video frame is (30, 12). First, calculate the coordinate offset Δx of vehicle B in the first direction. p=|x2-x1|=|30-32|=2. The width of the initial detection frame of vehicle B in the video frame is 1.3cm, or 0.013m. The actual width of vehicle B is 1.3m. The ratio between the image and the actual size is Then the actual offset in the first direction when vehicle B is traveling longitudinally is:
[0086] That is 200m.
[0087] Similarly, the formula for converting the coordinate offset of each vehicle in the second direction into the actual offset in the second direction is:
[0088]
[0089] or:
[0090] Among them, d y1 is the actual offset in the second direction when the vehicle is traveling longitudinally, d y2 is the actual offset in the second direction when the vehicle is traveling laterally, Δy p is the coordinate offset of the vehicle in the second direction, Δy r is the conversion ratio of the coordinate offset in the second direction to the actual offset in the second direction, y1 is the second direction coordinate corresponding to the position of the vehicle in the first video frame, y2 is the second direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, Vcar is the height of the vehicle, and h is the height of the initial detection frame of the vehicle.
[0091] When the vehicle is traveling longitudinally, the coordinate offset of the vehicle in the second direction is the difference between the second direction coordinate y2 of the vehicle in the second video frame and the second direction coordinate y1 in the first video frame, that is, Δy p =|y2-y1|.
[0092] The ratio between the image and the actual size is the ratio of the initial detection frame size, i.e., the size of the vehicle in the image, to the actual size of the vehicle. When the vehicle is traveling longitudinally, the vehicle in the video frame image, such as vehicle S, mainly includes the roof, i.e., the top view. At this time, the height h of the initial detection frame is the length of the vehicle in the video frame, which corresponds to the actual length H of the vehicle. car , the ratio between the image and the actual size is
[0093] When the vehicle is traveling sideways, the vehicle in the video frame, such as vehicle R, mainly includes the side view of the door. At this time, the height h of the initial detection frame is the height of the vehicle in the video frame, which corresponds to the actual height V of the vehicle. car , the ratio between the image and the actual size is
[0094]
[0095] Finally, multiply the coordinate offset of the vehicle in the video frame by the ratio between the image and the actual size to get:
[0096]
[0097] or,
[0098] For example, there is a vehicle B that is traveling longitudinally. Its position in the first video frame is (32, 11), and its position in the second video frame is (30, 12). First, calculate the coordinate offset Δy of vehicle B in the second direction. p =|y2-y1|=|12-11|=1. The height of the initial detection frame of vehicle B in the video frame is 3.8cm, or 0.038m. The actual length of vehicle B is 3.8m. The ratio between the image and the actual size is Then the actual offset in the second direction when vehicle B is traveling longitudinally is:
[0099] That is 100m.
[0100] Step 123: Calculate the actual offset distance of each vehicle based on the actual offset in the first direction and the actual offset in the second direction.
[0101] The moving distance of the vehicle is calculated as the straight-line distance between the two points, that is, a right triangle is constructed with the actual offset in the first direction and the actual offset in the second direction as right-angled sides, and the hypotenuse of the triangle is the actual offset distance.
[0102] The actual offset distance can be obtained by the formula: To obtain.
[0103] Among them, d car is the actual offset distance of the vehicle, d x is the actual offset distance of the vehicle in the first direction, which can be specifically d x1 or x2 , d y is the actual offset distance of the vehicle in the second direction, which can be specifically d y1 or y2 .
[0104] For example, there is a vehicle B that is traveling longitudinally, and its position in the first video frame is (32, 11), and its position in the second video frame is (30, 12). According to the calculation method in step 122, the actual offset of vehicle B in the first direction is 200m. The actual offset in the second direction is 100m. Then the actual offset generated by vehicle B moving from the position in the first video frame to the position in the second video frame is: About 223.6m.
[0105] Step 124: Divide the actual offset distance of each vehicle by the time difference between the first video frame and the second video frame to obtain the moving speed of the vehicle from the position in the first video frame to the position in the second video frame.
[0106] The moving speed of the vehicle from the position in the first video frame to the position in the second video frame can be expressed as:
[0107] Among them, V car is the moving speed of the vehicle, d car is the actual offset distance of the vehicle, t1 is the acquisition time of the first video frame, and t2 is the acquisition time of the second video frame.
[0108] For example, there is a vehicle E currently moving. The actual offset obtained by the calculation method in step 122 and step 123 is 23.6m. The acquisition time of the first video frame is 0.235s, and the acquisition time of the second video frame is 1.235s. Then, the moving speed of the vehicle E from the position in the first video frame to the position in the second video frame is: That is 23.6m / s.
[0109] Step 20: In response to the existence of low-speed vehicles in a proximity state, the vehicle detection model performs collision detection on the low-speed vehicles in the proximity state to mark a collision event; wherein the driving speed of the low-speed vehicle is lower than a preset speed threshold, and the proximity state is a distance relationship mark between vehicles.
[0110] The driving speed of each vehicle in the video frame is calculated by the method in step 10, and the driving speed of each vehicle is compared with the preset speed threshold. If the driving speed of the vehicle is lower than the preset speed threshold, the vehicle is a low-speed vehicle. The vehicle detection model performs collision detection on the low-speed vehicles in the adjacent state to determine whether a collision occurs between the two low-speed vehicles. If a collision occurs, a collision event is marked at that location.
[0111] Among them, the proximity state is a distance relationship mark between vehicles.
[0112] Further, see Figure 6 , before step 20, further comprising:
[0113] Step 21: Enlarging the initial detection frame of the low-speed vehicle with a preset first enlargement ratio to obtain an enlarged detection frame of each low-speed vehicle, wherein the initial detection frame is used to calibrate the position and size of the vehicle in the video frame.
[0114] The position and size of each vehicle are given by Figure 5 The initial detection frame of the rectangle shown in the figure can be expressed as B = (x, y, w, h), where x represents the first direction coordinate of the anchor point of the initial detection frame, y represents the second direction coordinate of the anchor point of the initial detection frame, and the anchor point coordinates (x, y) can be used as the coordinates of the initial detection frame corresponding to the vehicle in the video frame; w is the width of the initial detection frame, and h is the height of the initial detection frame. With the anchor point as the center, the width and height of the initial detection frame are enlarged according to the preset first enlargement ratio p, and the enlarged detection frame B' is obtained as follows:
[0115] B'=(x, y, w×(1+p), h×(1+p)).
[0116] For example, for a vehicle F traveling longitudinally, the initial detection frame is B=(1, 2, 0.5, 0.8), and the first enlargement ratio is 3. Then the enlarged detection frame B' after enlargement is:
[0117]
[0118] Step 22: Calculate the first overlap degree between the enlarged detection frames of the low-speed vehicles, so as to mark the relevant low-speed vehicles whose first overlap degree is greater than a preset first threshold as being in an adjacent state.
[0119] By calculating the intersection-over-union (IoU) ratio between the two vehicles, we can measure the first overlap between the two vehicles’ enlarged detection frames. The size of the IoU ratio indicates the proportion of the overlap between the two vehicles’ enlarged detection frames to the total area of the detection frames, which can be expressed as:
[0120]
[0121] Among them, IoU(B1', B2') represents the intersection-over-union ratio between the enlarged detection frame of vehicle 1 and the enlarged detection frame of vehicle 2, Area(B1'∩B2') represents the intersection between the enlarged detection frame of vehicle 1 and the enlarged detection frame of vehicle 2, that is, the overlapping area, and Area(B1'∪B2') represents the union between the enlarged detection frame of vehicle 1 and the enlarged detection frame of vehicle 2, that is, the total area.
[0122] The obtained first overlap degree intersection over union (IoU) is compared with a preset first threshold. If the first overlap degree is greater than the preset first threshold, vehicle 1 and vehicle 2 are in a close state.
[0123] For example, the existing longitudinally traveling vehicle G, expands the detection frame to BG '(3, 12, 2, 4) and the vehicle H traveling sideways, expand the detection box to B H '(4, 10, 5, 3). The intersection area of the enlarged detection frame of vehicle G and vehicle H is Area = (B H '∩B G ')=2, the union area of the enlarged detection frames of vehicle G and vehicle H is Area=(B H '∪B G ')=21, then the intersection-over-combination ratio of the enlarged detection frames of vehicle G and vehicle H is
[0124] The first threshold is 0.5, and at this time the first overlap degree between the vehicle G and the vehicle H is less than the first threshold, and the vehicle G and the vehicle H are not in an adjacent state.
[0125] Specifically, the first threshold value may be expressed in a decimal form, a fraction form, or a percentage form, such as 0.48, 0.69, 2 / 3, 4 / 5, 60% or 52%, etc.
[0126] Furthermore, step 20 also includes:
[0127] Step 23: In response to the existence of low-speed vehicles in close proximity, the vehicle detection model obtains suspected traffic accident frames of the low-speed vehicles in close proximity.
[0128] If the first overlap degree between vehicle 1 and vehicle 2 is greater than the first threshold, vehicle 1 and vehicle 2 are in proximity. At this time, the vehicle detection model obtains a suspected traffic accident frame between vehicle 1 and vehicle 2 as the location and range of the suspected collision event.
[0129] Specifically, step 23 also includes:
[0130] Step 231: The vehicle detection model obtains the duration of the low-speed vehicle in the approaching state.
[0131] When the first overlap degree between vehicle 1 and vehicle 2 is greater than the first threshold, the timestamp at this time is recorded, and the duration of the proximity between vehicle 1 and vehicle 2 is detected.
[0132] Step 232: In response to the duration being greater than or equal to a preset duration threshold, taking the union of the enlarged detection frames of the relevant low-speed vehicles as the union detection frame.
[0133] When the duration between vehicle 1 and vehicle 2 exceeds the preset duration threshold, the union of the expanded detection frames of vehicle 1 and vehicle 2 is taken as the union detection frame B. 12 .
[0134] For example, there are two vehicles G traveling longitudinally and H traveling transversely in an adjacent state. The expanded detection frame of vehicle G is B.G '(3, 12, 2, 4), the enlarged detection box of vehicle H is B H '(4, 10, 5, 3). The union of the enlarged detection boxes between vehicle G and vehicle H is: B GH (4, 10, 1, 2).
[0135] Step 233: enlarging the combined detection frame at a preset second enlargement ratio to obtain a suspected traffic accident frame.
[0136] Take the union detection box B 12 With the anchor point as the center, according to the enlargement method in step 21, the width and height of the union detection frame are enlarged by a preset second enlargement ratio n to obtain the enlarged suspected car accident frame B 12 '.
[0137] For example, there are two vehicles G traveling longitudinally and H traveling transversely in an adjacent state, and the union detection box between vehicle G and vehicle H is B. GH (4, 10, 1, 2). The second enlargement ratio n = 0.6, then the enlarged suspected car accident frame B GH 'for:
[0138]
[0139] Step 24: The vehicle detection model obtains a pedestrian detection frame from the video frame and calculates a second overlap degree between the pedestrian detection frame and the suspected traffic accident frame.
[0140] The vehicle detection model detects pedestrians that appear in the continuously acquired video frames and forms a pedestrian detection frame for marking pedestrians. The pedestrian detection frame is used to calibrate the position and size of pedestrians in the video frame. The representation of the pedestrian detection frame is the same as that of the vehicle detection frame.
[0141] When the vehicle detection model detects that a pedestrian appears in the video frame, a pedestrian detection frame is marked for the pedestrian, and a second overlap degree between the pedestrian detection frame and the suspected traffic accident frame is calculated by the method as in step 22.
[0142] Step 25: In response to the second overlap degree being greater than or equal to a preset second threshold, marking the low-speed vehicle associated with the suspected traffic accident frame as a collision event.
[0143] If the second overlap degree is greater than the preset second threshold, it means that a pedestrian is close to the suspected traffic accident area. The pedestrian can be regarded as a pedestrian who gets off the vehicle to deal with the collision event. The low-speed vehicle related to the suspected traffic accident frame is marked as having a collision event.
[0144] Specifically, the second threshold is expressed in the same way as the first threshold, and can be expressed in decimal form, fractional form or percentage form, such as 0.46, 0.79, 1 / 3, 2 / 5, 70% or 62%.
[0145] Step 30: Use the multimodal large model to review the accuracy of the collision event and generate warning information for the collision event that passes the review; wherein the model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model.
[0146] The information of the collision event and the video frames within a preset time interval before and after the collision event, such as 3s or 2s, are input into the multimodal large model to verify the accuracy of the collision event. The multimodal large model extracts and classifies the features of the vehicle to ensure that the event detected by the vehicle detection model is indeed a collision event.
[0147] The model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model.
[0148] The memory, computing power, and number of parameters of the multimodal large model are greater than those of the vehicle detection model. In comparison, the multimodal large model is a large model, and the vehicle detection model is a small model.
[0149] Large models can perform deeper learning and calculations and process more complex image information. Small models have poorer processing capabilities, but have faster processing speeds, are simple to train, and have low costs.
[0150] The detection results of the small model are simply reviewed with the multimodal large model to eliminate the misdetection of the small model, thereby improving the accuracy of congestion detection.
[0151] Further, step 30 includes:
[0152] Step 31: In response to the generation of a collision event, the multimodal large model obtains real-time video frames of the collision event multiple times at time intervals, and verifies whether the collision event is established in the real-time video frames.
[0153] When the second overlap between the pedestrian detection frame and the suspected car accident frame is greater than a preset second threshold, a collision event is generated. In response to the generation of the collision event, the multimodal large model obtains real-time video frames of the collision event multiple times at time intervals. Specifically, the information of the collision event and the video frames within a preset time interval before and after the collision event, such as 3s or 2s, are input into the multimodal large model to verify the accuracy of the collision event. The multimodal large model verifies whether the collision event is established in the real-time video frame by extracting and classifying the features of the vehicles related to the collision event.
[0154] Step 32: In response to the collision events being identified as established, generating collision event warning information.
[0155] When a collision event passes the review of the multimodal large model, an alarm message of the collision event is generated, including the vehicle ID related to the collision event, information about the area where the collision event occurred, timestamp and other information, to facilitate subsequent processing by traffic management personnel.
[0156] See also Fig.10 , Fig.10 It is a structural diagram of an embodiment of the storage medium provided by the present application.
[0157] The storage medium 300 stores program data 310. When the program data 310 is executed by the processor, the following is achieved: Figure 1 The steps of the collision event detection method are described.
[0158] The program data 310 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.
[0159] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.
[0160] See also Fig.11 , Fig.11 It is a structural schematic diagram of an embodiment of a computer device provided by the present application.
[0161] The computer device 400 includes a processor 420 and a memory 410 which are connected to each other. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the above-mentioned collision event detection method are implemented.
[0162] See also Fig.12 , Fig.12 1 is a schematic diagram of the structure of an embodiment of a collision event detection system provided in the present application. The collision event detection system 1000 includes a camera 500 and the computer device 400 as described above.
[0163] Camera 500 is installed above the road. It is a camera device specially used to monitor target objects such as vehicles, drivers, pedestrians and landmarks in motion on the traffic road. It includes but is not limited to gun cameras and smart ball cameras. It realizes the monitoring, switching and control functions of network users by digitally encoding and compressing video images and then outputting them.
[0164] The camera 500 is used to continuously acquire video frames and send them to the computer device 400 via a communication connection capable of transmitting video information, and extract the video frames for image processing.
[0165] The computer device 400 receives the video frames sent by the camera 500 to execute the collision event detection method as described above.
[0166] Different from the prior art, the present application provides a collision event detection method, medium, device and system. Preliminary collision event detection is performed through a small model of the vehicle detection model, which only calculates simple parameters, takes up little memory, has fast processing speed and low training cost. Then, the detection results are reviewed through a multimodal large model to eliminate the results of small model detection errors and issue an alarm, which makes up for the problem of low detection accuracy of the small model and improves the detection accuracy of collision events.
[0167] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0168] The present application can be used in many general or special computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.
[0169] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.
[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0172] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A collision event detection method based on a video hybrid model, characterized in that: include: Using the continuously acquired video frames as input to a vehicle detection model to detect the driving speed of each vehicle in the video frames; In response to the existence of low-speed vehicles in a proximity state, the vehicle detection model performs collision detection on the low-speed vehicles in the proximity state to mark a collision event; wherein the travel speed of the low-speed vehicle is lower than a preset speed threshold, and the proximity state is a distance relationship mark between the vehicles; A multimodal large model is used to review the accuracy of the collision event, and an alarm message of the collision event that passes the review is generated; wherein the model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model.
2. The collision event detection method according to claim 1, characterized in that: In response to the low-speed vehicles being in an adjacent state, the vehicle detection model performs collision detection on the low-speed vehicles in the adjacent state to mark a collision event, and further includes: Enlarging the initial detection frame of the low-speed vehicle at a preset first enlargement ratio to obtain an enlarged detection frame of each of the low-speed vehicles, wherein the initial detection frame is used to calibrate the position and size of the vehicle in the video frame; A first overlap degree between the enlarged detection frames of the low-speed vehicles is calculated, so as to mark the related low-speed vehicles whose first overlap degree is greater than a preset first threshold as being in an approaching state.
3. The collision event detection method according to claim 2, characterized in that: In response to the existence of low-speed vehicles in an adjacent state, the vehicle detection model performs collision detection on the low-speed vehicles in the adjacent state to mark a collision event, including: In response to the low-speed vehicles being in an adjacent state, the vehicle detection model acquires suspected traffic accident frames of the low-speed vehicles in the adjacent state; The vehicle detection model obtains a pedestrian detection frame from the video frame, and calculates a second overlap degree between the pedestrian detection frame and the suspected traffic accident frame, wherein the pedestrian detection frame is used to calibrate the position and size of the pedestrian in the video frame; In response to the second overlap degree being greater than or equal to a preset second threshold, the low-speed vehicle associated with the suspected traffic accident frame is marked as a collision event.
4. The collision event detection method according to claim 3, characterized in that: In response to the low-speed vehicles being in an adjacent state, the vehicle detection model acquires a suspected traffic accident frame of the low-speed vehicles in the adjacent state, including: The vehicle detection model obtains the duration of the low-speed vehicle being in the approaching state; In response to the duration being greater than or equal to a preset duration threshold, taking a union of the enlarged detection frames of the related low-speed vehicles as a union detection frame; The union detection frame is enlarged at a preset second enlargement ratio to obtain the suspected traffic accident frame.
5. The collision event detection method according to claim 1, characterized in that: The using of the multi-modal large model to review the accuracy of the collision event and generating warning information of the collision event that passes the review includes: In response to the generation of the collision event, the multimodal large model acquires the real-time video frames where the collision event occurs multiple times at time intervals, and verifies whether the collision event is established in the real-time video frames; In response to the collision events being identified as being established, warning information of the collision events is generated.
6. The collision event detection method according to claim 1, characterized in that: The continuously acquired video frames are used as input of the vehicle detection model to detect the driving speed of each vehicle in the video frames, including: Using the continuously acquired video frames as input to the vehicle detection model to obtain the position of each vehicle in the first video frame and the second video frame; The moving speed of each vehicle from the position in the first video frame to the position in the second video frame is calculated.
7. The collision event detection method according to claim 6, characterized in that: The continuously acquired video frames are used as inputs of the vehicle detection model to obtain the positions of each vehicle in the first video frame and the second video frame, including: Using the continuously acquired video frames as input to the vehicle detection model to obtain historical bounding box coordinates of each vehicle and a current image of each vehicle in the first video frame or the second video frame; constructing a motion model of each vehicle based on the historical bounding box coordinates of each vehicle to predict the predicted coordinates of each vehicle in the first video frame or the second video frame; The predicted coordinates are matched with a current image of each vehicle to obtain current bounding box coordinates of each vehicle in the current image as the position of each vehicle in the first video frame or the second video frame.
8. The collision event detection method according to claim 6, characterized in that: The calculating the moving speed of each vehicle from the position in the first video frame to the position in the second video frame includes: Calculate the coordinate offset in the first direction and the coordinate offset in the second direction of each vehicle in the process of moving from the position in the first video frame to the position in the second video frame; Convert the coordinate offset of each vehicle in the first direction into the actual offset in the first direction, and convert the coordinate offset of each vehicle in the second direction into the actual offset in the second direction; Calculating an actual offset distance of each vehicle based on an actual offset in the first direction and an actual offset in the second direction; The actual offset distance of each vehicle is divided by the time difference between the first video frame and the second video frame to obtain the moving speed of the vehicle from the position in the first video frame to the position in the second video frame.
9. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the collision event detection method according to any one of claims 1 to 8 are implemented.
10. A computer device, characterized in that: The method comprises a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the collision event detection method according to any one of claims 1 to 8 are implemented.
11. A collision event detection system, characterized in that: comprising a camera and a computer device as claimed in claim 10; The camera is installed above the road and is used to continuously acquire video frames and send them to the computer device; The computer device receives the video frames sent by the camera to execute the collision event detection method according to any one of claims 1-8.