Road congestion detection method, medium, equipment and system
By combining technical means of small models and multimodal large models in road congestion detection, the problems of poor flexibility and low accuracy of detection methods in the existing technology are solved, and fast and accurate road congestion detection is achieved.
Patent Information
- Application Number
- CN202411839110.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-05-02
AI Technical Summary
In the prior art, road congestion detection methods lack flexibility and breadth, and cannot meet the detection needs of different road conditions and environments. The deep learning small model development training process is complex, costly, low detection accuracy, and is greatly affected by weather and cameras.
The road congestion detection method based on the video mixing model is adopted to track vehicle, speed detection and congestion judgment through small models, and the multimodal large model is used to review the detection results to improve accuracy.
Fast and accurate road congestion detection is achieved, the training process of small models is simplified, training costs are reduced, detection accuracy is improved, and dependence on weather and cameras is reduced.
Smart Images
Figure CN119920086A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video stream processing, and in particular to a road congestion detection method, medium, device and system. Background Art
[0002] In the prior art, there are two main methods for detecting road congestion. One is to use various sensors to collect the characteristics of road traffic flow and judge the congestion situation in real time. The other is to use visual detection technology based on deep learning technology to capture images with cameras for congestion detection.
[0003] However, the problems with these two methods are: the rule-based sensor method of collecting road traffic flow characteristics lacks flexibility and extensiveness, and cannot meet the detection needs under different road conditions and environments; the development and training process of small deep learning models capable of visual detection is complex, with a long development cycle and high cost. At the same time, it is greatly affected by weather and front-end cameras and cannot achieve accurate detection. Summary of the invention
[0004] The present application mainly provides a road congestion detection method, medium, device and system to solve the problem of low accuracy of road congestion detection.
[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a road congestion detection method based on a video hybrid model, comprising: using continuously acquired video frames as input of a vehicle detection model to detect the driving speed of each vehicle in the video frame; in response to the proportion of low-speed vehicles being greater than a first threshold, and the number of the low-speed vehicles being greater than a second threshold, the vehicle detection model detects the congested vehicles in the video frame to mark the congested vehicles; wherein the driving speed of the low-speed vehicles is lower than a preset speed threshold; using a multimodal large model to review the feature consistency of the congested vehicles, and generating alarm information of the congested vehicles that have passed the review; wherein the model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model.
[0006] The small-model vehicle detection model is used to track vehicles, detect speeds and determine congestion, and quickly detect vehicle congestion. The large model reviews the detection results to improve the accuracy of the small-model detection results. The small and large models are combined to determine whether vehicles in the current video are congested, which simplifies the training process of the small model, reduces training costs, and ensures the accuracy of congestion detection.
[0007] In some embodiments, the continuously acquired video frames are used as input to a vehicle detection model to detect the driving speed of each vehicle in the video frames, including: using the continuously acquired video frames as input to a vehicle detection model to obtain the position of each vehicle in a first video frame and a second video frame; calculating the moving speed of each vehicle from a position in the first video frame to a position in the second video frame.
[0008] The vehicle positioning and congestion judgment in the image are completed through a small model, namely the vehicle detection model, which only relies on simple calculations, reducing the computational processing amount of the small model, simplifying the training process and reducing training costs. The detection results of the small model are reviewed in combination with the multimodal large model to further improve the accuracy of the small model detection.
[0009] In some embodiments, the continuously acquired video frames are used as input to a vehicle detection model to obtain the position of each vehicle in a first video frame and a second video frame, including: using the continuously acquired video frames as input to the vehicle detection model to obtain historical bounding box coordinates of each vehicle and a current image of each vehicle in the first video frame or the second video frame; constructing a motion model of each vehicle based on the historical bounding box coordinates of each vehicle to predict the predicted coordinates of each vehicle in the first video frame or the second video frame; matching the predicted coordinates with the current image of each vehicle to obtain the current bounding box coordinates of each vehicle in the current image as the position of each vehicle in the first video frame or the second video frame.
[0010] By building a motion model to continuously track the position of the vehicle, the vehicle's position information in the real-time video frame can be accurately obtained, thereby determining whether the vehicle is in a congested state.
[0011] In some embodiments, the calculation of the moving speed of each vehicle from the position in the first video frame to the position in the second video frame includes: respectively calculating the coordinate offset in the first direction and the coordinate offset in the second direction of each vehicle in the process of moving from the position in the first video frame to the position in the second video frame; converting the coordinate offset of each vehicle in the first direction to the actual offset in the first direction, and converting the coordinate offset of each vehicle in the second direction to the actual offset in the second direction; calculating the actual offset distance of each vehicle based on the actual offset in the first direction and the actual offset in the second direction; dividing the actual offset distance of each vehicle by the time difference between the first video frame and the second video frame to obtain the moving speed of the vehicle from the position in the first video frame to the position in the second video frame.
[0012] In some embodiments, the formula for converting the coordinate offset of each vehicle in the first direction into the actual offset in the first direction is:
[0013]
[0014] or:
[0015] Among them, d x1 is the actual offset in the first direction when the vehicle is traveling longitudinally, d x2 is the actual offset in the first direction when the vehicle is traveling laterally, Δx p is the coordinate offset of the vehicle in the first direction, Δx r is the conversion ratio of the coordinate offset in the first direction to the actual offset in the first direction, x1 is the first direction coordinate corresponding to the position of the vehicle in the first video frame, x2 is the first direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, Wcar is the width of the vehicle, and w is the width of the vehicle detection box.
[0016] In some embodiments, the formula for converting the coordinate offset of each vehicle in the second direction into the actual offset in the second direction is:
[0017]
[0018] or:
[0019] Among them, d y1 is the actual offset in the second direction when the vehicle is traveling longitudinally, d y2 is the actual offset in the second direction when the vehicle is traveling laterally, Δy p is the coordinate offset of the vehicle in the second direction, Δy r is the conversion ratio of the coordinate offset in the second direction to the actual offset in the second direction, y1 is the second direction coordinate corresponding to the position of the vehicle in the first video frame, y2 is the second direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, Vcar is the height of the vehicle, and h is the height of the vehicle detection frame.
[0020] In some embodiments, in response to the proportion of low-speed vehicles being greater than a first threshold and the number of low-speed vehicles being greater than a second threshold, the vehicle detection model detects congested vehicles in the video frame to mark the congested vehicles, including: in response to the proportion of low-speed vehicles being greater than a first threshold and the number of low-speed vehicles being greater than a second threshold, marking all vehicles in the current video frame as suspected congested vehicles; obtaining the position of each suspected congested vehicle in a third video frame; marking suspected congested vehicles whose positions in the third video frame have a similarity with those in the second video frame greater than a third threshold as congested vehicles.
[0021] The overall movement of vehicles within a certain area is used as the basis for judging congestion to determine whether congestion occurs on the current video surveillance section.
[0022] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the road congestion detection method as described above are implemented.
[0023] The beneficial effects of the storage medium refer to the beneficial effects of the above-mentioned road congestion detection method, which will not be repeated here.
[0024] The present application also provides a computer device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the road congestion detection method as described above are implemented.
[0025] The beneficial effects of the computer device refer to the beneficial effects of the above-mentioned road congestion detection method, which will not be repeated here.
[0026] The present application also provides a road congestion detection system, comprising a camera and a computer device as described above; the camera is installed above the road, and is used to continuously acquire video frames and send them to the computer device; the computer device receives the video frames sent by the camera to execute the road congestion detection method as described above.
[0027] The beneficial effects of the road congestion detection system refer to the beneficial effects of the above-mentioned road congestion detection method, which will not be repeated here.
[0028] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a road congestion detection method, medium, device and system. First, the continuously acquired video frames are used as the input of the vehicle detection model to detect the driving speed of each vehicle in the video frame, and a small model is used to perform simple operations on a small number of parameters to quickly process the video frames. In response to the proportion of low-speed vehicles being greater than a first threshold, and the number of low-speed vehicles being greater than a second threshold, the vehicle detection model detects the congested vehicles in the video frame to mark the congested vehicles; wherein the driving speed of the low-speed vehicles is lower than a preset speed threshold, and the small model vehicle detection model is used to simply identify the position and speed of the vehicles in the video frame, so as to make a preliminary judgment on the occurrence of road congestion. Fewer parameters and operations make the processing speed of the small model faster, and the simple judgment mechanism makes the training cost of the small model lower. A multimodal large model is used to verify the feature consistency of congested vehicles and generate warning information for congested vehicles that have passed the review. The model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model. The multimodal large model is used to simply review the detection results of the small model to eliminate the situation of false detection by the small model, thereby improving the detection accuracy of road congestion, and sending vehicle warning information to facilitate subsequent processing by traffic management personnel. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0030] Figure 1 It is a flowchart of an embodiment of a road congestion detection method provided by the present application;
[0031] Figure 2 Yes Figure 1 The flowchart of step 10 of the road congestion detection method shown in FIG.
[0032] Figure 3 Yes Figure 2 The flowchart of step 11 of the road congestion detection method shown in FIG.
[0033] Figure 4 Yes Figure 2 The flowchart of step 12 of the road congestion detection method shown in FIG.
[0034] Figure 5 Schematic diagram of the detection frame when the road vehicle is traveling horizontally and longitudinally;
[0035] Figure 6 Yes Figure 1 The flowchart of step 20 of the road congestion detection method shown in FIG.
[0036] Figure 7 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;
[0037] Figure 8 is a structural schematic diagram of an embodiment of a computer device provided by the present application;
[0038] Fig. 9 It is a structural schematic diagram of an embodiment of a road congestion detection system provided by the present application. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0040] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0042] See also Figure 1 , Figure 1: is a flow chart of an embodiment of a road congestion detection method provided by the present application. The road congestion detection method based on a video hybrid model includes:
[0043] Step 10: Use the continuously acquired video frames as input to the vehicle detection model to detect the driving speed of each vehicle in the video frames.
[0044] The key image frames are extracted from the video stream data sent back by the surveillance camera at a preset frame rate to obtain continuous video frames as the input of the vehicle detection model. The vehicle detection model outputs the boundary of each vehicle in the video frame image and forms a corresponding detection frame. The detection frame information includes the coordinates of the detection frame anchor point, such as the upper left corner coordinates, the lower left corner coordinates or the center coordinates.
[0045] The vehicle detection model marks the detection frame coordinates and category information for the vehicle according to the object category of the target in the detection frame, continuously locates and tracks the vehicle, and thus calculates the moving speed of each vehicle.
[0046] The vehicle detection model is a small model that has been parameterized and trained. Compared with the large deep learning model, it has lower memory, parameter types and computing power, but the small model has the advantages of simple construction, low cost, fast processing speed, etc., which makes it easy to use.
[0047] See also Figure 2 , step 10 further comprises:
[0048] Step 11: Use the continuously acquired video frames as input to the vehicle detection model to obtain the position of each vehicle in the first video frame and the second video frame.
[0049] According to the vehicle detection results of the vehicle detection model, the information corresponding to each vehicle ID in the vehicle tracking program is initialized, and the detection frame information and motion status of each vehicle at this time, such as direction, speed, etc., are recorded, and the above information is continuously updated based on the image obtained in each frame.
[0050] Further, see Figure 3 , step 11 further includes:
[0051] Step 111: using the continuously acquired video frames as inputs of the vehicle detection model to obtain historical bounding box coordinates of each vehicle and a vehicle image of each vehicle in the first video frame or the second video frame.
[0052] Each extracted video frame is input into the vehicle detection model, which performs image feature extraction through a convolutional neural network to extract feature information about the vehicle in the video frame, output the boundary of each vehicle in the video frame image, and form a corresponding detection frame. The detection frame information includes the coordinates of the detection frame anchor point, such as the upper left corner coordinates, the lower left corner coordinates, or the center coordinates.
[0053] Step 112: constructing a motion model of each vehicle based on the historical bounding box coordinates of each vehicle to predict the predicted coordinates of each vehicle in the first video frame or the second video frame.
[0054] The vehicle detection model marks the detection frame coordinates and category information for the vehicle according to the object category of the target in the detection frame, such as target ID: 123456, coordinates (12, 35), category: small motor vehicle. Through the vehicle's marking information, the vehicle can be continuously located and tracked.
[0055] After obtaining video frames within a certain period of time (such as all video frames extracted within 2 seconds), a motion model corresponding to each vehicle is constructed based on the position, direction and other information of each vehicle in each video frame to predict the vehicle position in the next frame.
[0056] Step 113: Match the predicted coordinates with the current image of each vehicle to obtain the current bounding box coordinates of each vehicle in the current image as the position of each vehicle in the first video frame or the second video frame.
[0057] The vehicle position in the next frame predicted by the vehicle detection model is matched with the actual next frame image, so that each vehicle in the image corresponds to the detection frame, and the motion model of each vehicle is updated according to the detection results.
[0058] Two video frames, a first video frame and a second video frame, separated by a certain time are captured, and the vehicles and the detection frames in the first video frame and the second video frame are matched in the above manner to determine the position coordinates of each vehicle in the first video frame and the second video frame, such as (x, y).
[0059] Among them, x is the first direction coordinate corresponding to the position of the vehicle in the first video frame or the second video frame, and y is the second direction coordinate corresponding to the position of the vehicle in the first video frame or the second video frame.
[0060] Step 12: Calculate the moving speed of each vehicle from the position in the first video frame to the position in the second video frame.
[0061] The moving speed of each vehicle from the position in the first video frame to the position in the second video frame is calculated by the time interval from the first video frame to the second video frame and the distance traveled by the vehicle from the position in the first video frame to the position in the second video frame.
[0062] Further, see Figure 4 , step 12 further includes:
[0063] Step 121: Calculate the coordinate offset in the first direction and the coordinate offset in the second direction of each vehicle in the process of moving from the position in the first video frame to the position in the second video frame.
[0064] Select a point in the video frame image as the original location to establish an image coordinate system, which uses the first direction and the second direction as the positive directions of the two coordinate axes to obtain the position coordinates (x1, y1) of the vehicle in the first video frame and the position coordinates (x2, y2) in the second video frame, and calculate the coordinate offset Δx of the vehicle in the first direction respectively. p and the coordinate offset Δy in the second direction p .
[0065] Specifically, the coordinate offset Δx of the vehicle in the first direction is calculated p The formula is:
[0066] Δx p =|x2-x1|.
[0067] Among them, x1 is the first direction coordinate corresponding to the position of the vehicle in the first video frame, and x2 is the first direction coordinate corresponding to the position of the vehicle in the second video frame.
[0068] Calculate the coordinate offset Δy of the vehicle in the second direction p The formula is:
[0069] Δy p =|y2-y1|.
[0070] Among them, y1 is the second direction coordinate corresponding to the position of the vehicle in the first video frame, and y2 is the second direction coordinate corresponding to the position of the vehicle in the second video frame.
[0071] For example, the current video surveillance includes vehicle A, and the position coordinates of vehicle A in the first video frame are (23, 46), and the position coordinates in the second video frame are (25, 52). In the process of vehicle A moving from the position in the first video frame to the position in the second video frame, the displacement in the first direction is |25-23|=2, and the displacement in the second direction is |52-46|=6.
[0072] Step 122: Convert the coordinate offset of each vehicle in the first direction into an actual offset in the first direction, and convert the coordinate offset of each vehicle in the second direction into an actual offset in the second direction.
[0073] The size of the data in the image is restored to the size of the data in the real space through the ratio of the image to the actual space. The ratio of the image to the actual space can be calculated based on the length and width data of the vehicle detection frame in the video frame and the actual length, width and height data of the vehicle.
[0074] See also Figure 5When the vehicle is traveling longitudinally, the vehicle in the video frame image, such as vehicle S, mainly includes the roof, that is, the top-down angle. The width of the detection frame is the width of the vehicle in the video frame, and the height of the detection frame is the length of the vehicle in the video frame.
[0075] When the vehicle is traveling sideways, the vehicle in the video frame image, such as vehicle R, mainly includes the side of the door, that is, the side view angle. The width of the detection frame is the length of the vehicle in the video frame, and the height of the detection frame is the height of the vehicle in the video frame.
[0076] On this basis, the ratio between the image and the actual size is calculated, and the moving distance on the image can be converted into the actual moving distance according to the calculated ratio.
[0077] Specifically, the formula for converting the coordinate offset of each vehicle in the first direction into the actual offset in the first direction is:
[0078]
[0079] or:
[0080] Among them, d x1 is the actual offset in the first direction when the vehicle is traveling longitudinally, d x2 is the actual offset in the first direction when the vehicle is traveling laterally, Δx p is the coordinate offset of the vehicle in the first direction, Δx r is the conversion ratio of the coordinate offset in the first direction to the actual offset in the first direction, x1 is the first direction coordinate corresponding to the position of the vehicle in the first video frame, x2 is the first direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, W car is the width of the vehicle, and w is the width of the vehicle detection box.
[0081] When the vehicle is traveling longitudinally, the coordinate offset of the vehicle in the first direction is the difference between the first direction coordinate x2 of the vehicle in the second video frame and the first direction coordinate x1 in the first video frame, that is, Δx p =|x2-x1|.
[0082] The ratio between the image and the actual size is the ratio of the detection frame size, i.e., the size of the vehicle in the image, to the actual size of the vehicle. When the vehicle is traveling longitudinally, the vehicle in the video frame image, such as vehicle S, mainly includes the roof, i.e., the top view. At this time, the width w of the detection frame is the width of the vehicle in the video frame, which corresponds to the actual width W of the vehicle. car , the ratio between the image and the actual size is
[0083] When the vehicle is traveling sideways, the vehicle in the video frame, such as vehicle R, mainly includes the side of the door, that is, the side view angle. At this time, the width w of the detection frame is the length of the vehicle in the video frame, which corresponds to the actual length H of the vehicle. car , the ratio between the image and the actual size is
[0084] Finally, multiply the coordinate offset of the vehicle in the video frame by the ratio between the image and the actual size to get:
[0085]
[0086] or,
[0087] For example, there is a vehicle B that is traveling longitudinally. Its position in the first video frame is (32, 11), and its position in the second video frame is (30, 12). First, calculate the coordinate offset Δx of vehicle B in the first direction. p =|x2-x1|=|30-32|=2. The width of the detection box of vehicle B in the video frame is 1.3cm, or 0.013m. The actual width of vehicle B is 1.3m. The ratio between the image and the actual size is Then the actual offset in the first direction when vehicle B is traveling longitudinally is:
[0088]
[0089] For another example, there is a vehicle C that is currently traveling sideways. Its position in the first video frame is (32, 10), and its position in the second video frame is (15, 11). First, calculate the coordinate offset Δx of vehicle C in the first direction: p =|x2-x1|=|15-32|=17. The width of the detection box of vehicle C in the video frame is 3.9 cm, or 0.039 m. The actual length of vehicle C is 3.9 m. The ratio between the image and the actual size is Then the actual offset in the first direction when vehicle C is traveling longitudinally is:
[0090] That is 1700m.
[0091] Similarly, the formula for converting the coordinate offset of each vehicle in the second direction into the actual offset in the second direction is:
[0092]
[0093] or:
[0094] Among them, d y1 is the actual offset in the second direction when the vehicle is traveling longitudinally, dy2 is the actual offset in the second direction when the vehicle is traveling laterally, Δy p is the coordinate offset of the vehicle in the second direction, Δy r is the conversion ratio of the coordinate offset in the second direction to the actual offset in the second direction, y1 is the second direction coordinate corresponding to the position of the vehicle in the first video frame, y2 is the second direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, Vcar is the height of the vehicle, and h is the height of the vehicle detection frame.
[0095] When the vehicle is traveling longitudinally, the coordinate offset of the vehicle in the second direction is the difference between the second direction coordinate y2 of the vehicle in the second video frame and the second direction coordinate y1 in the first video frame, that is, Δy p =|y2-y1|.
[0096] The ratio between the image and the actual size is the size of the detection frame, i.e., the ratio of the size of the vehicle in the image to the actual size of the vehicle. When the vehicle is traveling longitudinally, the vehicle in the video frame image, such as vehicle S, mainly includes the roof, i.e., the top view. At this time, the height h of the detection frame is the length of the vehicle in the video frame, which corresponds to the actual length H of the vehicle. car , the ratio between the image and the actual size is
[0097] When the vehicle is traveling sideways, the vehicle in the video frame, such as vehicle R, mainly includes the side view of the door. At this time, the height h of the detection frame is the height of the vehicle in the video frame, which corresponds to the actual height V of the vehicle. car , the ratio between the image and the actual size is
[0098] Finally, multiply the coordinate offset of the vehicle in the video frame by the ratio between the image and the actual size to get:
[0099]
[0100] or,
[0101] For example, there is a vehicle B that is traveling longitudinally. Its position in the first video frame is (32, 11), and its position in the second video frame is (30, 12). First, calculate the coordinate offset Δy of vehicle B in the second direction. p =|y2-y1|=|12-11|=1. The height of the detection box of vehicle B in the video frame is 3.8cm, or 0.038m. The actual length of vehicle B is 3.8m. The ratio between the image and the actual size is Then the actual offset in the second direction when vehicle B is traveling longitudinally is:
[0102]
[0103] For another example, there is a vehicle D that is currently traveling sideways. Its position in the first video frame is (32, 10), and its position in the second video frame is (15, 8). First, calculate the coordinate offset Δy of vehicle D in the second direction: p =|y2-y1|=|8-10|=2. The height of the detection box of vehicle D in the video frame is 1.6 cm, or 0.016 m. The actual height of vehicle D is 1.6 m. The ratio between the image and the actual size is Then the actual offset in the second direction when the vehicle D is traveling longitudinally is:
[0104]
[0105] Step 123: Calculate the actual offset distance of each vehicle based on the actual offset in the first direction and the actual offset in the second direction.
[0106] The moving distance of the vehicle is calculated as the straight-line distance between the two points, that is, a right triangle is constructed with the actual offset in the first direction and the actual offset in the second direction as right-angled sides, and the hypotenuse of the triangle is the actual offset distance.
[0107] The actual offset distance can be obtained by the formula: To obtain.
[0108] Among them, d car is the actual offset distance of the vehicle, d x is the actual offset distance of the vehicle in the first direction, which can be specifically d x1 or x2 , d y is the actual offset distance of the vehicle in the second direction, which can be specifically d y1 or y2 .
[0109] For example, there is a vehicle B that is traveling longitudinally, and its position in the first video frame is (32, 11), and its position in the second video frame is (30, 12). According to the calculation method in step 122, the actual offset of vehicle B in the first direction is 200m. The actual offset in the second direction is 100m. Then the actual offset generated by vehicle B moving from the position in the first video frame to the position in the second video frame is: About 223.6m.
[0110] Step 124: Divide the actual offset distance of each vehicle by the time difference between the first video frame and the second video frame to obtain the moving speed of the vehicle from the position in the first video frame to the position in the second video frame.
[0111] The moving speed of the vehicle from the position in the first video frame to the position in the second video frame can be expressed as:
[0112] Among them, V car is the moving speed of the vehicle, d car is the actual offset distance of the vehicle, t1 is the acquisition time of the first video frame, and t2 is the acquisition time of the second video frame.
[0113] For example, there is a vehicle E currently moving. The actual offset obtained by the calculation method in step 122 and step 123 is 23.6m. The acquisition time of the first video frame is 0.235s, and the acquisition time of the second video frame is 1.235s. Then, the moving speed of the vehicle E from the position in the first video frame to the position in the second video frame is: That is 23.6m / s.
[0114] Step 20: In response to the proportion of low-speed vehicles being greater than a first threshold and the number of low-speed vehicles being greater than a second threshold, the vehicle detection model detects congested vehicles in the video frame to mark the congested vehicles; wherein the driving speed of the low-speed vehicles is lower than a preset speed threshold.
[0115] The driving speed of each vehicle in the video frame is calculated by the method in step 10, and the driving speed of each vehicle is compared with a preset speed threshold. If the driving speed of the vehicle is lower than the preset speed threshold, the vehicle is considered a low-speed vehicle, and the number of low-speed vehicles in the current video frame is counted.
[0116] Specifically, the preset speed threshold may be 4 m / s, 5 m / s, 7 m / s, 9 m / s, 10 m / s, 13 m / s or 16 m / s, etc.
[0117] The total number of vehicles in the currently acquired video frame is counted, the ratio of the number of low-speed vehicles in the current video frame to the total number of vehicles is calculated, and the calculated ratio is compared with a preset first threshold.
[0118] Specifically, the first threshold may be 45%, 52%, 55%, 60%, 63% or 66%, etc.
[0119] When the proportion of low-speed vehicles is greater than the first threshold and the number of low-speed vehicles is greater than the second threshold, the entire road section enters a low-speed driving state and congestion may occur. The vehicle detection model further compares and detects all vehicles in the current video frame to determine whether there are congested vehicles.
[0120] Specifically, the second threshold may be: 12 vehicles, 13 vehicles, 18 vehicles, 20 vehicles, 25 vehicles, 36 vehicles, 48 vehicles or 66 vehicles, etc.
[0121] Further, see Figure 6 , step 20 further includes:
[0122] Step 21: In response to the proportion of low-speed vehicles being greater than a first threshold and the number of low-speed vehicles being greater than a second threshold, all vehicles in the current video frame are marked as suspected congested vehicles.
[0123] When the current video frame satisfies the condition that the proportion of low-speed vehicles is greater than the first threshold and the number of low-speed vehicles is greater than the second threshold, the entire road section enters a low-speed driving state and congestion may occur. All vehicles appearing in the current video frame are marked as suspected congested vehicles.
[0124] Step 22: Obtain the position of each suspected congested vehicle in the third video frame.
[0125] The position of the suspected congested vehicle is calculated again in the third video frame.
[0126] The time of the third video frame should be at least after the second video frame, that is, after the vehicle is marked as a suspected congestion vehicle.
[0127] Step 23: Mark suspected congested vehicles whose vehicle positions in the third video frame have a similarity greater than a third threshold as congested vehicles.
[0128] The similarity between the position of the suspected congested vehicle in the third video frame and the position of the vehicle when it is marked as a suspected congested vehicle (the second video frame) is calculated by a similarity algorithm. If the position similarity is greater than a preset third threshold, the vehicle may be a congested vehicle and the vehicle is marked as a congested vehicle.
[0129] Specifically, the third threshold may be 60%, 65%, 73%, 82%, 85%, 89%, 90% or 94%, etc.
[0130] By intercepting the third video frame, a preliminary check is performed on the speed detection results and congestion judgment results to determine that the congested vehicles have moved slightly within a time period, which is beneficial to improving the accuracy of road congestion detection.
[0131] Step 30: Use the multimodal large model to review the feature consistency of the congested vehicles, and generate warning information for the congested vehicles that have passed the review.
[0132] The information of congested vehicles is input into the multimodal large model for feature consistency review, that is, two video frames at different times, such as the first video frame and the third video frame or the second video frame and the third video frame, are taken to extract and classify the vehicle features to ensure that the vehicles identified at different times are the same vehicle.
[0133] If it is determined that the vehicles in different video frames are the same vehicle, the vehicle passes the review and is a congested vehicle. Alarm information is generated based on the vehicle information and the corresponding road information, including vehicle ID, congestion location, timestamp, etc., so that traffic management personnel can follow up on the congested road conditions based on the alarm information.
[0134] The model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model.
[0135] The memory, computing power, and number of parameters of the multimodal large model are greater than those of the vehicle detection model. In comparison, the multimodal large model is a large model, and the vehicle detection model is a small model.
[0136] Large models can perform deeper learning and calculations and process more complex image information. Small models have poorer processing capabilities, but have faster processing speeds, are simple to train, and have low costs.
[0137] The detection results of the small model are simply reviewed with the multimodal large model to eliminate the misdetection of the small model, thereby improving the accuracy of congestion detection.
[0138] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of the storage medium provided by the present application.
[0139] The storage medium 300 stores program data 310. When the program data 310 is executed by the processor, the following is achieved: Figure 1 The steps of the road congestion detection method are described.
[0140] The program data 310 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.
[0141] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.
[0142] See also Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of a computer device provided by the present application.
[0143] The computer device 400 includes a processor 420 and a memory 410 which are connected to each other. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the road congestion detection method described above are implemented.
[0144] See also Fig. 9, Fig. 9 1 is a schematic diagram of the structure of an embodiment of a road congestion detection system provided in the present application. The road congestion detection system 1000 includes a camera 500 and a computer device 400 as described above.
[0145] Camera 500 is installed above the road. It is a camera device specially used to monitor target objects such as vehicles, drivers, pedestrians and landmarks in motion on the traffic road. It includes but is not limited to gun cameras and smart ball cameras. It realizes the monitoring, switching and control functions of network users by digitally encoding and compressing video images and then outputting them.
[0146] The camera 500 is used to continuously acquire video frames and send them to the computer device 400 via a communication connection capable of transmitting video information, and extract the video frames for image processing.
[0147] The computer device 400 receives the video frames sent by the camera 500 to execute the road congestion detection method as described above.
[0148] Different from the prior art, the present application provides a road congestion detection method, medium, device and system, which uses a small vehicle detection model to make a preliminary congestion judgment, only calculates simple parameters, occupies little memory, has fast processing speed and low training cost. Then, the detection results are reviewed through a multimodal large model to exclude the results of small model detection errors and issue an alarm, which makes up for the low detection accuracy of the small model and improves the detection accuracy of road congestion.
[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0150] The present application can be used in many general or special computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.
[0151] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.
[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0154] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A road congestion detection method based on a video hybrid model, characterized in that: include: Using the continuously acquired video frames as input to a vehicle detection model to detect the driving speed of each vehicle in the video frames; In response to the proportion of low-speed vehicles being greater than a first threshold and the number of the low-speed vehicles being greater than a second threshold, the vehicle detection model detects congested vehicles in the video frame to mark the congested vehicles; wherein the travel speed of the low-speed vehicles is lower than a preset speed threshold; A multimodal large model is used to review the feature consistency of the congested vehicle, and alarm information of the congested vehicle that has passed the review is generated; wherein the model parameters of the multimodal large model are greater than the model parameters of the vehicle detection model.
2. The road congestion detection method according to claim 1, characterized in that: The continuously acquired video frames are used as input of the vehicle detection model to detect the driving speed of each vehicle in the video frames, including: Using the continuously acquired video frames as input to the vehicle detection model to obtain the position of each vehicle in the first video frame and the second video frame; The moving speed of each vehicle from the position in the first video frame to the position in the second video frame is calculated.
3. The road congestion detection method according to claim 2, characterized in that: The method of using the continuously acquired video frames as input of the vehicle detection model to obtain the position of each vehicle in the first video frame and the second video frame includes: Using the continuously acquired video frames as input to the vehicle detection model to obtain historical bounding box coordinates of each vehicle and a vehicle image of each vehicle in the first video frame or the second video frame; Building a motion model for each vehicle based on the historical bounding box coordinates of each vehicle to predict the predicted coordinates of each vehicle in the first video frame or the second video frame; The predicted coordinates are matched with a current image of each vehicle to obtain a current bounding box coordinate of each vehicle in the current image as a position of each vehicle in the first video frame or the second video frame.
4. The road congestion detection method according to claim 2, characterized in that: The calculating the moving speed of each vehicle from the position in the first video frame to the position in the second video frame includes: Respectively calculating a coordinate offset in a first direction and a coordinate offset in a second direction of each vehicle in a process of moving from a position in the first video frame to a position in the second video frame; Convert the coordinate offset of each vehicle in the first direction into the actual offset in the first direction, and convert the coordinate offset of each vehicle in the second direction into the actual offset in the second direction; Calculating an actual offset distance of each vehicle based on an actual offset in the first direction and an actual offset in the second direction; The actual offset distance of each vehicle is divided by the time difference between the first video frame and the second video frame to obtain the moving speed of the vehicle from the position in the first video frame to the position in the second video frame.
5. The road congestion detection method according to claim 4, characterized in that: The formula for converting the coordinate offset of each vehicle in the first direction into the actual offset in the first direction is: or: Among them, d x1 is the actual offset in the first direction when the vehicle is traveling longitudinally, d x2 is the actual offset in the first direction when the vehicle is traveling laterally, Δx p is the coordinate offset of the vehicle in the first direction, Δx r is the conversion ratio of the coordinate offset in the first direction to the actual offset in the first direction, x1 is the first direction coordinate corresponding to the position of the vehicle in the first video frame, x2 is the first direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, Wcar is the width of the vehicle, and w is the width of the vehicle detection box.
6. The road congestion detection method according to claim 4, characterized in that: The formula for converting the coordinate offset of each vehicle in the second direction into the actual offset in the second direction is: or: Among them, d y1 is the actual offset in the second direction when the vehicle is traveling longitudinally, d y2 is the actual offset in the second direction when the vehicle is traveling laterally, Δy p is the coordinate offset of the vehicle in the second direction, Δy r is the conversion ratio of the coordinate offset in the second direction to the actual offset in the second direction, y1 is the second direction coordinate corresponding to the position of the vehicle in the first video frame, y2 is the second direction coordinate corresponding to the position of the vehicle in the second video frame, H car is the length of the vehicle, Vcar is the height of the vehicle, and h is the height of the vehicle detection frame.
7. The road congestion detection method according to claim 1, characterized in that: In response to the proportion of low-speed vehicles being greater than a first threshold and the number of low-speed vehicles being greater than a second threshold, the vehicle detection model detects congested vehicles in the video frame to mark the congested vehicles, including: In response to the proportion of low-speed vehicles being greater than a first threshold and the number of the low-speed vehicles being greater than a second threshold, marking all vehicles in the current video frame as suspected congested vehicles; Obtaining the position of each suspected congested vehicle in the third video frame; The suspected congested vehicles whose vehicle positions in the third video frame have a similarity greater than a third threshold to the vehicle positions in the second video frame are marked as congested vehicles.
8. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the road congestion detection method as described in any one of claims 1 to 7 are implemented.
9. A computer device, characterized in that: The method comprises a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the road congestion detection method according to any one of claims 1 to 7 are implemented.
10. A road congestion detection system, characterized in that: comprising a camera and a computer device as claimed in claim 9; The camera is installed above the road and is used to continuously acquire video frames and send them to the computer device; The computer device receives the video frames sent by the camera to execute the road congestion detection method as described in any one of claims 1-7.