Road traffic state detection method and device based on panoramic monitoring video analysis

Abstract

The application provides a road traffic state detection method and device based on panoramic monitoring video analysis, at least two detection areas are configured on each road in a video picture; a real-time video stream is acquired, and the real-time video stream is decoded to obtain a video frame image; a detection tracking model is used for vehicle detection tracking on each detection area in the video frame image, to obtain vehicle detection tracking information in each detection area; the number of vehicles and the speed of vehicles in each detection area are calculated according to the vehicle detection tracking information in each detection area; the traffic state of each road is judged according to a preset judgment rule and the number of vehicles and the speed of vehicles in each detection area; wherein, the traffic state of each road reflects the current congestion situation of each road. The application scheme realizes the discovery of traffic congestion from a wider field of view, thereby assisting the road traffic management department to timely deal with traffic congestion events.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a method and apparatus for detecting road traffic conditions using panoramic monitoring video analysis. Background Technology

[0002] With rapid socio-economic growth and continuous improvement in productivity, people's living standards have significantly improved, and the era of motorization has long been established, leading to a surge in the number of motor vehicles. While this substantial increase in vehicles has brought numerous conveniences, it has also triggered a series of traffic problems. Severe road congestion is hindering the rapid urbanization process and easily triggers traffic accidents and other traffic incidents. Especially during morning and evening rush hours and around holidays, congestion is more likely to occur due to large-scale public travel. Therefore, the timely detection and handling of sudden traffic congestion incidents by road traffic management departments during morning and evening rush hours is of great significance for the safe and normal operation of the entire city's roads.

[0003] With the increasing application of high-definition panoramic surveillance in key roads such as overpasses, viaducts, and main roads, the wide field of view and high resolution of high-definition panoramic surveillance can quickly detect abnormal traffic conditions from a wider perspective, find the cause of road problems, and promptly guide traffic to restore normal traffic order and avoid causing greater traffic problems.

[0004] Therefore, real-time urban road congestion status recognition based on high-definition panoramic video analysis is the future development and application direction of urban road traffic situation recognition. The rich data it provides is an effective guarantee for realizing intelligent traffic management and control. Summary of the Invention

[0005] This invention provides a road traffic condition detection method and device based on panoramic surveillance video analysis, which can accurately detect and track small target vehicles in panoramic surveillance videos, enabling the discovery of traffic congestion from a wider field of view, thereby assisting road traffic management departments in timely handling of traffic congestion events.

[0006] In a first aspect, the present invention provides a road traffic condition detection method based on panoramic surveillance video analysis, which utilizes road video captured by a high-definition panoramic surveillance device mounted at a height of tens of meters to detect traffic conditions, including:

[0007] Configure at least two detection zones on each road in the video footage;

[0008] Acquire a real-time video stream and decode the real-time video stream to obtain video frame images;

[0009] The detection and tracking model is used to perform vehicle detection and tracking in each detection region of the video frame image to obtain vehicle detection and tracking information in each detection region; wherein, the vehicle detection and tracking information includes the vehicle's OBB detection box information and ID information;

[0010] The number of vehicles and their speed in each detection area are calculated based on the vehicle detection and tracking information in each detection area.

[0011] The traffic status of each road is determined based on preset judgment rules and the number and speed of vehicles in each detection area; wherein, the traffic status of each road reflects the current congestion situation of each road.

[0012] In an optional embodiment, configuring at least two detection areas on each road in the video frame includes:

[0013] Mark at least two polygonal regions on each road in the video footage;

[0014] By traversing the vertices of each polygonal region, the minimum bounding quadrilateral bounding box of each polygonal region is constructed. The image region enclosed by the minimum bounding quadrilateral bounding box is the detection region.

[0015] In an optional embodiment, the step of using a detection and tracking model to perform vehicle detection and tracking on each detection region in the video frame image to obtain vehicle detection and tracking information in each detection region includes:

[0016] The improved Yolov5 model, which is pre-trained, is used to detect vehicles in the image regions enclosed by the minimum bounding quadrilaterals in each video frame image, and obtains the AABB detection box and candidate OBB detection box corresponding to each vehicle in each detection region. The improved Yolov5 model is specifically an OBB detection box generation module added to the output of the original Yolov5 model.

[0017] The candidate OBB detection boxes corresponding to each vehicle in each detection area are filtered to obtain the final OBB detection boxes corresponding to each vehicle in each detection area.

[0018] For each detection area, a corresponding AABB bounding box is constructed, and the AABB bounding box of each detection area is input into the Sort tracking model to track each detected vehicle and determine the vehicle ID information of each detection area.

[0019] Further, the step of filtering the candidate OBB detection boxes corresponding to each vehicle in each detection area to obtain the final OBB detection boxes corresponding to each vehicle in each detection area includes:

[0020] Remove candidate OBB detection boxes whose confidence scores are lower than a preset score threshold in each detection area corresponding to each vehicle, and sort the remaining candidate OBB detection boxes in each detection area according to their confidence scores to obtain the candidate OBB detection box sequence in each detection area.

[0021] The remaining candidate vehicle OBB detection boxes and corresponding AABB detection boxes in each detection area are used to perform AABB collision and OBB collision respectively. The Poly-NMS algorithm is then used to filter the candidate OBB detection box sequences in each detection area after the collision to obtain the final OBB detection boxes for each detection area.

[0022] In an optional embodiment, calculating the number of vehicles and vehicle speed in each detection area based on the vehicle detection and tracking information in each detection area includes:

[0023] The number of vehicles in each detection area at the current moment is determined based on the final OBB detection frame of each detection area;

[0024] The speed of each vehicle in each detection area at the current moment is calculated based on the final OBB detection frame and ID information of the vehicles in each detection area within the preset time period.

[0025] The average speed of vehicles in each detection area at the current moment is calculated based on the number of vehicles in each detection area at the current moment and the speed of each vehicle.

[0026] Further, the step of calculating the speed of each vehicle in each detection area at the current moment based on the final OBB detection frame and ID information of vehicles in each detection area within a preset time period includes:

[0027] The final OBB detection frame for each vehicle within a preset time period is determined based on the vehicle ID information in each detection area.

[0028] The displacement of each vehicle within the preset time period is calculated using the center point of the final OBB detection frame corresponding to each vehicle in each detection area within the preset time period.

[0029] The speed of each vehicle in each detection area at the current moment is calculated using the displacement of each vehicle within a preset time period.

[0030] In an optional embodiment, determining the traffic status of each road based on preset judgment rules and the number and speed of vehicles in each detection area includes:

[0031] The number of vehicles and the speed of vehicles in each detection area are assigned different weight values, and the congestion parameter value of each detection area is calculated based on the weight values ​​corresponding to the number of vehicles and the speed of vehicles in each detection area. The congestion parameter value is the sum of the product of the number of vehicles and the speed of vehicles and the corresponding weight values.

[0032] Determine whether the congestion parameter values ​​in each detection area are greater than the preset parameter threshold;

[0033] If so, the traffic condition of the detection area is determined to be congested; otherwise, the traffic condition of the detection area is determined to be unobstructed.

[0034] The traffic status of each road is determined according to a preset judgment rule. The preset judgment rule is as follows: determine the total number of detection areas configured for the first road and the number of detection areas in a congested state; if the number of detection areas configured for the road is equal to 2 and each detection area is in a congested state, then the traffic status of the road is congested; if the number of detection areas configured for the road is greater than 2 and the proportion of the number of detection areas in a congested state to the total number of detection areas is greater than two-thirds, then the traffic status of the road is congested; continue the judgment process for the next road until the traffic status of all roads has been determined.

[0035] Secondly, the present invention provides a road traffic condition detection device based on panoramic surveillance video analysis, comprising:

[0036] The configuration module is used to configure at least two detection areas on each road in the video frame;

[0037] A decoding module is used to acquire a real-time video stream and decode the real-time video stream to obtain video frame images;

[0038] The vehicle detection and tracking module is used to perform vehicle detection and tracking on each detection region in the video frame image using a detection and tracking model, and to obtain vehicle detection and tracking information in each detection region; wherein, the vehicle detection and tracking information includes the vehicle's OBB detection box information and ID information;

[0039] The calculation module is used to calculate the number of vehicles and the vehicle speed in each detection area based on the vehicle detection and tracking information in each detection area.

[0040] The determination module is used to determine the traffic status of each road based on preset determination rules and the number and speed of vehicles in each detection area; wherein, the traffic status of each road reflects the current congestion situation of each road.

[0041] Thirdly, the present invention provides a computer device, comprising: at least one processor and a memory;

[0042] The memory stores computer-executed instructions;

[0043] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the road traffic state detection method according to any one of the first aspects.

[0044] This invention provides a road traffic condition detection method and apparatus based on panoramic surveillance video analysis. It utilizes high-definition panoramic surveillance equipment mounted at a height of tens of meters to capture road video for traffic condition detection, configuring at least two detection areas on each road in the video frame. The method acquires a real-time video stream and decodes it to obtain video frame images. A detection and tracking model is used to detect and track vehicles in each detection area of ​​the video frame images, obtaining vehicle detection and tracking information for each detection area. This vehicle detection and tracking information includes the vehicle's OBB detection frame information and ID information. The method calculates the number of vehicles and their speed in each detection area based on the vehicle detection and tracking information. Finally, it determines the traffic condition of each road based on preset judgment rules and the number and speed of vehicles in each detection area. The traffic condition of each road reflects its current congestion situation. Compared with existing technologies, this application can accurately detect and track small target vehicles in panoramic surveillance videos, enabling the discovery of traffic congestion from a wider field of view, thereby assisting road traffic management departments in timely handling of traffic congestion events. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of a scenario architecture upon which this disclosure is based;

[0047] Figure 2 A flowchart illustrating a road traffic condition detection method based on panoramic surveillance video analysis provided in this embodiment of the present disclosure;

[0048] Figure 3 A schematic flowchart of the vehicle detection and tracking method provided in this embodiment of the disclosure;

[0049] Figure 4 A schematic flowchart illustrating the road traffic condition determination method provided in this embodiment of the disclosure;

[0050] Figure 5 To configure the detection area effect diagram;

[0051] Figure 6 A schematic diagram of the structure of a road traffic condition detection device based on panoramic monitoring video analysis provided in this embodiment of the present disclosure;

[0052] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Currently, high-definition panoramic monitoring equipment installed at a height of tens of meters has a wide shooting angle and can capture large-scale road traffic videos. It can monitor hundreds of vehicle targets and is increasingly being used on key roads such as overpasses, viaducts, and main roads. However, traffic management departments currently mainly determine traffic congestion events by manually reviewing panoramic monitoring images, and cannot automatically detect traffic congestion events.

[0055] With the development of communication technology, deep learning technology, graphics and image processing technology, and the improvement of monitoring equipment hardware performance, using intelligent technology to process videos captured by panoramic monitoring equipment and automatically determine traffic congestion has become an important means of solving existing technical problems.

[0056] Figure 1 This is a schematic diagram of a scenario architecture upon which this disclosure is based, such as... Figure 1 As shown, the scenario architecture based on this disclosure may include a road traffic condition detection device 1 and a panoramic monitoring device 2.

[0057] The road traffic condition detection device 1 is hardware or software that can interact with the panoramic monitoring device 2 via a network, and it can be used to execute the road traffic condition detection methods described in the following embodiments.

[0058] When the road traffic condition detection device 1 is hardware, it can be an electronic device with computing capabilities. When the road traffic condition detection device 1 is software, it can be installed in an electronic device with computing capabilities. Such electronic devices include, but are not limited to, servers, laptops, and desktop computers.

[0059] The panoramic monitoring device 2 can be a hardware device with a wide-range scene shooting function, such as a bullet camera or a PTZ camera. The road traffic condition detection device 1 can be a server integrated or installed on the panoramic monitoring device 2. The road traffic condition detection device 1 can run on the panoramic monitoring device 2 and provide road traffic condition detection services for the panoramic monitoring device 2. Furthermore, the road traffic condition detection device 1 uses its display component to display the road traffic condition detection results to the user.

[0060] Furthermore, the road traffic condition detection device 1 can also be integrated into a server for processing vehicle videos. In this case, the panoramic monitoring device 2 is a device that can communicate and exchange data with the aforementioned road traffic condition detection device 1 via a network. The panoramic monitoring device 2 can then send real-time video streams to the road traffic condition detection device 1, enabling the road traffic condition detection device 1 to perform road traffic condition detection on the real-time video stream using the method shown below.

[0061] The following will further describe the road traffic condition detection method and device based on panoramic surveillance video analysis provided in this application:

[0062] Figure 2 This is a flowchart illustrating a road traffic condition detection method based on panoramic surveillance video analysis, provided as an embodiment of this disclosure. Figure 2 As shown in the embodiments of this disclosure, a road traffic condition detection method based on panoramic surveillance video analysis is provided. This method utilizes road video captured by a high-definition panoramic surveillance device mounted at a height of tens of meters to detect traffic conditions, including:

[0063] S21. Configure at least two detection areas on each road in the video frame;

[0064] The detection area is the target area in the video frame that needs to be detected, and the detection area can be a polygonal area.

[0065] In this embodiment, since it is difficult to detect and track all vehicles in the video, it is possible to detect and track only vehicles in specific areas of the road in the video. Then, the road traffic status is determined based on the detection and tracking results of vehicles in specific areas of the road. The detection area configured in the video is generally a polygonal area, while the inference operation uses a rectangle. Therefore, it is necessary to construct the minimum bounding rectangle bounding box for each polygonal area. Thus, at least two polygonal areas are marked on each road in the video. The minimum bounding quadrilateral bounding box of each polygonal area is constructed by traversing the vertices of each polygonal area. The image area enclosed by the minimum bounding quadrilateral bounding box is the detection area.

[0066] Specifically, when configuring the detection region in the video frame, the video player displays the video frame where the detection region needs to be configured. A frame drawing plugin is used to draw wireframes on various paths in the video frame as required. These wireframes are polygonal structures. The minimum bounding quadrilateral (MBT) of each wireframe is constructed by traversing its vertices. The area enclosed by the MBT is the detection region. The position of the configured detection region in the video frame is fixed.

[0067] S22. Acquire the real-time video stream and decode the real-time video stream to obtain video frame images.

[0068] In this embodiment, after configuring the detection area, GPU decoding can be used to achieve real-time decoding.

[0069] S23. Using a detection and tracking model, vehicle detection and tracking are performed on each detection region in the video frame image to obtain vehicle detection and tracking information in each detection region; wherein, the vehicle detection and tracking information includes the vehicle's OBB detection box and ID information.

[0070] Among them, the vehicle's OBB detection frame is the detection frame closest to the vehicle and has a direction, and the vehicle's ID information is a unique identification number assigned to the detected vehicle.

[0071] In this embodiment, a vehicle detection model can be used first to detect vehicles in each detection region of the image. The vehicle detection model includes at least one of YOLO, SSD, and Faster-RCNN. Then, the two-dimensional vehicle detection boxes generated by the vehicle detection model are combined with the SORT multi-object tracking model to track the detected vehicle targets until the vehicles leave the detection region. Finally, the OBB detection boxes and ID information of vehicles in each detection region of each frame image are obtained.

[0072] S24. Calculate the number of vehicles and vehicle speed in each detection area based on the vehicle detection and tracking information in each detection area.

[0073] In this embodiment, since the number of vehicles and vehicle speed can reflect the road congestion, in order to determine whether traffic congestion has occurred in each detection area, the number of vehicles and vehicle speed in each detection area can be calculated using the vehicle detection and tracking information in each detection area, and then the traffic status of each detection area can be determined using the number of vehicles and vehicle speed in each detection area.

[0074] S25. Determine the traffic status of each road according to the preset judgment rules and the number and speed of vehicles in each detection area; wherein, the traffic status of each road reflects the current congestion situation of each road.

[0075] In this embodiment, in order to determine the congestion situation of each road in the video in real time, the number of vehicles and vehicle speed in each detection area in each frame of the image are used to determine the congestion situation of each detection area. Then, based on the judgment result of the congestion situation of the detection area on each road and the preset judgment rule, the traffic status of each road is determined, and the congestion situation of each road can be determined.

[0076] This embodiment provides a road traffic condition detection method based on panoramic surveillance video analysis. It utilizes road video captured by a high-definition panoramic surveillance device mounted at a height of tens of meters for traffic condition detection, configuring at least two detection areas on each road in the video frame. Real-time video streams are acquired and decoded to obtain video frame images. A detection and tracking model is used to detect and track vehicles in each detection area of ​​the video frame images, obtaining vehicle detection and tracking information for each detection area. This vehicle detection and tracking information includes the vehicle's OBB detection frame information and ID information. The number of vehicles and their speeds in each detection area are calculated based on the vehicle detection and tracking information. The traffic condition of each road is determined according to preset judgment rules and the number and speed of vehicles in each detection area. The traffic condition of each road reflects its current congestion situation. Compared with existing technologies, this application can accurately detect and track small target vehicles in panoramic surveillance videos, enabling the discovery of traffic congestion from a wider field of view, thereby assisting road traffic management departments in timely handling of traffic congestion events.

[0077] To accurately obtain the OBB detection bounding box information and ID information of vehicles in each detection region of each frame image, in Figure 2 Based on the aforementioned embodiments Figure 3 This is a flowchart illustrating the vehicle detection and tracking method provided in this embodiment. The method provided in this embodiment further explains step S23 in the foregoing embodiment. S23 includes:

[0078] S231. Using a pre-trained improved Yolov5 model, vehicles are simultaneously detected in the image regions enclosed by the minimum bounding quadrilaterals in each video frame image, resulting in AABB detection boxes and candidate OBB detection boxes for each vehicle in each detection region; the improved Yolov5 model specifically adds an OBB detection box generation module to the output of the original Yolov5 model.

[0079] In this embodiment, since the detection boxes generated by the original YOLOv5 model after detecting vehicles are AABB detection boxes, when there are many vehicles, the AABB detection boxes corresponding to each vehicle are prone to overlap. When filtering the detection boxes, many correct detection boxes are filtered out. An OBB detection box generation module can be added to the output of the original YOLOv5 model to transform the AABB detection boxes output by the original YOLOv5 model into OBB detection boxes. The overlap of the OBB detection boxes after transformation is greatly reduced. Filtering the OBB detection boxes can retain more correct detection boxes and significantly improve the detection rate.

[0080] S232. Filter the candidate OBB detection boxes corresponding to each vehicle in each detection area to obtain the final OBB detection boxes corresponding to each vehicle in each detection area.

[0081] In this embodiment, since the OBB detection box still contains redundant detection boxes, these detection boxes need to be filtered to remove duplicate and erroneous OBB detection boxes. The OBB detection boxes can be filtered using methods such as OBB detection box confidence score, AABB collision, OBB collision, and nonmaximum suppression.

[0082] Specifically, the method for filtering candidate OBB detection boxes may include: removing candidate OBB detection boxes whose confidence scores are lower than a preset score threshold for each vehicle in each detection region; sorting the remaining candidate OBB detection boxes in each detection region according to their confidence scores to obtain a sequence of candidate OBB detection boxes in each detection region; performing AABB collision and OBB collision using the remaining candidate vehicle OBB detection boxes and their corresponding AABB detection boxes in each detection region; and using the Poly-NMS algorithm to filter the sequence of candidate OBB detection boxes in each detection region after the collision to obtain the final OBB detection boxes for each detection region.

[0083] S233. Construct corresponding AABB bounding boxes for the final OBB detection boxes of each detection area, and input the AABB bounding boxes of each detection area into the Sort tracking model to track each detected vehicle and determine the vehicle ID information of each detection area.

[0084] In this embodiment, since the SORT tracking model is designed based on AABB bounding boxes, tracking chain breaks frequently occur with OBB bounding boxes, resulting in poor tracking accuracy. Therefore, AABB bounding boxes are still used to perform SORT multi-target tracking. During tracking, an AABB bounding box is first constructed for each OBB bounding box using the detection module. This AABB bounding box is then fed into the SORT tracking model for tracking. The main purpose of tracking is to determine the vehicle ID, which is used to associate vehicle targets in previous and subsequent frames to calculate vehicle speed.

[0085] To calculate the number of vehicles and vehicle speed in each detection area, this embodiment further explains step S24 in the foregoing embodiment, which includes:

[0086] S241. Determine the number of vehicles in each detection area at the current moment based on the final OBB detection frame of each detection area.

[0087] In this embodiment, the number of final OBB detection boxes in each detection region of the current image is the number of vehicles in each detection region at the current moment.

[0088] S242. Calculate the speed of each vehicle in each detection area at the current moment based on the final OBB detection frame and ID information of the vehicles in each detection area within the preset time period.

[0089] In this embodiment, the relative speed of each vehicle is calculated using the center point of the OBB detection frame of each vehicle, rather than the center point of the AABB detection frame, which provides a more accurate relative speed.

[0090] Specifically, the method for calculating the speed of each vehicle may include: determining the final OBB detection frame corresponding to each vehicle within a preset time period based on the vehicle ID information in each detection area; calculating the displacement of each vehicle within the preset time period using the center point of the final OBB detection frame corresponding to each vehicle in each detection area within the preset time period; and calculating the speed of each vehicle in each detection area at the current moment using the displacement of each vehicle within the preset time period.

[0091] S243. Calculate the average speed of vehicles in each detection area at the current moment based on the number of vehicles in each detection area and the speed of each vehicle.

[0092] In this embodiment, the average speed of vehicles in each detection area can be obtained by calculating the ratio of the sum of the speeds of each vehicle in each detection area to the number of vehicles at the current moment.

[0093] Based on the number of vehicles and their speeds in each detection area, the traffic conditions of each road can be further determined. Figure 2 Based on the aforementioned embodiments Figure 4 This is a flowchart illustrating the road traffic state determination method provided in this embodiment. This embodiment further explains step S25 in the foregoing embodiments, which includes:

[0094] S251. Assign different weight values ​​to the number of vehicles and the speed of vehicles in each detection area, and calculate the congestion parameter value of each detection area according to the weight value corresponding to the number of vehicles and the speed of vehicles in each detection area. The congestion parameter value is the sum of the number of vehicles and the speed of vehicles multiplied by the corresponding weight value.

[0095] S252. Determine whether the congestion parameter value of each detection area is greater than the preset parameter threshold.

[0096] S253. If yes, the traffic status of the detection area is determined to be congested; otherwise, the traffic status of the detection area is determined to be unobstructed.

[0097] S254. Determine the traffic status of each road according to preset judgment rules. The preset judgment rules are as follows: determine the total number of detection areas configured on the first road and the number of detection areas in a congested state; if the number of detection areas configured on the road is equal to 2, and each detection area is in a congested state, then the traffic status of the road is congested; if the number of detection areas configured on the road is greater than 2, and the proportion of the number of detection areas in a congested state to the total number of detection areas is greater than two-thirds, then the traffic status of the road is congested; continue the judgment process for the next road until the traffic status of all roads has been determined.

[0098] In this embodiment, the congestion parameter value of each detection area can be calculated based on the number of vehicles and vehicle speed in each detection area. The traffic status of each detection area can be determined based on the congestion parameter value of each detection area. The traffic status of each road can be determined based on the traffic status of the detection areas on each road.

[0099] The following specific examples are provided in conjunction with any of the above embodiments. It should be understood that the following specific examples are merely illustrative of the specific implementation of any of the above embodiments and are not intended to limit the technical solutions of any of the above embodiments.

[0100] In the following specific examples, such as Figure 5As shown, the video frame includes two roads, R1 and R2, which intersect. One detection region (R1-A1 and R1-A2) is configured on each side of the intersection point of road R1, and another detection region (R2-A1 and R2-A2) is configured on each side of the intersection point of road R2. The real-time video stream is acquired and decoded to obtain video frame images, which are then input into the detection and tracking model. The YOLOv5 model is used to detect vehicles in the R1-A1 region of the first frame, resulting in 110 AABB detection boxes and 110 candidate OBB detection boxes, numbered 1-110. The 110 candidate OBB detection boxes are filtered to obtain 100 final OBB detection boxes and their corresponding AABB detection boxes. The YOLOv5 model is then used to detect vehicles in the following frames. A single frame of image is used, combined with 100 AABB detection boxes obtained from the previous frame, and the SORT multi-object tracking model is used to track the detected vehicle targets. The newly obtained candidate OBB detection boxes are filtered to obtain 105 final OBB detection boxes and their corresponding AABB detection boxes. Among them, 100 cars are those already detected in the first frame, and the other 5 cars are those that have just entered the R1-A1 area. The 100 previously detected cars are numbered 1-100, and the other 5 cars are numbered 101-105. The average speed of the 105 cars is calculated to be 10. Therefore, the number of vehicles in the R1-A1 area at the current moment is 105 and the vehicle speed is 10. The weights of the number of vehicles and the vehicle speed are set to 0.6 and 0.4, respectively. Therefore, the congestion parameter value of the R1-A1 area is 105*0.6+10*0.4=67. Similarly, while processing the R1-A1 area, the R1-A2, R2-A1, and R2-A2 areas are also processed, resulting in congestion parameter values ​​of 64, 42, and 46 for the R1-A2, R2-A1, and R2-A2 areas, respectively. Based on the preset parameter threshold of 50, the traffic status of the R1-A1 and R1-A2 areas is determined to be congested, while the traffic status of the R2-A1 and R2-A2 areas is determined to be unimpeded. Thus, according to the preset judgment rules, the traffic status of road R1 is determined to be congested and unimpeded.

[0101] Corresponding to the road traffic condition detection method based on panoramic surveillance video analysis in the above embodiment, Figure 6 This is a schematic diagram of a road traffic condition detection device based on panoramic surveillance video analysis, provided as an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 6 The road traffic condition detection device based on panoramic surveillance video analysis includes:

[0102] Configuration module 61 is used to configure at least two detection areas on each road in the video frame;

[0103] Decoding module 62 is used to acquire real-time video stream and decode the real-time video stream to obtain video frame images;

[0104] The vehicle detection and tracking module 63 is used to perform vehicle detection and tracking on each detection area in the video frame image using a detection and tracking model, and to obtain vehicle detection and tracking information in each detection area; wherein, the vehicle detection and tracking information includes the vehicle's OBB detection box information and ID information;

[0105] The calculation module 64 is used to calculate the number of vehicles and the vehicle speed in each detection area based on the vehicle detection and tracking information in each detection area.

[0106] The determination module 65 is used to determine the traffic status of each road based on preset determination rules and the number and speed of vehicles in each detection area; wherein the traffic status of each road reflects the current congestion situation of each road.

[0107] Optionally, the configuration module 61 is specifically used for:

[0108] Mark at least two polygonal regions on each road in the video footage;

[0109] By traversing the vertices of each polygonal region, the minimum bounding quadrilateral bounding box of each polygonal region is constructed. The image region enclosed by the minimum bounding quadrilateral bounding box is the detection region.

[0110] Optionally, the vehicle detection and tracking module 63 is specifically used for:

[0111] The improved Yolov5 model, which is pre-trained, is used to detect vehicles in the image regions enclosed by the minimum bounding quadrilaterals in each video frame image, and obtains the AABB detection box and candidate OBB detection box corresponding to each vehicle in each detection region. The improved Yolov5 model is specifically an OBB detection box generation module added to the output of the original Yolov5 model.

[0112] The candidate OBB detection boxes corresponding to each vehicle in each detection area are filtered to obtain the final OBB detection boxes corresponding to each vehicle in each detection area.

[0113] For each detection area, a corresponding AABB bounding box is constructed, and the AABB bounding box of each detection area is input into the Sort tracking model to track each detected vehicle and determine the vehicle ID information of each detection area.

[0114] Furthermore, the vehicle detection and tracking module 63 is specifically used for:

[0115] Remove candidate OBB detection boxes whose confidence scores are lower than a preset score threshold in each detection area corresponding to each vehicle, and sort the remaining candidate OBB detection boxes in each detection area according to their confidence scores to obtain the candidate OBB detection box sequence in each detection area.

[0116] The remaining candidate vehicle OBB detection boxes and corresponding AABB detection boxes in each detection area are used to perform AABB collision and OBB collision respectively. The Poly-NMS algorithm is then used to filter the candidate OBB detection box sequences in each detection area after the collision to obtain the final OBB detection boxes for each detection area.

[0117] Optionally, the calculation module 64 is specifically used for:

[0118] The number of vehicles in each detection area at the current moment is determined based on the final OBB detection frame of each detection area;

[0119] The speed of each vehicle in each detection area at the current moment is calculated based on the final OBB detection frame and ID information of the vehicles in each detection area within the preset time period.

[0120] The average speed of vehicles in each detection area at the current moment is calculated based on the number of vehicles in each detection area at the current moment and the speed of each vehicle.

[0121] Furthermore, the computing module 64 is specifically used for:

[0122] The final OBB detection frame for each vehicle within a preset time period is determined based on the vehicle ID information in each detection area.

[0123] The displacement of each vehicle within the preset time period is calculated using the center point of the final OBB detection frame corresponding to each vehicle in each detection area within the preset time period.

[0124] The speed of each vehicle in each detection area at the current moment is calculated using the displacement of each vehicle within a preset time period.

[0125] Optionally, the determination module 65 is specifically used for:

[0126] The number of vehicles and the speed of vehicles in each detection area are assigned different weight values, and the congestion parameter value of each detection area is calculated based on the weight values ​​corresponding to the number of vehicles and the speed of vehicles in each detection area. The congestion parameter value is the sum of the product of the number of vehicles and the speed of vehicles and the corresponding weight values.

[0127] Determine whether the congestion parameter values ​​in each detection area are greater than the preset parameter threshold;

[0128] If so, the traffic condition of the detection area is determined to be congested; otherwise, the traffic condition of the detection area is determined to be unobstructed.

[0129] The traffic status of each road is determined according to a preset judgment rule. The preset judgment rule is as follows: determine the total number of detection areas configured for the first road and the number of detection areas in a congested state; if the number of detection areas configured for the road is equal to 2 and each detection area is in a congested state, then the traffic status of the road is congested; if the number of detection areas configured for the road is greater than 2 and the proportion of the number of detection areas in a congested state to the total number of detection areas is greater than two-thirds, then the traffic status of the road is congested; continue the judgment process for the next road until the traffic status of all roads has been determined.

[0130] This embodiment provides a road traffic condition detection device based on panoramic surveillance video analysis. A configuration module 61 configures at least two detection areas on each road in the video frame; a decoding module 62 acquires the real-time video stream and decodes it to obtain video frame images; a vehicle detection and tracking module 63 uses a detection and tracking model to perform vehicle detection and tracking on each detection area in the video frame images, obtaining vehicle detection and tracking information for each detection area; wherein, the vehicle detection and tracking information includes the vehicle's OBB detection frame information and ID information; a calculation module 64 calculates the number of vehicles and vehicle speed in each detection area based on the vehicle detection and tracking information in each detection area; a judgment module 65 determines the traffic condition of each road according to preset judgment rules and the number of vehicles and vehicle speed in each detection area; wherein, the traffic condition of each road reflects the current congestion situation of each road. The road traffic condition detection device based on panoramic surveillance video analysis provided in this embodiment can accurately detect and track small target vehicles in panoramic surveillance videos, realizing the discovery of traffic congestion from a wider field of view, thereby assisting road traffic management departments in timely handling of traffic congestion events.

[0131] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure, such as... Figure 7 As shown, the electronic device 70 in this embodiment may include: a memory 71 and a processor 72.

[0132] The memory 71 is used to store computer programs (such as application programs, functional modules, etc. that implement the above-mentioned road traffic condition detection method based on panoramic monitoring video analysis), computer instructions, etc.

[0133] The aforementioned computer programs and instructions can be stored in one or more partitions of memory 71. Furthermore, the aforementioned computer programs and instructions can be invoked by processor 72.

[0134] The processor 72 is configured to execute the computer program stored in the memory 71 to implement the various steps in the methods described in the above embodiments.

[0135] For details, please refer to the relevant descriptions in the preceding method embodiments.

[0136] The memory 71 and the processor 72 can be independent structures or integrated structures. When the memory 71 and the processor 72 are independent structures, the memory 71 and the processor 72 can be coupled together via bus 73.

[0137] An electronic device in this embodiment can perform... Figures 2-5 For the technical solution of the method shown, please refer to the specific implementation process and technical principles. Figures 2-5 The relevant descriptions of the methods shown will not be repeated here.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. It will be clearly understood by those skilled in the art that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be indirect couplings or communication connections through some ports, apparatuses, or units, and may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims of the present invention.

Claims

1. A road traffic condition detection method based on panoramic surveillance video analysis, characterized in that, Traffic condition detection is performed using road video captured by high-definition panoramic monitoring equipment mounted at a height of tens of meters, including: Configure at least two detection zones on each road in the video footage; Acquire a real-time video stream and decode the real-time video stream to obtain video frame images; The detection and tracking model is used to perform vehicle detection and tracking in each detection region of the video frame image to obtain vehicle detection and tracking information in each detection region; wherein, the vehicle detection and tracking information includes the vehicle's OBB detection box information and ID information; The number of vehicles and their speed in each detection area are calculated based on the vehicle detection and tracking information in each detection area. The traffic status of each road is determined based on preset judgment rules and the number and speed of vehicles in each detection area; wherein, the traffic status of each road reflects the current congestion situation of each road. The provision of at least two detection areas on each road in the video frame includes: Mark at least two polygonal regions on each road in the video footage; The minimum bounding box of each polygon region is constructed by traversing the vertices of each polygon region. The image region enclosed by the minimum bounding box is the detection region. The method of using a detection and tracking model to perform vehicle detection and tracking in each detection region of the video frame image, and obtaining vehicle detection and tracking information in each detection region, includes: The improved Yolov5 model, which is pre-trained, is used to detect vehicles in the image regions enclosed by the minimum bounding quadrilaterals in each video frame image, and obtains the AABB detection box and candidate OBB detection box corresponding to each vehicle in each detection region. The improved Yolov5 model is specifically an OBB detection box generation module added to the output of the original Yolov5 model. The candidate OBB detection boxes corresponding to each vehicle in each detection area are filtered to obtain the final OBB detection boxes corresponding to each vehicle in each detection area. Construct corresponding AABB bounding boxes for the final OBB detection boxes of each detection area, and input the AABB bounding boxes of each detection area into the Sort tracking model to track each detected vehicle and determine the vehicle ID information of each detection area. The step of filtering the candidate OBB detection boxes corresponding to each vehicle in each detection area to obtain the final OBB detection boxes corresponding to each vehicle in each detection area includes: Remove candidate OBB detection boxes whose confidence scores are lower than a preset score threshold in each detection area corresponding to each vehicle, and sort the remaining candidate OBB detection boxes in each detection area according to their confidence scores to obtain the candidate OBB detection box sequence in each detection area. Using the remaining candidate vehicle OBB detection boxes and corresponding AABB detection boxes in each detection area, AABB collision and OBB collision are performed respectively. The Poly-NMS algorithm is then used to filter the candidate OBB detection box sequences in each detection area after the collision to obtain the final OBB detection boxes for each detection area. The calculation of the number of vehicles and vehicle speed in each detection area based on the vehicle detection and tracking information in each detection area includes: The number of vehicles in each detection area at the current moment is determined based on the final OBB detection frame of each detection area; The speed of each vehicle in each detection area at the current moment is calculated based on the final OBB detection frame and ID information of the vehicles in each detection area within the preset time period. The average speed of vehicles in each detection area at the current moment is calculated based on the number of vehicles in each detection area at the current moment and the speed of each vehicle. The step of calculating the speed of each vehicle in each detection area at the current moment based on the final OBB detection frame and ID information of vehicles in each detection area within a preset time period includes: The final OBB detection frame for each vehicle within a preset time period is determined based on the vehicle ID information in each detection area. The displacement of each vehicle within the preset time period is calculated using the center point of the final OBB detection frame corresponding to each vehicle in each detection area within the preset time period. The speed of each vehicle in each detection area at the current moment is calculated using the displacement of each vehicle within a preset time period.

2. The road traffic condition detection method according to claim 1, characterized in that, The step of determining the traffic status of each road based on preset judgment rules and the number and speed of vehicles in each detection area includes: The number of vehicles and the speed of vehicles in each detection area are assigned different weight values, and the congestion parameter value of each detection area is calculated based on the weight values ​​corresponding to the number of vehicles and the speed of vehicles in each detection area. The congestion parameter value is the sum of the product of the number of vehicles and the speed of vehicles and the corresponding weight values. Determine whether the congestion parameter values ​​in each detection area are greater than the preset parameter threshold; If so, the traffic condition of the detection area is determined to be congested; otherwise, the traffic condition of the detection area is determined to be unobstructed. The traffic status of each road is determined according to a preset judgment rule. The preset judgment rule is as follows: determine the total number of detection areas configured for the first road and the number of detection areas in a congested state; if the number of detection areas configured for the road is equal to 2 and each detection area is in a congested state, then the traffic status of the road is congested; if the number of detection areas configured for the road is greater than 2 and the proportion of the number of detection areas in a congested state to the total number of detection areas is greater than two-thirds, then the traffic status of the road is congested; continue the judgment process for the next road until the traffic status of all roads has been determined.

3. A road traffic condition detection device based on panoramic surveillance video analysis, characterized in that, include: The configuration module is used to mark at least two polygonal regions on each road in the video frame; The minimum bounding box of each polygon region is constructed by traversing the vertices of each polygon region. The image region enclosed by the minimum bounding box is the detection region. A decoding module is used to acquire a real-time video stream and decode the real-time video stream to obtain video frame images; The vehicle detection and tracking module utilizes a pre-trained improved Yolov5 model to simultaneously detect vehicles within image regions enclosed by minimum bounding boxes (MBCs) in each video frame image, obtaining AABB detection boxes and candidate OBB detection boxes for each vehicle in each detection region. Specifically, the improved Yolov5 model adds an OBB detection box generation module to the output of the original Yolov5 model. Candidate OBB detection boxes with confidence scores lower than a preset threshold are removed from the candidate OBB detection boxes corresponding to each vehicle in each detection region. The remaining candidate OBB detection boxes in each detection region are then processed according to their confidence scores. Row sorting is used to obtain the candidate OBB detection box sequence in each detection region; AABB collision and OBB collision are performed using the remaining candidate vehicle OBB detection boxes and corresponding AABB detection boxes in each detection region, and the Poly-NMS algorithm is used to filter the candidate OBB detection box sequence in each detection region after collision to obtain the final OBB detection box in each detection region; corresponding AABB bounding boxes are constructed for the final OBB detection boxes in each detection region, and the AABB bounding boxes of each detection region are input into the Sort tracking model to track each detected vehicle and determine the vehicle ID information of each detection region; The calculation module is used to determine the number of vehicles in each detection area at the current time based on the final OBB detection frame of each detection area; determine the final OBB detection frame of each vehicle within a preset time period based on the ID information of the vehicles in each detection area; and calculate the displacement of each vehicle within the preset time period using the center point of the final OBB detection frame of each vehicle in each detection area within the preset time period. The speed of each vehicle in each detection area at the current moment is calculated using the displacement of each vehicle within a preset time period. The average speed of vehicles in each detection area at the current moment is calculated based on the number of vehicles in each detection area at the current moment and the speed of each vehicle. The determination module assigns different weight values ​​to the number of vehicles and vehicle speed in each detection area, and calculates the congestion parameter value for each detection area based on the weight values ​​corresponding to the number of vehicles and vehicle speed in each detection area. The congestion parameter value is the sum of the product of the number of vehicles and vehicle speed and their corresponding weight values. It then determines whether the congestion parameter value of each detection area is greater than a preset threshold. If it is, the traffic state of the detection area is determined to be congested; otherwise, the traffic state of the detection area is determined to be free-flowing. The module then determines the traffic state of each road according to preset judgment rules. These preset judgment rules are: determining the total number of detection areas configured for the first road and the number of detection areas in a congested state; if the number of detection areas configured for the road is equal to 2, and each detection area is in a congested state, then the traffic state of the road is congested; if the number of detection areas configured for the road is greater than 2, and the proportion of detection areas in a congested state to the total number of detection areas is greater than two-thirds, then the traffic state of the road is congested. The determination process continues for the next road until the traffic state of all roads has been determined.

4. A computer device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the road traffic condition detection method as described in any one of claims 1 and 2.

Images