Traffic accident detection method and electronic equipment based on real-time images of drone inspection

Abstract

The present invention proposes a traffic accident detection method and electronic equipment based on real-time images of drone inspections, which utilize drones to inspect highways. At the same time, based on target detection and semantic segmentation technology in deep learning, a comprehensive judgment of traffic accidents is made through multiple preset traffic accident judgment rules, and traffic accident scores are calculated. This method can not only meet the real-time detection requirements, making the detection more flexible and no longer limited to specific areas captured by fixed cameras, but also greatly improve the detection accuracy of traffic accidents by enriching the judgment rules of traffic accidents.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic supervision, and in particular relates to a traffic accident detection method and electronic equipment based on real-time images of unmanned aerial vehicle inspection. Background Art

[0002] Traditional traffic accident detection methods mainly detect accidents based on downstream traffic flow changes, changes in single-vehicle sensor data, and induction coil technology. Among them, the real-time performance of traffic accident detection based on macro traffic flow characteristics is not high, the reliability of traffic accident detection based on single-vehicle sensor data is low, and the post-maintenance of traffic accident detection based on induction coil technology is extremely inconvenient.

[0003] Traffic accident detection methods based on deep learning use the target detection technology in deep learning to detect and frame vehicles in the image. For any two vehicles, the intersection and union ratio of the two detection frames is calculated to determine whether the two vehicles have a traffic accident. Although this type of method can meet real-time requirements, it will lead to judgment errors when the detection frame is inaccurate. In addition, the judgment rules of traffic accident detection algorithms based on deep learning are relatively simple, which will lead to low detection accuracy of traffic accidents.

[0004] At present, among the traditional traffic accident detection methods and traffic accident detection methods based on deep learning, some methods are based on video data for traffic accident detection, but these methods obtain video data through fixed surveillance cameras. Since the area captured by fixed surveillance cameras is limited, traffic accident detection can only be performed on sections within the field of view of fixed surveillance cameras, which has certain limitations. Summary of the invention

[0005] In view of the problem that the accuracy and real-time performance of traditional traffic accident detection methods and the accuracy of traffic accident detection methods based on deep learning cannot meet the changing needs of highway video surveillance detection, the present invention is based on the target detection and semantic segmentation technology in deep learning, and proposes a traffic accident detection method and electronic equipment based on real-time images of drone inspections. The method performs comprehensive judgment of traffic accidents through multiple preset traffic accident judgment rules, and calculates traffic accident scores. It can not only meet the real-time detection requirements, making the detection more flexible and no longer limited to the specific areas captured by fixed cameras, but also greatly improve the detection accuracy of traffic accidents by enriching the judgment rules of traffic accidents.

[0006] The first aspect of the present invention discloses a traffic accident detection method based on real-time images of drone inspection, which comprises:

[0007] Using drones to inspect highways;

[0008] Acquire the inspection video stream captured by the camera on the drone in real time, and read the frame images in the video stream;

[0009] Using the target detection model to detect the vehicle in the frame image, and obtaining the coordinate information of the vehicle detection frame; using the semantic segmentation model to perform semantic segmentation on the frame image, and obtaining the highway segmentation area and the vehicle segmentation area;

[0010] According to the coordinate information of the vehicle segmentation area and the vehicle detection frame, when the vehicle detection frame and the vehicle segmentation area have an intersection, the vehicle segmentation area in the vehicle detection frame is retained, that is, the vehicle segmentation correction area; the highway segmentation area is filtered according to a preset area threshold to obtain the highway segmentation correction area;

[0011] Based on the vehicle segmentation correction area and the highway segmentation correction area, determine whether the type of the vehicle corresponding to the vehicle segmentation correction area is a normal vehicle. If it is determined to be an abnormal vehicle, return the center point coordinates, length, width and angle γ of the minimum circumscribed rectangle of the vehicle segmentation correction area, where the angle γ is the angle between the width of the minimum circumscribed rectangle and the opposite direction of the horizontal direction;

[0012] Calculating the aspect ratio of the minimum circumscribed rectangle of the vehicle segmentation correction area, the distance between vehicles, and the vehicle angle and the highway angle;

[0013] Performing a comprehensive traffic accident determination based on pre-set traffic accident determination rules related to the aspect ratio, distance, vehicle angle, and highway angle, and determining that the vehicle is an accident vehicle if any one of the traffic accident determination rules is met;

[0014] The final traffic accident score of the vehicle is calculated; the traffic accident score is used to indicate the probability of the vehicle having a traffic accident.

[0015] Optionally, the target detection model uses a yolov5 model.

[0016] Optionally, the semantic segmentation model is an STDC segmentation model.

[0017] Optionally, the traffic accident determination rules include the following three items:

[0018] The first rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the aspect ratio of the minimum circumscribed rectangle of the vehicle segmentation correction area, including:

[0019] Define A, B, F, E as the four vertices of the minimum circumscribed rectangle of the vehicle segmentation correction area, AE is the short side of rectangle ABFE, its length is represented by w, AB is the long side of rectangle ABFE, its length is represented by h;

[0020] If w / h ≤ d0, the vehicle is determined to be a normal vehicle; otherwise, the vehicle is determined to be an accident vehicle; d0 is a preset first threshold, 0.7≤d0<1;

[0021] Take point D on AE and point H on the extension line of AE, set the ratio of AD to AB equal to d0, and the ratio of AH to AB equal to 1;

[0022] When point E is located on line segment AD, the ratio of AE to AB is always less than or equal to d0, and the vehicle is determined to be a normal vehicle, and the traffic accident score of the normal vehicle is 0;

[0023] When point E is on line segment DH and point E does not coincide with point D, the ratio of AE to AB is greater than d0 and less than or equal to 1, then the vehicle is determined to be an accident vehicle, and its traffic accident score is defined as the ratio of DE to DH;

[0024] When point E coincides with point H, the vehicle is determined to be an accident vehicle, and its traffic accident score is defined as 1;

[0025] The second rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the distance between vehicles, which can be either of the following two methods:

[0026] The first method is: define the minimum circumscribed rectangle of one vehicle segmentation correction area as rectangle ABCD, and the minimum circumscribed rectangle of another vehicle segmentation correction area as rectangle EFGH; O1 is the center point of rectangle ABCD, and O2 is the center point of rectangle EFGH; w1 and w2 represent the width of rectangle ABCD and rectangle EFGH respectively;

[0027] With the center point O1 of rectangle ABCD as the center and r as the radius, we get a circular area;

[0028] If the center point O2 of the rectangle EFGH falls within the circular area, the shortest distance d1 between the contours of the two vehicle segmentation correction areas is calculated;

[0029] Compare w1 and w2, and take the product of the smaller value and the scale factor θ as the second threshold d2, where the value range of θ is (0, 1);

[0030] If the closest distance d1 is less than d2, then the two vehicles are determined to be accident vehicles, and the traffic accident scores of the two vehicles are defined as 1 - d1 / d2; otherwise, the two vehicles are determined to be normal vehicles, and the traffic accident score of the normal vehicle is 0;

[0031] The second method is: define the minimum circumscribed rectangle of one vehicle segmentation correction area as rectangle ABCD, and the minimum circumscribed rectangle of another vehicle segmentation correction area as rectangle EFGH; O1 is the center point of rectangle ABCD, and O2 is the center point of rectangle EFGH; w1 and w2 represent the width of rectangle ABCD and rectangle EFGH respectively, and w1≤w2;

[0032] Calculate the distance d between the center point O1 of the minimum circumscribed rectangle of one vehicle segmentation area and the center point O2 of the minimum circumscribed rectangle of another vehicle segmentation area:

[0033] When d=(w1 + w2) / 2, it is determined that both vehicles are accident vehicles, and the traffic accident scores of both vehicles are defined as 1;

[0034] When d≥w1+w2 / 2, it is determined that both vehicles are normal vehicles, and the traffic accident score of the normal vehicle is 0;

[0035] When (w1 + w2) / 2<d<w1 + w2 / 2, the two vehicles are judged to be accident vehicles, and the traffic accident scores of the two vehicles are defined as 1-(d-(w1+w2) / 2) / (w1 / 2);

[0036] The third rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the highway angle and vehicle angle, including:

[0037] Determine whether the center point coordinates of the minimum circumscribed rectangle of a vehicle segmentation correction area are located in the highway segmentation correction area: if so, it is determined that the vehicle is on this highway;

[0038] Difference between the highway angle and the vehicle angle, the absolute value of the difference is angle:

[0039] If 90<angle≤180, the traffic accident score is defined as (180 - angle) / 90;

[0040] If 0≤angle≤90, the traffic accident score is defined as angle / 90;

[0041] If the traffic accident score calculated above is greater than 0, it is determined that the vehicle has been involved in a traffic accident; otherwise, it is determined that the vehicle has not been involved in a traffic accident.

[0042] Optionally, determining whether the category of the vehicle corresponding to the vehicle segmentation correction area is a normal vehicle based on the vehicle segmentation correction area and the highway segmentation correction area specifically includes:

[0043] Use the cv2.pointPolygonTest() function in opencv to determine whether the contours of the vehicle segmentation correction area are within the contours of the highway segmentation correction area:

[0044] If not, it is determined to be a normal vehicle; otherwise, it is further determined whether the number of coordinate points of the contour boundary of the vehicle segmentation correction area is less than a preset number:

[0045] If it is less than, the minAreaRect() function in opencv cannot be used to fit the minimum enclosing rectangle of the vehicle segmentation correction area, and the vehicle is determined to be a normal vehicle; otherwise, the minAreaRect() function in opencv returns the center point coordinates, length and width, and the angle between the width of the minimum enclosing rectangle and the opposite direction of the horizontal direction of the minimum enclosing rectangle of the vehicle segmentation correction area and the highway segmentation correction area respectively.

[0046] Optionally, the highway angle or vehicle angle is calculated as follows:

[0047] The length of the minimum circumscribed rectangle of the expressway segmentation correction area or the minimum circumscribed rectangle of the vehicle segmentation correction area is represented by h, and the width is represented by w; the angle between the width of the minimum circumscribed rectangle and the opposite direction of the horizontal direction is represented by γ, and the angle between the direction of the minimum circumscribed rectangle and the opposite direction of the horizontal direction is represented by α;

[0048] When w ≤ h, if 0°≤γ < 90°, then α = 90° + γ;

[0049] When w ≤ h, if γ = 90°, then α = 0;

[0050] When w>h, if 0°≤ γ ≤ 90°, then α= γ.

[0051] Optionally, calculating the final traffic accident score of the vehicle includes: for the accident vehicle, respectively calculating the traffic accident score corresponding to each determination rule; and taking the maximum value of the scores as the final traffic accident score of the vehicle.

[0052] Optionally, the traffic accident detection method further includes framing the accident vehicle in the vehicle detection frame and identifying a traffic accident score of the vehicle.

[0053] Optionally, a rectangle is drawn in the vehicle detection frame using the rectangle() function in opencv to frame the accident vehicle; and a final traffic accident score of the vehicle is displayed in the upper left corner of the rectangle using the putText() function in opencv.

[0054] The second aspect of the present invention discloses an electronic device, characterized in that it includes a processor and a memory, and the processor is used to call a computer program stored in the memory to execute the traffic accident detection method based on real-time images of drone inspections described in the first aspect and any optional scheme.

[0055] The present invention has the following beneficial effects:

[0056] (1) The sum of the total inference time and post-processing time of the YOLOv5 and STDC models used in the present invention is less than 10 ms, which can meet the real-time requirements compared with the traditional traffic accident detection algorithm.

[0057] (2) In the present invention, the video stream information of the highway is obtained in real time by relying on drone inspection. Compared with fixed cameras, drones have greater flexibility and are easy to operate, so that the detection range is no longer limited to a fixed area. Therefore, the detection is more convenient and covers a wider range.

[0058] (3) In the present invention, target detection and semantic segmentation technology are used simultaneously, and then a comprehensive judgment of traffic accidents is made based on the results of target detection and semantic segmentation. Compared with other methods with relatively simple judgment rules, the present invention has a higher detection accuracy for traffic accidents.

[0059] (4) In the present invention, there are three traffic accident determination rules, which are considered from three aspects: the angle between the minimum circumscribed rectangle of the vehicle segmentation correction area and the minimum circumscribed rectangle of the highway segmentation correction area, the distance between the vehicle segmentation areas, and the aspect ratio of the minimum circumscribed rectangle of the vehicle segmentation correction area. As long as one of them is met, it is determined that the corresponding vehicle has a traffic accident. In other traffic accident detection methods, there are often only one or two determination rules. Therefore, compared with other methods, the determination rules of the present invention are more comprehensive and have a lower missed detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the present invention.

[0061] Figure 2 A schematic diagram of a vehicle detection box returned by the target detection model.

[0062] Figure 3 It is the relationship between the direction of the minimum circumscribed rectangle of the vehicle segmentation correction area and the horizontal direction. There are four types of angles between the two, namely obtuse angle, right angle, zero angle and acute angle. Figure 3-1 It corresponds to the situation when the angle between the direction of the minimum circumscribed rectangle of the vehicle segmentation correction area and the opposite direction of the horizontal direction is an obtuse angle or a right angle; Figure 3-2It corresponds to the situation when the angle between the direction of the minimum circumscribed rectangle of the vehicle segmentation correction area and the direction opposite to the horizontal direction is zero; Figure 3-3 This corresponds to the situation when the angle between the direction of the minimum circumscribed rectangle of the vehicle segmentation correction area and the direction opposite to the horizontal direction is an acute angle.

[0063] Figure 4 Schematic diagram of the minimum circumscribed rectangle of the vehicle segmentation correction area; wherein A, B, F, and E are the four vertices of the minimum circumscribed rectangle of the vehicle segmentation area.

[0064] Figure 5 Schematic diagram for determining the shortest distance between the contours of the vehicle segmentation correction area of ​​any two vehicles.

[0065] Figure 6 It is a schematic diagram of the positional relationship between the minimum circumscribed rectangles of the vehicle segmentation correction areas of any two vehicles. Figure 6-1 This corresponds to the situation when the distance between the minimum circumscribed rectangles of the vehicle segmentation correction areas of the two vehicles is the smallest; Figure 6-2 This corresponds to the situation when the minimum circumscribed rectangles of the vehicle segmentation correction areas of the two vehicles have no overlapping parts. DETAILED DESCRIPTION

[0066] In order to further understand the present invention, the technical solution of the present invention is explained below in conjunction with specific embodiments and drawings.

[0067] The present invention uses a drone to inspect the highway, reads the inspection video stream in real time through the camera carried by the drone, and sends the video stream to the background, processes each frame of the video stream in the background, and determines whether a traffic accident has occurred. The core technologies used in the present invention mainly include target detection and semantic segmentation, and the deep learning models used are yolov5 and STDC. Among them, the yolov5 target detection model is used to detect the vehicle in the video frame to obtain the upper left corner coordinates and lower right corner coordinates of the vehicle detection frame; the STDC segmentation model is used to semantically segment the video frame to obtain the segmented areas of the highway and the vehicle. The present invention models the traffic accident, sets the traffic accident judgment rule, and then completes the traffic accident judgment based on the prediction results of the yolov5 model and the STDC model combined with the traffic accident judgment rule.

[0068] like Figure 1 As shown, the traffic accident detection method based on real-time images of drone inspection disclosed in the present invention mainly includes the following steps:

[0069] Step 1: Use drones to inspect highways, read video streams in real time through the cameras on the drones, and send the video streams to the backend.

[0070] Step 2: After the background reads a frame of image (i.e., video frame) from the video stream, it is input into the target detection model to detect the vehicle in the frame image and return the coordinate information of the upper left corner vertex and the lower right corner vertex of the vehicle detection frame. At the same time, the read frame of image is sent to the STDC segmentation model to obtain the vehicle segmentation area and the highway segmentation area.

[0071] Step 3: Based on the coordinate information of the upper left corner vertex and the lower right corner vertex of the vehicle detection frame and the vehicle segmentation area, the vehicle segmentation area in each vehicle detection frame is obtained, which is called the "vehicle segmentation correction area". The specific steps are as follows:

[0072] Assume that the vehicle detection box returned by the target detection model is Figure 2 In the rectangle ABCD in the figure, the upper left vertex of the detection box is represented by A, and the lower right vertex of the vehicle detection box is represented by C. For the vehicle segmentation area, there are two situations in its relationship with the rectangle ABCD, namely: the vehicle segmentation area has an intersection with the rectangle ABCD or does not have an intersection. When the vehicle segmentation area intersects with the rectangle ABCD, only the vehicle segmentation area inside the rectangle ABCD is retained. The part of the vehicle segmentation area that will be retained is called the "vehicle segmentation correction area". When the vehicle segmentation area does not intersect with the rectangle ABCD, the vehicle segmentation area is discarded and no longer considered.

[0073] At the same time, the obtained highway segmentation areas are filtered according to the area threshold, and the highway segmentation areas retained after filtering are called "highway segmentation correction areas". When the highway segmentation areas are filtered according to the area threshold, the area threshold here is set in advance by us, and it is a numerical value. When the STDC model is used to segment the roads in the image, some road areas with very small areas may be generated due to segmentation errors or inaccurate segmentation. These road areas with very small areas may interfere with subsequent judgments. In order to eliminate these interferences as much as possible, an area threshold is set in the present invention. When the area of ​​the minimum circumscribed rectangle of a highway segmentation area is less than the area threshold, the highway segmentation area will no longer be considered in the subsequent traffic accident judgment, that is, the highway segmentation area is filtered out, and the road segmentation area that is retained is the "highway segmentation correction area".

[0074] Step 4: Use the cv2.pointPolygonTest() function in opencv (a cross-platform computer vision library released under the BSD license (open source)) to determine whether the contours of the vehicle segmentation correction area are in the contours of the highway segmentation correction area. If the contours of a vehicle are not in the contours of any highway, the vehicle corresponding to the vehicle segmentation correction area is determined to be a normal vehicle; otherwise, determine whether the number of coordinate points on the boundary of the contours of the vehicle segmentation correction area is less than 5. If it is less than 5, the minimum enclosing rectangle of the vehicle segmentation correction area cannot be fitted by the minAreaRect() function in opencv, and the vehicle is treated as a normal vehicle; if the minimum enclosing rectangle can be fitted, the length-width ratio of the minimum enclosing rectangle of the vehicle segmentation correction area, the distance between vehicles, and the angle between the direction of the minimum enclosing rectangle of the vehicle segmentation correction area and the opposite direction of the horizontal direction (referred to as "vehicle angle") are calculated based on the center point coordinates, length, width and angle γ of the minimum enclosing rectangle returned by the minAreaRect() function in opencv. In the same way, the angle between the direction of the minimum enclosing rectangle of the highway segmentation correction area and the opposite direction of the horizontal direction (referred to as the "highway angle") is calculated. The first two items are used as a traffic accident determination rule, and the last item is combined with the highway angle as a traffic accident determination rule. The above angle γ represents the angle between the width of the corresponding minimum enclosing rectangle and the opposite direction of the horizontal direction. Figure 3 .

[0075] It can be understood that the direction of the minimum enclosing rectangle is the orientation of the minimum enclosing rectangle. The minimum enclosing rectangle is obtained based on the vehicle segmentation correction area or the highway segmentation correction area, so the orientation of the vehicle or highway (for example: the orientation of the vehicle is the direction facing the front of the vehicle) is almost consistent with the orientation of its minimum enclosing rectangle. Therefore, choosing to use the orientation of the corresponding minimum enclosing rectangle to represent the orientation of the vehicle or highway makes the problem concrete.

[0076] like Figure 3 As shown in the figure, the minimum enclosing rectangle of the highway segmentation correction area and the minimum enclosing rectangle of the vehicle segmentation correction area are uniformly represented by rectangle ABCD. The length of line segment AB is h, which represents the length of the minimum enclosing rectangle; the length of line segment BC is w, which represents the width of the minimum enclosing rectangle. Angle γ represents the angle between the width of the minimum enclosing rectangle and the opposite direction of the horizontal direction, and angle α represents the angle between the direction of the minimum enclosing rectangle and the opposite direction of the horizontal direction. Among them, angle γ is the third item in the return value of the minAreaRect() function.

[0077] The highway angle and the vehicle angle are represented by angle α1 and angle α2 respectively. Among them, angle α1 and angle α2 are calculated based on the minimum circumscribed rectangle of the corresponding segmented area, and their calculation methods are exactly the same. Therefore, in the present invention, the calculation methods of angle α1 and angle α2 are abstracted and unified through Figure 3 A demonstration is provided to illustrate how this is calculated. Figure 3 In the example, angle α is used to represent the angle between the direction of the minimum circumscribed rectangle of the corresponding segmented area and the opposite direction of the horizontal direction. That is, angle α represents angle α1 or angle α2. When calculating the angle of the highway, Figure 3 The angle α in represents α1; when the vehicle angle is calculated, Figure 3 The angle α in represents the angle α2. For the highway segmentation correction area, after finding its minimum circumscribed rectangle, use the following rules (1), (2), and (3) to find the angle α1, that is, the highway angle. Similarly, for the vehicle segmentation correction area, after finding its minimum circumscribed rectangle, use the following rules (1), (2), and (3) to find the angle α2, that is, the vehicle angle.

[0078] The angle α is calculated as follows:

[0079] (1) If Figure 3-1 As shown, when w ≤ h, if 0°≤γ < 90°, then α= 90°+ γ;

[0080] (2) If Figure 3-2 As shown, when w ≤ h, if γ = 90°, then α = 0;

[0081] (3) If Figure 3-3 As shown, when w > h, if 0°≤ γ ≤ 90°, then α= γ.

[0082] Step 5: Output the vehicle category and traffic accident score, where the vehicle category is a normal vehicle and an accident vehicle, and the traffic accident score indicates the probability of the vehicle having a traffic accident.

[0083] The present invention is based on the target detection yolov5 model and the semantic segmentation STDC model, and comprehensively judges the vehicle from three aspects: the highway angle and the vehicle angle, the aspect ratio of the minimum circumscribed rectangle of the vehicle segmentation area, and the distance between vehicles. The judgment rules are more comprehensive and more rigorous. Therefore, compared with other methods with relatively single judgment rules, the present invention has higher detection accuracy for traffic accidents.

[0084] Step 6: Determine the final traffic accident score and frame the accident vehicle. For vehicles involved in traffic accidents, take the maximum value of the traffic accident score in the judgment rule as the final traffic accident score of the vehicle; then, based on the coordinate information of the upper left and lower right corners of the vehicle detection frame previously returned by the target detection yolov5 model, draw a rectangle through the rectangle() function in opencv; finally, use the putText() function in opencv to display the final traffic accident score of the vehicle in the upper left corner of the rectangle.

[0085] It should be noted that the average value of the traffic accident scores of the above-mentioned determination rules may also be calculated as the final traffic accident score of the vehicle, or calculated in a weighted manner, and the present invention is not limited to this.

[0086] In the process of traffic accident determination, as long as any one of the rules appears, it is determined that the corresponding vehicle has had a traffic accident, making the determination of traffic accidents more three-dimensional. The traffic accident determination rules mainly include the following three items:

[0087] The first rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the aspect ratio of the minimum circumscribed rectangle of the vehicle segmentation correction area.

[0088] If the ratio of the shortest side to the longest side of the minimum circumscribed rectangle of the vehicle segmentation correction area is greater than the threshold d0 (d0=0.7), it is determined that the vehicle has been involved in a traffic accident. Figure 3 In the formula, A, B, F, and E are the four vertices of the minimum circumscribed rectangle of the vehicle segmentation correction area, AE is the short side of the rectangle ABFE, and its length is represented by w, and AB is the long side of the rectangle ABFE, and its length is represented by h. If the ratio of AE to AB is less than or equal to 0.7, it is determined that the vehicle has not had a traffic accident. Otherwise, it is determined that the vehicle has had a traffic accident. The value range of the threshold d0 here can be adjusted based on experience, usually 0.7≤d0<1.

[0089] Furthermore, take point D on AE and point H on the extension line of AE, assuming that the ratio of AD to AB is 0.7, and the ratio of AH to AB is 1. Among them, AD represents a line segment on the shortest side AE ​​of the minimum circumscribed rectangle. When point E coincides with point D, AD represents the shortest side of the minimum circumscribed rectangle; AH represents a line segment on the shortest side AE ​​of the minimum circumscribed rectangle. When point E coincides with point H, AH represents the shortest side of the minimum circumscribed rectangle.

[0090] It can be seen that when point E is on line segment AD, the ratio of AE to AB is always less than or equal to 0.7, then it is determined that the vehicle has not had a traffic accident, and when no traffic accident has occurred, the vehicle's traffic accident score is defined as 0. When point E is on line segment DH and point E does not coincide with point D, the ratio of AE to AB will be greater than 0.7 and less than or equal to 1, then it is determined that the vehicle has had a traffic accident, and its traffic accident score is defined as the ratio of DE to DH, and when point E coincides with point H, the accident score is 1.

[0091] The second rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the distance between vehicles.

[0092] This rule mainly includes two methods:

[0093] Method 1: Figure 5 In the figure, it is assumed that the minimum bounding rectangle of one vehicle segmentation correction area is rectangle ABCD, and the minimum bounding rectangle of another vehicle segmentation correction area is rectangle EFGH. AB and AD are the length and width of rectangle ABCD, respectively, and EF and EH are the length and width of rectangle EFGH, respectively; O1 is the center point of rectangle ABCD, and O2 is the center point of rectangle EFGH. The irregular closed figure inside rectangle ABCD represents one vehicle segmentation correction area; the irregular closed figure inside rectangle EFGH represents another vehicle segmentation correction area. Among them, the minimum bounding rectangle of any vehicle segmentation correction area is obtained by the minAreaRect() function in opencv. Specifically, the contours of a vehicle segmentation correction area are passed as parameters to the minAreaRect() function, and the minAreaRect() function can return the information of the minimum bounding rectangle of the vehicle segmentation correction area, which includes the coordinates of the center point of the minimum bounding rectangle, the length and width of the minimum bounding rectangle, and the angle between the width of the minimum bounding rectangle and the horizontal direction.

[0094] A circular area is defined with O1 as the center and r as the radius. The specific value of r can be set as needed. For the vehicle segmentation correction area inside the rectangle ABCD, if the center point O2 of the minimum circumscribed rectangle of another vehicle segmentation correction area falls within the above circular area, then the shortest distance between the contours of the two vehicle segmentation correction areas is calculated, represented by the parameter d1.

[0095] For any vehicle segmentation correction area, its contours are composed of a series of coordinate points. Therefore, to find the shortest distance between the contours of two different vehicle segmentation correction areas, that is, to find the distance between the two closest coordinate points in the two contours. The formula for finding the distance between two coordinate points is as follows:

[0096]

[0097] Among them, (x1, y1) and (x2, y2) represent the coordinate points in the contours of one vehicle segmentation correction area and the coordinate points in the contours of another vehicle segmentation correction area respectively.

[0098] Figure 5 In the example, if the distance between point I and point J is the shortest, the length of line segment IJ is d1. Compare the lengths of AD and EH, and take the product of the smaller value and the scale factor θ as the threshold d2. The value range of θ is (0, 1). If d1 is less than d2, it is determined that both vehicles have a traffic accident, and their traffic accident scores are defined as 1 - d1 / d2. Otherwise, it is determined that neither vehicle has a traffic accident, and their traffic accident scores are defined as 0.

[0099] Method 2: Calculate the distance d between the center point of the minimum circumscribed rectangle of one vehicle segmentation correction area and the center point of the minimum circumscribed rectangle of another vehicle segmentation correction area.

[0100] Specifically, Figure 6 The rectangle ABCD and rectangle EFGH in the figure represent the minimum circumscribed rectangles of different vehicle segmentation correction areas. O1 and O2 are the centers of rectangle ABCD and rectangle EFGH, respectively. The length of line segment O1O2 is represented by d. w1 and w2 represent the shortest sides (i.e., width) of rectangle ABCD and rectangle EFGH, respectively, where w1≤w2.

[0101] (1) When d = (w1 + w2) / 2, it is determined that both vehicles have had a traffic accident, and the traffic accident scores of both vehicles are defined as 1.

[0102] (2) When d ≥ w1 + w2 / 2, it is determined that neither of the two vehicles has been involved in a traffic accident, and the traffic accident scores of the two vehicles are defined as 0.

[0103] (3) When (w1 + w2) / 2<d<w1 + w2 / 2, it is determined that both vehicles have had a traffic accident, and the traffic accident scores of both vehicles are defined as 1-(d-(w1+w2) / 2) / (w1 / 2).

[0104] It should be noted that, in actual application, only one of method 1 and method 2 needs to be selected. The present invention provides two methods for calculating traffic accident scores for determining whether a traffic accident occurs based on the distance between vehicles.

[0105] The third rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the highway angle and vehicle angle.

[0106] First, according to the segmentation results of the video frame, the minimum enclosing rectangle of each highway segmentation correction area and the vehicle segmentation correction area is calculated respectively; then it is determined whether the center point coordinates of the minimum enclosing rectangle of the vehicle segmentation correction area are located in the highway segmentation correction area: if its center point coordinates are located in a highway segmentation correction area, it is determined that the vehicle is on this highway; finally, the angle between the direction of the minimum enclosing rectangle of the vehicle segmentation correction area and the horizontal direction and the angle between the direction of the minimum enclosing rectangle of the highway segmentation correction area and the horizontal direction are calculated respectively, and the difference between the two is taken, and the traffic accident score is calculated based on the absolute value of the difference angle and the vehicle category is determined.

[0107] This rule determines whether the vehicle has been involved in a traffic accident based on the angle of the highway and the vehicle. When the vehicle is determined to be an accident vehicle, that is, a traffic accident has occurred, a traffic accident score is calculated. Specifically, the absolute value of the difference between the highway angle and the vehicle angle is defined as angle. If 90<angle≤180, the traffic accident score is defined as (180 - angle) / 90; if 0≤angle≤90, the traffic accident score is defined as angle / 90; if the above calculated traffic accident score is greater than 0, it is determined that the vehicle has been involved in a traffic accident; otherwise, it is determined that the vehicle has not been involved in a traffic accident.

[0108] Furthermore, an electronic device is disclosed in the embodiment, which mainly includes a processor and a memory. The processor is mainly used to call the computer program stored in the memory and execute the traffic accident detection method based on real-time images of drone inspection provided in the above embodiment, which will not be repeated here.

[0109] Finally, it should be noted that although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of this specification, a person of ordinary skill in the art can also make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A traffic accident detection method based on real-time images of drone inspection, characterized in that: include: Use drones to inspect highways; Acquire the inspection video stream captured by the camera on the drone in real time, and read the frame images in the video stream; Using the target detection model to detect the vehicle in the frame image, and obtaining the coordinate information of the vehicle detection frame; using the semantic segmentation model to perform semantic segmentation on the frame image, and obtaining the highway segmentation area and the vehicle segmentation area; According to the coordinate information of the vehicle segmentation area and the vehicle detection frame, when the vehicle detection frame and the vehicle segmentation area have an intersection, retain the vehicle segmentation area in the vehicle detection frame, that is, the vehicle segmentation correction area; Filtering the expressway segmentation area according to a preset area threshold to obtain an expressway segmentation correction area; Based on the vehicle segmentation correction area and the highway segmentation correction area, determine whether the type of the vehicle corresponding to the vehicle segmentation correction area is a normal vehicle. If it is determined to be an abnormal vehicle, return the center point coordinates, length, width and angle γ of the minimum circumscribed rectangle of the vehicle segmentation correction area, where the angle γ is the angle between the width of the minimum circumscribed rectangle and the opposite direction of the horizontal direction; Calculating the aspect ratio of the minimum circumscribed rectangle of the vehicle segmentation correction area, the distance between vehicles, and the vehicle angle and the highway angle; A comprehensive traffic accident determination is performed based on pre-set traffic accident determination rules respectively related to the aspect ratio, the distance between vehicles, the vehicle angle and the highway angle, and the category of the vehicle is determined to be an accident vehicle if any one of the traffic accident determination rules is met; the vehicle angle refers to the angle between the direction of the minimum circumscribed rectangle of the vehicle segmentation correction area and the opposite direction of the horizontal direction; the highway angle refers to the angle between the direction of the minimum circumscribed rectangle of the highway segmentation correction area and the opposite direction of the horizontal direction; Calculating a final traffic accident score of the vehicle; The traffic accident score is used to indicate the probability of the vehicle having a traffic accident; Traffic accident determination rules Including the following three items: The first rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the aspect ratio of the minimum circumscribed rectangle of the vehicle segmentation correction area, including: Define A, B, F, E as the four vertices of the minimum circumscribed rectangle of the vehicle segmentation correction area, AE is the short side of rectangle ABFE, its length is represented by w, AB is the long side of rectangle ABFE, its length is represented by h; If w / h ≤ d0, the vehicle is determined to be a normal vehicle; otherwise, the vehicle is determined to be an accident vehicle; d0 is a preset first threshold, 0.7≤d0<1; Take point D on AE and point H on the extension line of AE, set the ratio of AD to AB equal to d0, and the ratio of AH to AB equal to 1; When point E is located on line segment AD, the ratio of AE to AB is always less than or equal to d0, and the vehicle is determined to be a normal vehicle, and the traffic accident score of the normal vehicle is 0; When point E is on line segment DH and point E does not coincide with point D, the ratio of AE to AB is greater than d0 and less than or equal to 1, then the vehicle is determined to be an accident vehicle, and its traffic accident score is defined as the ratio of DE to DH; When point E coincides with point H, the vehicle is determined to be an accident vehicle, and its traffic accident score is defined as 1; The second rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the distance between vehicles, which can be either of the following two methods: The first method is: define the minimum circumscribed rectangle of one vehicle segmentation correction area as rectangle ABCD, and the minimum circumscribed rectangle of another vehicle segmentation correction area as rectangle EFGH; O1 is the center point of rectangle ABCD, and O2 is the center point of rectangle EFGH; w1 and w2 represent the width of rectangle ABCD and rectangle EFGH respectively; With the center point O1 of rectangle ABCD as the center and r as the radius, we get a circular area; If the center point O2 of the rectangle EFGH falls within the circular area, the shortest distance d1 between the contours of the two vehicle segmentation correction areas is calculated; Compare w1 and w2, and take the product of the smaller value and the scale factor θ as the second threshold d2, where the value range of θ is (0, 1); If the closest distance d1 is less than d2, then the two vehicles are determined to be accident vehicles, and the traffic accident scores of the two vehicles are defined as 1 - d1 / d2; otherwise, the two vehicles are determined to be normal vehicles, and the traffic accident score of the normal vehicle is 0; The second method is: define the minimum circumscribed rectangle of one vehicle segmentation correction area as rectangle ABCD, and the minimum circumscribed rectangle of another vehicle segmentation correction area as rectangle EFGH; O1 is the center point of rectangle ABCD, and O2 is the center point of rectangle EFGH; w1 and w2 represent the width of rectangle ABCD and rectangle EFGH respectively, and w1≤w2; Calculate the distance d between the center point O1 of the minimum circumscribed rectangle of one vehicle segmentation area and the center point O2 of the minimum circumscribed rectangle of another vehicle segmentation area: When d=(w1 + w2) / 2, it is determined that both vehicles are accident vehicles, and the traffic accident scores of both vehicles are defined as 1; When d≥w1+w2 / 2, it is determined that both vehicles are normal vehicles, and the traffic accident score of the normal vehicle is 0; when (w1+w2) / 2<d<w1+w2 / 2, it is determined that both vehicles are accident vehicles, and the traffic accident scores of the two vehicles are defined as 1-(d-(w1+w2) / 2) / (w1 / 2); The third rule is to determine whether a traffic accident has occurred and calculate the traffic accident score based on the highway angle and vehicle angle, including: Determine whether the center point coordinates of the minimum circumscribed rectangle of a vehicle segmentation correction area are located in the highway segmentation correction area: if so, it is determined that the vehicle is on this highway; Difference between the highway angle and the vehicle angle, the absolute value of the difference is angle: If 90<angle≤180, the traffic accident score is defined as (180 - angle) / 90; If 0≤angle≤90, the traffic accident score is defined as angle / 90; If the traffic accident score calculated above is greater than 0, it is determined that the vehicle has been involved in a traffic accident; otherwise, it is determined that the vehicle has not been involved in a traffic accident; Determining whether the category of the vehicle corresponding to the vehicle segmentation correction area is a normal vehicle based on the vehicle segmentation correction area and the highway segmentation correction area specifically includes: The cv2.pointPolygonTest() function in opencv is used to determine whether the contours of the vehicle segmentation correction area are within the contours of the highway segmentation correction area: If not, the vehicle is determined to be a normal vehicle; otherwise, it is further determined whether the number of coordinate points of the contour boundary of the vehicle segmentation correction area is less than a preset number: If it is less than, the minimum circumscribed rectangle of the vehicle segmentation correction area cannot be fitted by the minAreaRect() function in opencv, and the vehicle is determined to be a normal vehicle; otherwise, the center point coordinates, length and width of the minimum circumscribed rectangle of the vehicle segmentation correction area and the highway segmentation correction area, and the angle between the width of the minimum circumscribed rectangle and the opposite direction of the horizontal direction are returned by the minAreaRect() function in opencv; The highway angle or vehicle angle is calculated as follows: The length of the minimum circumscribed rectangle of the expressway segmentation correction area or the minimum circumscribed rectangle of the vehicle segmentation correction area is represented by h, and the width is represented by w; the angle between the width of the minimum circumscribed rectangle and the opposite direction of the horizontal direction is represented by γ, and the angle between the direction of the minimum circumscribed rectangle and the opposite direction of the horizontal direction is represented by α; When w ≤ h, if 0°≤γ < 90°, then α = 90° + γ; When w ≤ h, if γ = 90°, then α = 0; When w>h, if 0°≤ γ ≤ 90°, then α= γ.

2. The traffic accident detection method according to claim 1, characterized in that: The target detection model uses the yolov5 model.

3. The traffic accident detection method according to claim 1, characterized in that: The semantic segmentation model uses the STDC segmentation model.

4. The traffic accident detection method according to any one of claims 1 to 3, characterized in that: Calculating the final traffic accident score of the vehicle includes: calculating the traffic accident score corresponding to each determination rule for the accident vehicle; and taking the maximum value of the scores as the final traffic accident score of the vehicle.

5. The traffic accident detection method according to any one of claims 1 to 3, characterized in that: The method also includes framing the accident vehicle in the vehicle detection frame and identifying the traffic accident score of the vehicle.

6. The traffic accident detection method according to claim 5, characterized in that: The rectangle() function in opencv is used to draw a rectangle in the vehicle detection frame to frame the accident vehicle; the putText() function in opencv is used to display the final traffic accident score of the vehicle in the upper left corner of the rectangle.

7. An electronic device, characterized in that: It includes a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the traffic accident detection method based on real-time images of drone inspection as described in any one of claims 1 to 6.

Images