Drone-based Traffic Incident Detection Method, Device and Storage Medium

Abstract

This application is applicable to the field of drone technology, and provides a traffic event detection method, device and storage medium based on a drone. The method includes: obtaining a traffic event detection image, which is collected by an image acquisition device of the drone; inputting the traffic event detection image into a deep neural network model to obtain a point set, where the point set includes the points of the lane lines in the traffic event detection image, and the deep neural network model is used to identify the points with lane line features in the traffic event detection image; fitting the point set to obtain a lane line; segmenting the traffic event detection image based on the lane line to obtain a target detection area; identifying the target detection area to obtain target identification information; and performing traffic event detection according to the target identification information, which improves the efficiency and accuracy of traffic event detection.

Description

Technical Field

[0001] This application relates to the technical field of unmanned aerial vehicles, and particularly to a traffic event detection method, device and storage medium based on unmanned aerial vehicles. Background Art

[0002] The rapid development of unmanned aerial vehicle technology has brought new possibilities to the field of traffic event detection. Unmanned aerial vehicles have many advantages such as being flexible, having a wide monitoring range, and being remotely controllable. They can easily fly over obstacles, obtain more comprehensive traffic condition information, and provide strong support for dealing with traffic safety challenges.

[0003] However, during the automatic cruise flight of an unmanned aerial vehicle, there may be off-road interference objects in the photos collected by the unmanned aerial vehicle, such as buildings, trees, parking lots, etc. These interference objects may be misidentified as traffic targets, affecting the accuracy of event detection. Summary of the Invention

[0004] This application provides a traffic event detection method, device and storage medium based on unmanned aerial vehicles to improve the efficiency and accuracy of traffic event detection.

[0005] To achieve the above object, this application adopts the following technical solutions:

[0006] In a first aspect, a detection method is provided. The detection method includes:

[0007] Obtain a traffic event detection image, which is collected by an image acquisition device of an unmanned aerial vehicle;

[0008] Input the traffic event detection image into a deep neural network model to obtain a point set, where the point set includes the points of the lane lines in the traffic event detection image, and the deep neural network model is used to identify the points with lane line features in the traffic event detection image;

[0009] Fit the point set to obtain a lane line;

[0010] Segment the traffic event detection image based on the lane line to obtain a target detection area;

[0011] Identify the target detection area to obtain target identification information;

[0012] Perform traffic event detection according to the target identification information.

[0013] In some embodiments, the fitting the point set to obtain a lane line includes:

[0014] Traverse the points of the point set in the traffic event detection image according to a preset rule to obtain a starting point;

[0015] Determine a first neighborhood according to the said starting point, and determine the points in the point set of the first neighborhood;

[0016] Fit the points in the point set of the first neighborhood to obtain a sub-lane line in the first neighborhood;

[0017] Delete the points in the point set of the first neighborhood, and determine a second neighborhood according to the end point of the sub-lane line corresponding to the first neighborhood, where the end point is the end point of the sub-lane line corresponding to the first neighborhood that is far from the starting point; determine the second neighborhood according to the end point, and determine the points in the point set of the second neighborhood, fit the points in the point set of the second neighborhood to obtain a sub-lane line in the second neighborhood, and cycle in turn to obtain multiple sub-lane lines, and determine the lane line according to the multiple sub-lane lines.

[0018] In some embodiments, after fitting the point set to obtain a lane line, if the number of lane lines is multiple, the method further includes:

[0019] Determine the slope of the lane line according to the starting point and the end point of the multiple neighborhoods corresponding to the lane line;

[0020] Determine a target lane line according to the slopes of the multiple lane lines, where the slopes of the multiple lane lines of the same target lane line are the same or close.

[0021] In some embodiments, if the number of target lane lines is multiple, the method further includes:

[0022] Determine a correlation slope according to the starting point and the end point of the multiple neighborhoods corresponding to any two target lane lines among the multiple target lane lines;

[0023] If the correlation slope is the same or close to the slopes of the two target lane lines, the two target lane lines are the same lane line;

[0024] If the correlation slope is not the same or not close to the slopes of the two target lane lines, the two target lane lines are different lane lines.

[0025] In some embodiments, the step of inputting the traffic event detection image into the deep neural network model to obtain a point set includes:

[0026] If there is a missing frame in the traffic event detection image, perform a first filtering process on the traffic event detection image to obtain a first image to be processed, and input the first image to be processed into the deep neural network model to obtain a point set;

[0027] If the body temperature of the UAV is greater than or equal to the preset temperature, perform a second filtering process on the traffic event detection image to obtain a second image to be processed, and input the second image to be processed into the deep neural network model to obtain a point set;

[0028] If there is no missing frame in the traffic event detection image and the body temperature of the UAV is less than the preset temperature, input the traffic event detection image into the deep neural network model to obtain a point set.

[0029] In some embodiments, the traffic event detection according to the target recognition information includes:

[0030] Based on the target recognition information, track the targets in the target detection area through a target tracking method to obtain the trajectory information of each target;

[0031] Perform traffic event detection according to the target recognition information and the trajectory information of each target.

[0032] In some embodiments, the target tracking method is ByteTrack. The tracking of the targets in the target detection area through the target tracking method based on the target recognition information to obtain the trajectory information of each target includes:

[0033] Based on the target recognition information, predict the prediction box corresponding to the current position of the target through a Kalman filter. The target recognition information includes the detection box of the target, and the KF state vector of the Kalman filter is:

[0034] x k =[x c (k), y c (k), w(k), h(k), x v (k), y v (k), w′(k), h′(k)] T ;

[0035] Where, x and y are the coordinates of the detection box, w is the predicted width of the detection box, h is the height of the detection box, x v is the speed in the x direction, y v is the speed in the y direction, w′ is the area of the detection box, and h′ is the height change rate of the detection box;

[0036] Determine the similarity between the detection box and the prediction box; and match the detection box based on the similarity to obtain the trajectory information of the target.

[0037] In some embodiments, the fitting of the point set to obtain the lane line includes:

[0038] Fitting a point set based on lane line features to obtain a lane line; the lane line features include color features and straight line features.

[0039] In some embodiments, the traffic event detection according to the target recognition information includes:

[0040] Obtaining the actual speed of the target:

[0041]

[0042] where v car is the actual speed of the target, (x1, y1) and (x2, y2) are the central coordinates of the starting point and the ending point of the target; v uav is the actual speed of the UAV; dt is the time difference between the starting point and the ending point of the target; h is the flight altitude of the UAV, f is the camera focal length of the UAV; α is the angle between the lower part of the UAV camera lens and the vertical direction, α is greater than 0° and less than 90°, β is the field angle of the UAV camera lens; p x is the horizontal pixel size of the UAV lens, p y is the vertical pixel size, C is a correction function,

[0043] C(y) = a*y 2 + b*y + c,

[0044] Performing traffic event detection based on the target recognition information and the actual speed.

[0045] In a second aspect, a traffic event detection device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the detection method in any optional implementation manner of the first aspect are implemented.

[0046] In a third aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the detection method described in any item of the first aspect are implemented.

[0047] The detection method, device, and storage medium provided by the embodiments of the present application have the following beneficial effects:

[0048] To achieve extensive coverage and real-time detection of traffic areas, drones can be used. Drones have high flexibility and an aerial perspective, can easily fly over obstacles, obtain comprehensive traffic condition information, and improve the coverage and accuracy of traffic incidents. Due to the aerial perspective and mobility flexibility, there are interference areas in the images collected by drones, such as interference outside the road. In this application, points corresponding to vehicle lines are identified, and then the identified points are fitted to obtain lane lines. Based on the lane lines, the traffic incident detection images are segmented, and only the segmented areas need to be identified and traffic incidents detected, which improves the efficiency of traffic incident detection. Moreover, the interference areas are removed through segmentation, enhancing the accuracy of traffic incident detection. Description of the Drawings

[0049] Figure 1 It is a schematic flowchart of a traffic incident detection method based on drones provided by an embodiment of this application;

[0050] Figure 2 It is a schematic diagram of traffic incident detection image segmentation provided by an embodiment of this application;

[0051] Figure 3 It is a schematic diagram of the determination process of lane lines in each neighborhood provided by an embodiment of this application;

[0052] Figure 4 It is a schematic diagram of partial occlusion of lane lines provided by an embodiment of this application;

[0053] Figure 5 It is a schematic diagram of the comparison effect of detection frames provided by an embodiment of this application;

[0054] Figure 6 It is a schematic diagram of the state of the Nth frame provided by an embodiment of this application;

[0055] Figure 7 It is a schematic diagram of the (N + 50)th frame provided by an embodiment of this application;

[0056] Figure 8 It is a side view of a drone taken in an embodiment of this application;

[0057] Figure 9 It is a schematic diagram of a static experiment test provided by an embodiment of this application;

[0058] Figure 10 It is a schematic structural diagram of a traffic incident detection device provided by an embodiment of this application. Detailed Implementation Manner

[0059] It should be noted that the terms used in the embodiments of the present application are only for explaining the specific embodiments of the present application, rather than aiming to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, "a plurality" means two or more than two, "at least one", "one or more" means one, two or more than two. The terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0060] Reference to "one embodiment" or "some embodiments" etc. described in this specification means that a specific feature, structure or characteristic described in combination with the embodiment is included in one or more embodiments of the present application. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] The embodiments of the present application provide a traffic event detection method based on an unmanned aerial vehicle (UAV). The detection method performs point detection on the traffic event detection images collected by the UAV to determine the point set of the lane lines in the traffic event detection images, fits the point set, and performs image segmentation based on the fitted lane lines, and performs recognition and traffic event detection according to the segmented images, which can reduce the interference of targets outside the lane lines, reduce the traffic event detection calculation amount, improve the traffic event detection efficiency, and improve the detection accuracy.

[0062] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a traffic event detection method based on an unmanned aerial vehicle provided by the embodiments of the present application. The detection method includes the following steps:

[0063] S101. Obtain a traffic event detection image, which is collected by an image acquisition device of the UAV.

[0064] During the UAV cruise, an image of the target area, that is, a traffic event detection image, is obtained through an image acquisition device (such as a camera or a video camera of the UAV) installed on the UAV, so as to determine whether there is a traffic event in the target area and the type of the traffic event through the traffic event detection image. The event can be a traffic event, such as a rear-end collision of two vehicles.

[0065] Optionally, the drone is a drone that can automatically cruise. For example, after setting a cruising area, the drone can determine a corresponding cruising route based on the cruising area.

[0066] Optionally, the cruising area of the drone is the area where the highway is located. Since there are limited cameras on the highway and the traffic conditions of the entire section cannot be comprehensively obtained, traffic event detection can be performed on the blind spots of the highway cameras through the images obtained by the drone's cruise.

[0067] S102: Input the traffic event detection image into a deep neural network model to obtain a point set. The point set includes the points of the lane lines in the traffic event detection map. The deep neural network model is used to identify the points with lane line features in the traffic event detection map.

[0068] Optionally, the deep neural network model is an ENet network model, which has a fast recognition speed for images. Use the ENet network model to detect the road line points of the input traffic event detection image and collect the point set of the points that may be road lines in the image. The data structure of "point" stores the coordinate data of the point sets of all road line positions.

[0069] Optionally, for the spaced lane lines, processing can be performed during labeling. For example, multiple points on multiple extension lines on the same straight line are all regarded as the points of the lane line, so that the blank intervals of a lane line are all regarded as part of the lane line.

[0070] Optionally, the deep neural network model can identify the points with lane line features in the traffic event detection map. Among them, the features include color features (such as white and yellow) and straight line features (such as continuous straight lines).

[0071] Optionally, the ENet network model is implemented by Python and built using the Pytorch framework. A dataset is constructed using the tusimple open-source dataset and the self-built drone aerial photography data; including 7000 road images under different traffic conditions in the tusimple open-source dataset and the self-built drone aerial photography data dataset. Road images are obtained by frame extraction of the videos recorded by the drone's automatic cruise, with a total of 3000 images. There are a total of 10000 road images, among which the training set accounts for 70% and the test set accounts for 30%. The LabelImg annotation software is used to annotate the images to generate json files for subsequent model training and testing.

[0072] The parameters are set as Epoch = 300, batch_size = 16, Adam optimizer (lr = 1e-4, weight_decay = 0.0003), the learning rate decay interval step_size = 10, and the learning rate adjustment multiple gamma = 0.1. The test results are shown in Table 1.

[0073] Table 1

[0074] Network model Lane line detection accuracy FPS ENet 90.17％ 35.13

[0075] As can be seen from Table 1, the lane line detection accuracy of this model is high, and the FPS (Frames Per Second) performance is also high.

[0076] S103. Fit the point set to obtain the lane line.

[0077] Optionally, after obtaining the point set, perform fitting through the Random Sample Consensus (RANSAC) method to find the straight line corresponding to the point set. The RANSAC method is a robust fitting method that can effectively process data sets containing outliers and has a fast fitting speed.

[0078] Optionally, the point set can be fitted based on lane line features to reduce the off-road interference on the highway. For example, the lane line feature is a continuous straight line, and the lines of roadside parking lots are line segments. Among them, the features include color features (such as white and yellow) and straight line features (such as continuous straight lines, rather than line segments).

[0079] Optionally, first screen the point set based on color features to obtain a target point set, where the points in the target point set meet the color features; then fit the target point set to obtain an initial lane line, and fit the initial lane line based on straight line features to obtain the lane line, where the lane line is a lane line that meets the straight line features.

[0080] S104. Segment the traffic event detection image based on the lane line to obtain the target detection area.

[0081] Among them, the target detection area is a partial area in the traffic event detection image, that is, the partial area segmented from the traffic event detection image. Through segmentation, the area for event detection is reduced, and the detection speed is improved.

[0082] Please refer to Figure 2 , Figure 2 which is a schematic diagram of traffic event detection image segmentation provided by an embodiment of this application.

[0083] Figure 2 In the figure, the target is a vehicle, area A is the segmented area, and areas B and C are the parking lot area and the building area. Since there are vehicles in the parking lot and building areas, areas B and C are interference areas. Through the above steps, the interference areas can be removed, and the required detection area, that is, the target detection area (area A), can be segmented.

[0084] Optionally, this application is applied to traffic event detection on highways, which include expressways, express roads, etc.

[0085] Since the traffic event detection images are collected by an automatically cruising drone, and the drone is in continuous motion, the traffic event detection images include not only the highway area but also vehicle interference in other areas. Through the above segmentation method, the highway area (i.e., the target detection area) can be segmented, and subsequent detection is only performed on the highway area, which reduces the computational complexity of traffic event detection and can also reduce interference from other areas.

[0086] Optionally, the target detection area is the area enclosed by the outermost road line among multiple road lines in the traffic event detection image.

[0087] S105. Identify the target detection area to obtain target identification information.

[0088] Optionally, the target detection area of the traffic event detection image can be input into a neural network model, and the neural network model is used to identify the targets in the target detection area to obtain target identification information.

[0089] Optionally, the target identification information includes target category, target location information, and confidence information. Among them, the target category can be multiple types of traffic participants, such as pedestrians, non-motor vehicles, motorcycles, cars, trucks, buses, etc.

[0090] S106. Perform traffic event detection based on the target identification information.

[0091] For example, if the position areas of a car and a truck coincide at the same time, it can be determined that there is a vehicle collision event at the corresponding position at that moment.

[0092] Thus, in order to achieve wide coverage and real-time detection of traffic areas, a drone can be used. The drone has high flexibility and an aerial perspective, can easily fly over obstacles, obtain comprehensive traffic condition information, and improve the coverage range and accuracy of traffic events. Due to the aerial perspective and moving flexibility, the images collected by the drone have interference areas, such as interference outside the road. In this application, the points corresponding to the vehicle lines are identified, and then the identified points are fitted to obtain lane lines. Based on the lane lines, the traffic event detection image is segmented, and only the segmented area needs to be identified and traffic event detection is performed, which improves the traffic event detection efficiency and removes the interference area through segmentation, improving the accuracy of traffic event detection.

[0093] It is easy to understand that the traffic event detection images collected during the drone's cruise are easily subject to various interferences. For example, when the drone is close to the vicinity of the wire, it is easy to cause electromagnetic interference to the antenna, resulting in frame loss in the video detection image. Among them, frame loss is caused by problems such as the network, camera sensor, and transcoding device, resulting in unsmooth video images and lost frames, and the picture received by the algorithm is empty. During high-temperature weather in summer or during the operation time of the drone, it may cause the temperature of the drone body to be too high. For example, when it is greater than 50°, it affects the image acquisition device of the drone, resulting in distortion in the traffic event detection image of the image acquisition device. In order to improve the accuracy of traffic event detection, before performing traffic event detection based on the traffic event detection image, the traffic event detection image can be preprocessed.

[0094] In some embodiments, S102 includes:

[0095] If there are missing frames in the traffic event detection image, perform a first filtering process on the traffic event detection image to obtain a first image to be processed, and input the first image to be processed into the deep neural network model to obtain a point set;

[0096] If the temperature of the drone body is greater than or equal to the preset temperature, perform a second filtering process on the traffic event detection image to obtain a second processed image, and input the second image to be processed into the deep neural network model to obtain a point set;

[0097] If there are no missing frames in the traffic event detection image and the temperature of the drone body is less than the preset temperature, input the traffic event detection image into the deep neural network model to obtain a point set.

[0098] In this way, when the traffic event detection images collected by the drone are affected differently, different filtering processing methods can be adopted for them. Through filtering processing, the influence on the image can be eliminated, and the accuracy of traffic event detection can be improved. Since only the traffic event detection images affected are filtered to improve the image quality; for traffic event detection images that are not affected, such as those without missing frames or the drone temperature is less than the preset temperature, there is no need to perform filtering processing to improve the efficiency of image processing.

[0099] Optionally, after obtaining the traffic event detection image, it can be detected whether the traffic event detection image meets the missing frame condition. If it meets the missing frame condition, there are missing frames in the traffic event detection image; if it does not meet the missing frame condition, there are no missing frames in the traffic event detection image.

[0100] Optionally, the missing frame condition is that the cumulative number of missing frames in the video frames per minute is greater than 10 times, then there are missing frames in the traffic event detection image; the cumulative number of missing frames in the video frames per minute is less than or equal to 10 times, then there are no missing frames in the traffic event detection image.

[0101] Optionally, the VideoCapture method in the OpenCV library can be used to detect whether there are missing frames in the video image.

[0102] Optionally, a temperature sensor is installed on the drone. The temperature sensor is used to sense the body temperature of the drone and transmit this information to the traffic event detection device via a communication link.

[0103] Optionally, the first filtering process is mean filtering, which is used to solve the problem of missing frames in the detected video image.

[0104] Optionally, the second filtering process is a fast Gaussian filtering operation.

[0105] In the Gaussian filtering process, the two-dimensional Gaussian function is the basis for constructing the Gaussian filter, and its formula is as follows:

[0106]

[0107] Where x and y are the center point coordinates, and σ is the variance. According to the separability of the Gaussian function, the two-dimensional Gaussian function is derived into the product of one-dimensional Gaussian functions on the X-axis and Y-axis, so as to first perform convolution on the entire image matrix in the X-axis direction, and then perform convolution on the entire image matrix in the Y-axis direction to achieve fast processing of the image by Gaussian filtering. Specifically, as follows:

[0108]

[0109] Since the traffic accident detection image is a visible light image with three RGB channels, the image after Gaussian filtering is as follows:

[0110] G(x, y) R,G,B = ∑ i∈(R,G,B) G(x, y) i ;

[0111] Among them, mean filtering replaces the pixel value of each point in the image with the mean value of the pixel values of all points, and adds the corresponding pixel values of the three channels to obtain the processed image. Its formula is as follows:

[0112]

[0113] Where f(x, y) is the original image, and A(x, y) R,G，B is the image after mean filtering processing, s is the set of pixels in the template centered on the point (x, y), and the template size is m × n.

[0114] In some embodiments, S103 includes:

[0115] Traverse the points in the point set of the traffic event detection image according to preset rules to obtain the first point;

[0116] Determine the first neighborhood according to the first point, and determine the points in the point set of this first neighborhood;

[0117] Fit the points in the point set of the first neighborhood to obtain the sub-lane line in the first neighborhood, where the sub-lane line is a line segment;

[0118] Delete the points in the point set of the first neighborhood, and determine the second neighborhood according to the end point of the sub-lane line corresponding to the first neighborhood. This end point is the end point of the sub-lane line corresponding to the first neighborhood that is far from the first point;

[0119] Determine the second neighborhood according to this end point, and determine the points in the point set of the second neighborhood. Fit the points in the point set of the second neighborhood to obtain the sub-lane line in the second neighborhood, and delete the points in the point set of the second neighborhood. Loop in turn to obtain multiple sub-lane lines, and determine the lane line according to the multiple sub-lane lines. In this way, after determining the point set in the traffic event detection image, traverse the points in the traffic event detection image according to preset rules. After obtaining the first point (i.e., the first point), determine the first neighborhood according to the first point, and then traverse the points in the point set of the first neighborhood. By fitting the points in the point set of the first neighborhood, the sub-lane line in the first neighborhood can be determined, and then all the points in the point set of the first neighborhood are deleted; then use the end point of the sub-lane line in the first neighborhood that is far from the first point as the first point of another neighborhood, and loop in turn to obtain the sub-lane lines of multiple neighborhoods. That is, divide the points in the point set into multiple neighborhoods, fit multiple points in the point set of each neighborhood to obtain the sub-lane lines of multiple neighborhoods. After fitting the sub-lane line through the points in the point set of this neighborhood, delete the points in the point set of this neighborhood. When fitting the sub-lane line in the next neighborhood, there is no need to fit the points that have been used in the previous neighborhood, which improves the fitting efficiency of the sub-lane line. It reduces the storage of point information in the point set and improves the data processing efficiency. That is, in the process of obtaining the sub-lane lines of multiple neighborhoods in the kernel, delete the points that have been used in the fitting process, and the next neighborhood fitting process does not need to use this point, which improves the fitting efficiency.

[0120] Optionally, if the end point of one neighborhood is used as the first point of another neighborhood, and the neighborhood determined based on this first point does not contain other points in the point set except the end point, stop traversing. Thus, all the lane lines in the traffic event detection image are obtained.

[0121] Optionally, the specific rule can be a specific direction, for example, traverse from the lower left corner to the upper right corner of the traffic event detection image.

[0122] Optionally, determining the first neighborhood according to the first point includes: drawing a circle with the first point as the center and a preset length as the radius. This circle is the first neighborhood. The points in the point set within the first neighborhood are the points that meet the neighborhood threshold of the first neighborhood.

[0123] Optionally, the preset length is determined based on the height of the drone and the angle between the drone and the first point. Since the images captured by the drone may cause image distortion, the distortion degrees are different for different heights and angles. Optionally, the determining of the lane line according to multiple sub-lane lines includes: determining target sub-lane lines according to multiple sub-lane lines, and at least one endpoint of each target sub-lane line coincides with an endpoint of another sub-lane line.

[0124] Exemplarily, please refer to Figure 3 in (A), from the lower left corner to the upper right corner, traverse the points in the point set of the traffic event detection image to obtain the first point (x1, y1); draw a circle with this first point as the center and the preset length as the radius. This circle is the first neighborhood. Traverse the points in the point set of the first neighborhood, and fit the points in the point set of the first neighborhood through RANSAC to obtain the sub-lane lines in the first neighborhood. Delete the points in the point set of the first neighborhood. An endpoint of the sub-lane line in the first neighborhood that is far from the first point is used as the tail point, that is, (x2, y2). Refer to Figure 3 in (B), use the tail point (x2, y2) as the first point (x3, y3) of the second neighborhood, determine the second neighborhood based on this first point, traverse the points in the point set of the second neighborhood, and fit the points in the point set of the second neighborhood through RANSAC to obtain the sub-lane lines in the second neighborhood. Delete the points in the point set of the second neighborhood. An endpoint of the sub-lane line in the second neighborhood that is far from the first point is used as the tail point, that is, (x4, y4). Loop in sequence to obtain Figure 3 the multiple sub-lane lines fitted in (C). Form the lane line according to the multiple sub-lane lines. By deleting the points that have been used in each neighborhood, when fitting the sub-lane lines in the next neighborhood, there is no need to calculate the points in the point set of the previous neighborhood, reducing the number of points to be fitted and improving the efficiency of fitting the sub-lane lines.

[0125] Optionally, during the process of traversing the points in the point set of the traffic event detection image, if the boundary of the traffic event detection image is traversed, the traversal stops.

[0126] It is easy to understand that in the actual scenario, as Figure 4 shown in (A), the lane line is blocked by an obstacle (such as a garbage bag), resulting in the inability to identify the points of this section of the lane line, and causing interruptions in the multiple sub-lane lines of the lane line.

[0127] When shooting video or image data with a drone, the phenomenon of "objects being larger when closer and smaller when farther" is a common problem. Because from the perspective of the drone, objects closer occupy more pixel points in the image. As Figure 4As shown in (B), there are multiple actually parallel lane lines. In the perspective of the drone, the multiple lane lines intersect in the distance. Due to the distortion of the lane lines, the slopes of the straight lines where the multiple lane lines are located change. Therefore, after determining the lane lines, if the number of lanes is multiple, since each lane line is formed by sub-lane lines corresponding to multiple neighborhoods, the slope of each lane line can be determined according to the starting point and the ending point of each lane line corresponding to each neighborhood. If the slopes of two lane lines are the same or similar, then the two lane lines belong to the same target lane line. After S103, if the number of lane lines includes multiple, then the method further includes: determining the slope of the lane line according to the starting point and the ending point of the neighborhood corresponding to the lane line; determining the target lane line according to the slopes of the multiple lane lines, wherein the slopes of the multiple lane lines of the same target lane line are the same or similar. For example, if the difference between the slopes of two lane lines is less than or equal to a preset threshold, then the slopes of the two lane lines are similar; if the difference between the slopes of two lane lines is greater than the preset threshold, then the slopes of the two lane lines are not similar.

[0128] As Figure 4 shown in (C), there are three lane lines, and each lane line is formed by sub-lane lines corresponding to multiple neighborhoods. The straight lines where the three lane lines are located can be determined based on the starting point and the ending point of the sub-lane lines corresponding to each neighborhood: y = kx + b, where k is the slope of each lane line. Lanes with k values close to or the same belong to the same lane line. For example Figure 4 in (C), the k values corresponding to ② and ③ are the same, and the two belong to the same lane line, and ① belongs to another lane line.

[0129] Since the bending degree of the road lines closer to the center of the road becomes smaller and smaller, and the k values are also closer. If the slopes of multiple lane lines are all similar, the relevant slope can be determined based on the starting point and the ending point of any two lane lines corresponding to the multiple lane lines with the same or similar slopes. If the relevant slope is the same or similar to the slope of each lane, it is determined that the two lane lines belong to the same lane line.

[0130] Exemplarily, the coordinates of the starting points and the ending points of the multiple neighborhoods corresponding to the three lane lines are respectively: [(x a , y a )......(x b , y b )], [(x i , y i )......(x j , y j )], [(x m , y m )......(x n , y n)], if the slopes of the three lane lines ①, ②, and ③ are all approximately the same, that is, k1≈k2≈k3, where k1, k2, and k3 are the slopes of the three lane lines ①, ②, and ③ respectively, then calculate the relevant slopes of ①②, ②③, and ①③. Among them, the relevant slope of ②③ is: The relevant slopes of ①② and ①③ are respectively: That is, the relevant slope of ②③ is close to the slopes of ② and ③, that is, the lane lines ② and ③ are the same lane line; the relevant slope of ①② is different from the slopes of ② and ①; the relevant slope of ①③ is different from the slopes of ① and ①, that is, the lane line ① and the lane line ② are not the same lane line; the lane line ① and the lane line ③ are not the same lane line.

[0131] In some embodiments, if the number of the target lane lines is multiple, the method further includes:

[0132] Determine the relevant slope according to the starting points and ending points of multiple neighborhoods corresponding to any two target lane lines among the multiple target lane lines;

[0133] If the relevant slope is the same as or close to the slopes of the two target lane lines, the two target lane lines are the same lane line;

[0134] If the relevant slope is different from or not close to the slopes of the two target lane lines, the two target lane lines are different lane lines.

[0135] Optionally, if the difference between the relevant slope and the slopes of the two target lane lines is less than or equal to a preset threshold, the relevant slope is close to the slopes of the two target lane lines; if the difference between the relevant slope and the slopes of the two target lane lines is greater than the preset threshold, the relevant slope is not close to the slopes of the two target lane lines.

[0136] It is easy to understand that there are usually multiple vehicles running on the highway at the same time. After detecting multiple targets in the target detection area to obtain target recognition information, the multiple targets in the target detection area can be tracked by the ByteTrack method based on the target recognition information to obtain the trajectory information of each target. Then the above S106 includes:

[0137] Track the multiple targets in the target detection area by ByteTrack according to the target recognition information to obtain the trajectory information of each target, where the trajectory information includes the target trajectory and the target speed.

[0138] Detect traffic events according to the target recognition information and the trajectory information.

[0139] Exemplarily, the coordinate point set of the target is recorded at a frequency of 10 frames within 30 seconds, and the points are sequentially connected by lines to form a trajectory; at an interval of 1 second, the moving speed of the target on the image.

[0140] According to the target recognition information, ByteTrack is used to track multiple targets within the target recognition area, and the trajectory information of each target is obtained through the following process:

[0141] 1. Detection box classification: According to the detection score threshold, all detection boxes are divided into high-score detection boxes (D_high) and low-score detection boxes (D_low). The high-score threshold is usually set to a relatively high value (such as 0.6), while the low-score threshold is relatively low (such as 0.1).

[0142] 2. Trajectory prediction: The Kalman filter is used to predict the new position in the current frame of each trajectory. The Kalman filter is based on a linear motion model and can predict the position of the target in the next frame.

[0143] 3. Data association:

[0144] First matching: The first association is performed between the high-score detection boxes (D_high) and all trajectories (including lost trajectories T_lost). The similarity can be calculated by the IOU (Intersection over Union) between the detection box and the predicted trajectory box. Then, the Hungarian algorithm is used to complete the matching based on similarity.

[0145] Second matching: After the first association, the second association is performed between the low-score detection boxes (D_low) and the remaining unmatched trajectories (T_remain). The association process is similar to the first one, but usually only the IOU is used as the similarity metric because the low-score detection boxes may contain severe occlusion or motion blur and the appearance features are unreliable.

[0146] 4. Trajectory management:

[0147] For unmatched trajectories, a certain lifespan (such as 30 frames) is reserved. If no detection box is matched within this period, it will be deleted.

[0148] For high-score detection boxes that do not match a trajectory, if their score is higher than a certain threshold (such as 0.7) and they can be detected continuously for multiple frames, a new trajectory will be assigned to them.

[0149] In the process of trajectory prediction by ByteTrack, the Kalman filter is used to predict the new position in the current frame of each trajectory. Specifically, the predicted box corresponding to the current position of the target is predicted through the Kalman filter. The target recognition information includes the detection box of the target, and the KF state vector of the Kalman filter is:

[0150] x k = [x c (k), y c (k), w(k), h(k), x v (k), y v (k), w′(k), h′(k)] T ;

[0151] Where x and y are the coordinates of the detection box, w is the predicted width of the detection box, h is the height of the detection box, x v is the velocity in the x direction, y v is the velocity in the y direction, w′ is the area of the detection box, and h′ is the rate of change of the height of the detection box.

[0152] Please refer to Figure 5 , Figure 5 which is a schematic diagram of the comparison effect of a detection box provided by an embodiment of the present application. Among them Figure 5 A is the existing detection box, and B is the detection box determined based on the Kalman filter of the present application. As can be seen from Figure 5 , by improving the state vector of the Kalman filter, the detection box A is larger than the detection box B, and the trajectory of the target is determined based on the detection box. The position of the target can be accurately limited based on a more accurate detection box.

[0153] Please refer to Figure 6 and Figure 7 , Figure 6 which is a schematic diagram of the state of the Nth frame provided by an embodiment of the present application, Figure 7 and

[0154] which is a schematic diagram of the (N + 50)th frame provided by an embodiment of the present application. Taking the vehicles numbered 87, 88, and 92 in the right lane as an example, by comparing the nth frame and the (n + 50)th frame, the trajectory of each vehicle can be clearly determined, thereby achieving effective tracking of the target vehicle.

[0155] In some embodiment sets, S106 includes:

[0156] Obtaining the actual speed of the target:

[0157]

[0158] where v car is the actual speed of the target, (x1, y1) and (x2, y2) are the central coordinates of the starting point and the central coordinates of the ending point of the target; v uavis the actual speed of the drone; dt is the time difference of the target from the starting point to the ending point; h is the flight altitude of the drone, f is the camera focal length of the drone; α is the angle between the lower part of the drone camera lens and the vertical direction, where α is greater than 0° and less than 90°, β is the viewing angle of the drone camera lens; p x is the horizontal pixel size of the drone lens, p y is the vertical pixel size. C is the correction function;

[0159] Traffic event detection is performed based on the target recognition information and the actual speed.

[0160] Please refer to Figure 8 , Figure 8 is for the drone to take a side view, where β is the viewing angle of the drone lens, a is the angle between the viewing angle line under the drone lens and the vertical direction line. θ is the angle between the connection line from the center point of the vehicle) to the drone lens and the viewing angle line under the drone lens, and θ ∈ [0, β].

[0161] For the same vehicle in the same lane, within the dt time, the vehicle moves from the position (x1, y1) to the position (x2, y2).

[0162] Exemplarily, the sensor size of the drone is 1 inch (about 22.2mm x 16.6mm), and the resolution is 20MP (5472x3648 pixels). It can be derived as follows:

[0163] Horizontal pixel size = sensor width / number of horizontal pixels;

[0164] Vertical pixel size = sensor height / number of vertical pixels;

[0165] The main camera of the drone:

[0166] p y = 22.2mm / 5472pix ≈ 0.00406mm / pix;

[0167] p x = 16.6mm / 3648pix ≈ 0.00455mm / pix;

[0168] In the determination of the vehicle direction, if the vehicle satisfies:

[0169]

[0170] it indicates that the vehicle and the drone move in the same direction, otherwise the vehicle and the drone move in the same direction.

[0171] Among them,

[0172]

[0173]

[0174] Among them, y pix is the ordinate of the center point of the target vehicle in the image, and Y maxpix is the image height (the image size is 1280*720), and the value is 720.

[0175] Among them, g(y) is:

[0176]

[0177] The correction function C is:

[0178] C(y) = a*y 2 + b*y + c;

[0179] Among them, a = -0.0005, b = 0.6541, c = 43.7312. The function is used to eliminate the errors caused by the lens angle shaking (<3°) and the route deviation (<0.5m) during the flight of the UAV.

[0180] Exemplarily, set the constant parameters of the UAV: the value of the lens field angle (β) of the UAV is 77.25°, the value of the angle (α) between the field angle line under the UAV lens and the vertical direction line is 11.50°, the flight speed value of the UAV is 18 km / h, the flight height is 50 m, the lens focal length is 50 mm, and p x is the horizontal pixel size of the UAV lens, which is 0.00455 mm / pix, and p y is the vertical pixel size, which is 0.00406 mm / pix, and the time difference dt between the two displacement points is 10. Figure 9 The engineering vehicle in [] is the test vehicle, parked on the emergency lane, that is, the measured speed is 0, and the actual speed of the vehicle is measured based on the method of the above embodiment.

[0181] Table 2 Experimental result table

[0182]

[0183] As can be seen from Table 2, based on the above speed measurement method, the measured speed on the road can be accurately measured, and the average error is: 1.324 km / h, which is 0.0367 m / s, approximately 0 m / s, and the error is small.

[0184] Please refer to Figure 10, which is a schematic structural diagram of a traffic event detection device provided by an embodiment of the present application. As shown in the figure, the traffic event detection device 100 provided by this embodiment may include: a processor 1040, a memory 1041, and a computer program 1042 stored in the memory 1041 and executable on the processor 1040, such as a program corresponding to the detection method. When the processor 1040 executes the computer program 1042, the steps in the above-mentioned detection method embodiment are implemented, such as Figure 1 the steps shown.

[0185] Exemplarily, the computer program 1042 may be divided into one or more modules / units. One or more modules / units are stored in the memory 1041 and executed by the processor 1040 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 1042 in the traffic event detection device 100.

[0186] Those skilled in the art can understand that Figure 10 this is only an example of the traffic event detection device 100, and does not constitute a limitation on the traffic event detection device 100. It may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0187] The processor 1040 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0188] The memory 1041 may be an internal storage unit of the traffic event detection device 100, such as the hard disk or memory of the traffic event detection device 100. The memory 1041 may also be an external storage device of the traffic event detection device 100, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, or a flash card equipped on an electronic device, etc. Further, the memory 1041 may also include both the internal storage unit and the external storage device of the traffic event detection device 100.

[0189] The memory 1041 is used to store computer programs and other programs and data required by the electronic device. The memory 1041 can also be used to temporarily store the data that has been output or will be output.

[0190] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional unit is used as an example. In practical applications, the above functions can be allocated to different functional units according to needs, that is, the internal structure of the power supply control device is divided into different functional units to complete all or part of the functions described above. Each functional unit in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.

[0191] The embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

[0192] The embodiment of the present application provides a computer program product, when the computer program product runs on the detection device, the detection device implements the steps in the foregoing method embodiments.

[0193] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0194] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by the combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0195] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included within the protection scope of the present application.

[0196] Finally, it should be noted that the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims described.

Claims

1. A traffic incident detection method based on drones, characterized in that: The method comprises: Acquiring a traffic event detection image, where the traffic event detection image is acquired by an image acquisition device of a drone; Inputting the traffic event detection image into a deep neural network model to obtain a point set, wherein the point set includes points of lane lines in the traffic event detection image, and the deep neural network model is used to identify points in the traffic event detection image that have lane line features; Fitting the point set to obtain lane lines; Segmenting the traffic event detection image based on the lane lines to obtain a target detection area; Identifying the target detection area to obtain target identification information; performing traffic event detection according to the target identification information; The step of fitting the point set to obtain lane lines includes: Traversing the points of the point set in the traffic event detection image according to a preset rule to obtain a first point; Determine a first neighborhood according to the first point, and determine points of the point set in the first neighborhood; Fitting the points of the point set in the first neighborhood to obtain sub-lane lines in the first neighborhood; Deleting points of the point set in the first neighborhood, and determining a second neighborhood based on an end point of the sub-lane line corresponding to the first neighborhood, where the end point is an endpoint of the sub-lane line corresponding to the first neighborhood that is away from the first point; Determine a second neighborhood based on the tail point, determine points in the point set in the second neighborhood, fit the points in the point set in the second neighborhood, obtain sub-lane lines in the second neighborhood, repeat in sequence to obtain multiple sub-lane lines, and determine lane lines based on the multiple sub-lane lines.

2. The method according to claim 1, characterized in that After fitting the point set to obtain lane lines, if there are multiple lane lines, the method further includes: Determining the slope of the lane line according to the first and last points of a plurality of neighborhoods corresponding to the lane line; A target lane line is determined according to the slopes of the multiple lane lines, wherein the slopes of the multiple lane lines corresponding to the same target lane line are the same or similar.

3. The method according to claim 2, characterized in that If there are multiple target lane lines, the method further includes: Determining relevant slopes according to the first and last points of a plurality of neighborhoods corresponding to any two target lane lines among the plurality of target lane lines; If the relevant slope is the same as or similar to the slopes of the two target lane lines, then the two target lane lines are the same lane line; If the relevant slope is different from or not similar to the slopes of the two target lane lines, the two target lane lines are different lane lines.

4. The method according to claim 1, wherein The traffic incident detection image is input into a deep neural network model to obtain a point set, including: If there are missing frames in the traffic event detection image, performing a first filtering process on the traffic event detection image to obtain a first image to be processed, and inputting the first image to be processed into a deep neural network model to obtain a point set; If the body temperature of the drone is greater than or equal to a preset temperature, performing a second filtering process on the traffic event detection image to obtain a second image to be processed, and inputting the second image to be processed into a deep neural network model to obtain a point set; If there is no missing frame in the traffic event detection image and the body temperature of the drone is lower than a preset temperature, the traffic event detection image is input into a deep neural network model to obtain a point set.

5. The method according to claim 1, wherein The detecting of traffic incidents according to the target identification information includes: Based on the target recognition information, tracking the targets in the target detection area by a target tracking method to obtain trajectory information of each target; Traffic incident detection is performed based on the target identification information and the trajectory information of each target.

6. The method according to claim 5, characterized in that The target tracking method is ByteTrack, and based on the target identification information, the target in the target detection area is tracked by the target tracking method to obtain the trajectory information of each target, including: Based on the target recognition information, the prediction box corresponding to the current position of the target is predicted by the Kalman filter. The target recognition information includes the detection box of the target. The KF state vector of the Kalman filter is: x k＝ [x c (k),y c (k),w(k),h(k),x v (k),y v (k),w ′ (k),h ′ (k)] T ; Among them, x c ,y c is the coordinate of the detection box, w is the predicted width of the detection box, h is the height of the detection box, x v is the velocity in the x direction, y v is the velocity in the y direction, w ′ is the width change rate of the detection frame, h ′ is the height change rate of the detection frame; Determine the similarity between the detection frame and the prediction frame; and match the detection frames based on the similarity to obtain trajectory information of the target.

7. The method according to claim 1, characterized in that The detecting of traffic incidents according to the target identification information includes: Get the actual speed of the target: Among them, v car is the actual speed of the target, (x1, y1) and (x2, y2) are the center coordinates of the starting point and the end point of the target respectively; v uav is the actual speed of the drone; dt is the time difference between the target's starting point and the target's end point; h is the drone's flight altitude, f is the focal length of the drone's camera; α is the angle between the bottom of the drone's camera lens and the vertical direction, α is greater than 0° and less than 90°, β is the field of view angle of the drone's camera lens; p x is the horizontal pixel size of the drone lens, p y is the vertical pixel size of the drone lens, C is the correction function, y pix is the vertical coordinate of the center point of the target vehicle in the image, y maxpix is the image height in image size; C(y)=a*y 2 +b*y+c; Where a = -0.0005, b = 0.6541, c = 43.7312; Traffic incident detection is performed based on the target recognition information and the actual speed.

8. A traffic incident detection device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the detection method according to any one of claims 1 to 7 are implemented.

Images