Traffic incident detection method and device based on unmanned aerial vehicle, and storage medium

Abstract

The invention is suitable for the technical field of unmanned aerial vehicles, and provides a traffic incident detection method and device based on an unmanned aerial vehicle, and a storage medium, the method comprises the steps: obtaining a traffic incident detection image, the traffic incident detection image being collected by an image collection device of the unmanned aerial vehicle; inputting the traffic event detection image into a deep neural network model to obtain a point set, the point set comprising points of lane lines in the traffic event detection image, and the deep neural network model being used for identifying points with lane line features in the traffic event detection image; fitting the point set to obtain a lane line; segmenting the traffic event detection image based on the lane line to obtain a target detection area; identifying the target detection area to obtain target identification information; and carrying out traffic event detection according to the target identification information. The traffic event detection efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular to a drone-based traffic incident detection method, device, and storage medium. Background Art

[0002] The rapid development of drone technology has brought new possibilities to the field of traffic incident detection. Drones have many advantages such as flexibility, wide monitoring range, and remote control. They can easily fly over obstacles and obtain more comprehensive traffic conditions information, providing strong support for addressing traffic safety challenges.

[0003] However, during the automatic cruising of a drone, off-road interference objects such as buildings, trees, parking lots, etc. may appear in the photos collected by the drone. These interference objects may be mistakenly identified as traffic targets, affecting the accuracy of event detection. Summary of the invention

[0004] The present application provides a traffic incident detection method, device and storage medium based on drones to improve the efficiency and accuracy of traffic incident detection.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a detection method is provided, the detection method comprising:

[0007] Acquire a traffic event detection image, where the traffic event detection image is acquired by an image acquisition device of a drone;

[0008] Inputting the traffic event detection image into a deep neural network model to obtain a point set, wherein the point set includes points of lane lines in the traffic event detection image, and the deep neural network model is used to identify points with lane line features in the traffic event detection image;

[0009] Fitting the point set to obtain lane lines;

[0010] Segmenting the traffic event detection image based on the lane lines to obtain a target detection area;

[0011] Identifying the target detection area to obtain target identification information;

[0012] Traffic incident detection is performed according to the target recognition information.

[0013] In some embodiments, fitting the point set to obtain the lane line includes:

[0014] Traversing the points of the point set in the traffic event detection image according to a preset rule to obtain a first point;

[0015] Determine a first neighborhood according to the first point, and determine points of a point set in the first neighborhood;

[0016] Fitting points of the point set in the first neighborhood to obtain sub-lane lines in the first neighborhood;

[0017] Delete the points in the point set in the first neighborhood, and determine the second neighborhood based on the tail point of the sub-lane line corresponding to the first neighborhood, where the tail point is the endpoint of the sub-lane line corresponding to the first neighborhood away from the first point; determine the second neighborhood based on the tail point, and determine the points in the point set in the second neighborhood, fit the points in the point set in the second neighborhood, obtain the sub-lane line in the second neighborhood, repeat in sequence to obtain multiple sub-lane lines, and determine the lane line based on the multiple sub-lane lines.

[0018] In some embodiments, after fitting the point set to obtain lane lines, if there are multiple lane lines, the method further includes:

[0019] Determine the slope of the lane line according to the first point and the last point of a plurality of neighborhoods corresponding to the lane line;

[0020] The target lane line is determined according to the slopes of the multiple lane lines, wherein the slopes of the multiple lane lines of the same target lane line are the same or similar.

[0021] In some embodiments, if there are multiple target lane lines, the method further includes:

[0022] Determine the relevant slope according to the first point and the last point of a plurality of neighborhoods corresponding to any two target lane lines among the plurality of target lane lines;

[0023] If the relevant slope is the same as or similar to the slopes of the two target lane lines, the two target lane lines are the same lane line;

[0024] If the relevant slope is different from or not similar to the slopes of the two target lane lines, the two target lane lines are different lane lines.

[0025] In some embodiments, the step of inputting the traffic event detection image into a deep neural network model to obtain a point set includes:

[0026] If there are missing frames in the traffic event detection image, performing a first filtering process on the traffic event detection image to obtain a first image to be processed, and inputting the first image to be processed into a deep neural network model to obtain a point set;

[0027] If the fuselage temperature of the drone is greater than or equal to a preset temperature, performing a second filtering process on the traffic event detection image to obtain a second image to be processed, and inputting the second image to be processed into a deep neural network model to obtain a point set;

[0028] If there is no missing frame in the traffic event detection image and the fuselage temperature of the drone is lower than a preset temperature, the traffic event detection image is input into a deep neural network model to obtain a point set.

[0029] In some embodiments, the detecting of traffic incidents according to the target recognition information includes:

[0030] Based on the target recognition information, tracking the targets in the target detection area by a target tracking method to obtain trajectory information of each target;

[0031] Traffic incident detection is performed based on the target identification information and the trajectory information of each target.

[0032] In some embodiments, the target tracking method is ByteTrack, and based on the target identification information, the target in the target detection area is tracked by the target tracking method to obtain the trajectory information of each target, including:

[0033] Based on the target recognition information, a prediction box corresponding to the current position of the target is predicted by a Kalman filter. The target recognition information includes a detection box of the target. The KF state vector of the Kalman filter is:

[0034] x k =[x c (k), y c (k), w(k), h(k), x v (k), y v (k), w′(k), h′(k)] T ;

[0035] Among them, x, y are the coordinates of the detection box, w is the predicted width of the detection box, h is the height of the detection box, and x v is the velocity in the x direction, y v is the speed in the y direction, w′ is the detection frame area, and h′ is the height change rate of the detection frame;

[0036] Determine the similarity between the detection frame and the prediction frame; and match the detection frame based on the similarity to obtain trajectory information of the target.

[0037] In some embodiments, fitting the point set to obtain the lane line includes:

[0038] The point set is fitted based on the lane line features to obtain the lane line; the lane line features include color features and straight line features.

[0039] In some embodiments, the detecting of traffic incidents according to the target recognition information includes:

[0040] Get the actual speed of the target:

[0041]

[0042] Among them, v car is the actual speed of the target, (x1, y1) and (x2, y2) are the center coordinates of the starting point and the end point of the target; v uav is the actual speed of the drone; dt is the time difference from the starting point to the end point of the target; h is the flight altitude of the drone, f is the focal length of the drone's camera; α is the angle between the bottom of the drone's camera lens and the vertical direction, α is greater than 0° and less than 90°, β is the field of view angle of the drone's camera lens; p x is the horizontal pixel size of the drone lens, p y is the vertical pixel size, C is the correction function,

[0043] C(y)=a*y 2 +b*y+c,

[0044] Traffic incident detection is performed based on target recognition information and actual speed.

[0045] In a second aspect, a traffic event detection device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the detection method in any optional implementation of the first aspect are implemented.

[0046] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the detection method as described in any one of the first aspects are implemented.

[0047] The detection method, device and storage medium provided in the embodiments of the present application have the following effective effects:

[0048] In order to achieve wide coverage and real-time detection of traffic areas, drones can be used. Drones have high flexibility and high-altitude viewing angles, can easily fly over obstacles, obtain comprehensive traffic conditions information, and improve the coverage and accuracy of traffic events. Due to the high-altitude viewing angle and flexibility of movement, there are interference areas in the images collected by drones, such as interference outside the road. This application identifies the points corresponding to the vehicle lines, and then fits the identified points to obtain lane lines, and segments the traffic event detection image based on the lane lines. It is only necessary to identify and detect traffic events in the segmented areas, which improves the efficiency of traffic event detection, and removes interference areas through segmentation, thereby improving the accuracy of traffic event detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart of a method for detecting traffic incidents based on a drone provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of image segmentation for traffic incident detection provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a lane line determination process for each neighborhood provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a lane line being partially blocked provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of a comparison effect of a detection frame provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the state of the Nth frame provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of the N+50th frame provided in an embodiment of the present application;

[0056] Figure 8 A side view of a drone according to an embodiment of the present application;

[0057] Fig. 9 A schematic diagram of a static experimental test of an embodiment of the present application;

[0058] Fig.10 A schematic diagram of the structure of a traffic incident detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] It should be noted that the terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, "multiple" refers to two or more than two, and "at least one", "one or more" refers to one, two or more. The terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, it is defined that the "first" and "second" features can explicitly or implicitly include one or more of the features.

[0060] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] An embodiment of the present application provides a traffic event detection method based on a drone. The detection method performs point detection on a traffic event detection image collected by a drone to determine a point set of lane lines in the traffic event detection image, fits the point set, and performs image segmentation based on the fitted lane lines. Recognition and traffic event detection are performed based on the segmented image, thereby reducing interference from targets outside the lane lines, reducing the amount of traffic event detection calculations, improving traffic event detection efficiency, and improving detection accuracy.

[0062] See also Figure 1 , Figure 1 A flow chart of a traffic incident detection method based on a drone provided in an embodiment of the present application, the detection method comprises the following steps:

[0063] S101. Acquire a traffic event detection image, where the traffic event detection image is acquired by an image acquisition device of a drone.

[0064] During the drone's cruising process, an image of the target area, namely a traffic event detection image, is obtained through an image acquisition device installed on the drone (such as a camera or a video camera of the drone), so as to determine whether there is a traffic event in the target area and the type of traffic event through the traffic event detection image, wherein the event may be a traffic event, such as a rear-end collision between two vehicles.

[0065] Optionally, the drone is a drone capable of automatic cruising. For example, after setting a cruising area, the drone can determine a corresponding cruising route based on the cruising area.

[0066] Optionally, the drone's cruising area is the area where the expressway is located. Due to the limited number of cameras on the expressway, it is impossible to fully obtain the traffic conditions of the entire road section. The images obtained by the drone cruising can be used to detect traffic incidents in the blind spots of the expressway's cameras.

[0067] S102: Input the traffic event detection image into a deep neural network model to obtain a point set, where the point set includes points of lane lines in the traffic event detection image. The deep neural network model is used to identify points with lane line features in the traffic event detection image.

[0068] Optionally, the deep neural network model is an ENet network model, which has a fast image recognition speed. The ENet network model is used to detect road line points on the input traffic event detection image, and a point set that may be a road line in the image is collected. The "point" data structure is obtained, which stores the point set coordinate data of all road line points.

[0069] Optionally, the spaced lane lines may be processed during labeling, for example, multiple points of multiple extended lines on the same straight line may be regarded as points of the lane line, so that the blank spaces of a lane line may be regarded as part of the lane line.

[0070] Optionally, the deep neural network model may identify points having lane line features in the traffic event detection image, wherein the features include color features (eg, white and yellow) and straight line features (eg, continuous straight lines).

[0071] Optionally, the ENet network model is implemented in Python and built using the Pytorch framework. The dataset is constructed using the tusimple open source dataset and self-built drone aerial photography data; including the tusimple open source dataset, which contains 7,000 road images under different traffic conditions and the self-built drone aerial photography data dataset. The road images are obtained by cutting frames of the video recorded by the drone's automatic cruise, totaling 3,000 images. There are a total of 10,000 road images, of which 70% are training sets and 30% are test sets. The images are annotated using LabelImg annotation software to generate json files for subsequent model training and testing.

[0072] The parameters used are Epoch=300, batch_size=16, Adam optimizer (lr=1e-4, weight_decay=0.0003), learning rate decrease interval step_size=10, and learning rate adjustment multiple gamma=0.1. The test results are shown in Table 1.

[0073] Table 1

[0074] Network Model Lane detection accuracy FPS ENet 90.17％ 35.13

[0075] It can be seen from Table 1 that the lane line detection accuracy of the model is high and the FPS (Frames Per Second) performance is high.

[0076] S103: Fit the point set to obtain lane lines.

[0077] Optionally, after obtaining the point set, a random sampling consensus (RANSAC) method is used for fitting to find a straight line corresponding to the point set. The RANSAC method is a robust fitting method that can effectively process a data set containing abnormal points and has a fast fitting speed.

[0078] Optionally, the point set can be fitted based on lane line features to reduce off-road interference on expressways, such as lane line features as continuous straight lines and roadside parking lot lines as line segments. The features include color features (such as white and yellow) and line features (such as continuous straight lines, not line segments).

[0079] Optionally, the point set is first screened based on color features to obtain a target point set, wherein the points in the target point set satisfy the color features; then the target point set is fitted to obtain an initial lane line, and the initial lane line is fitted based on a straight line feature to obtain a lane line, wherein the lane line is a lane line that satisfies the straight line feature.

[0080] S104: Segment the traffic event detection image based on lane lines to obtain a target detection area.

[0081] The target detection area is a partial area in the traffic event detection image, that is, a partial area segmented from the traffic event detection image. The segmentation is used to reduce the area for event detection and improve the detection speed.

[0082] See also Figure 2 , Figure 2 A schematic diagram of image segmentation for traffic incident detection provided in an embodiment of the present application.

[0083] Figure 2 The target is a vehicle, area A is the segmented area, area B and area C are the parking lot area and the building area. Since there are vehicles in the parking lot and building areas, area B and area C are interference areas. The above steps can remove the interference areas and segment the target detection area (i.e. area A).

[0084] Optionally, the present application is applied to traffic incident detection on expressways, which include highways, expressways, and the like.

[0085] Since the traffic incident detection images are collected by an automatic cruising drone, and since the drone is in continuous motion, the traffic incident detection images include not only the expressway area but also vehicle interference in other areas. The expressway area (i.e., the target detection area) can be segmented through the above segmentation method, and only the expressway area is subsequently detected, which reduces the computational complexity of traffic incident detection and reduces interference in other areas.

[0086] Optionally, the target detection area is an area enclosed by the outermost road lines among multiple road lines in the traffic event detection image.

[0087] S105: Identify the target detection area to obtain target identification information.

[0088] Optionally, the target detection area of ​​the traffic event detection image may be input into a neural network model, and the target within the target detection area may be identified by the neural network model to obtain target identification information.

[0089] Optionally, the target recognition information includes target category, target location information and confidence information, wherein the target category may be multiple types of traffic participants, such as pedestrians, non-motor vehicles, motorcycles, cars, trucks, buses, etc.

[0090] S106: Perform traffic incident detection based on the target recognition information.

[0091] For example, if the position areas of a car and a truck overlap at the same time, it can be determined that a vehicle collision event occurs at the corresponding positions at that moment.

[0092] In this way, in order to achieve wide coverage and real-time detection of traffic areas, drones can be used. Drones have high flexibility and high-altitude viewing angles, can easily fly over obstacles, obtain comprehensive traffic conditions information, and improve the coverage and accuracy of traffic events. Due to the high-altitude viewing angle and flexibility of movement, there are interference areas in the images collected by drones, such as interference outside the road. This application identifies the points corresponding to the vehicle lines, and then fits the identified points to obtain lane lines, and segments the traffic event detection image based on the lane lines. It is only necessary to identify and detect traffic events in the segmented areas, which improves the efficiency of traffic event detection, and removes interference areas through segmentation, thereby improving the accuracy of traffic event detection.

[0093] It is easy to understand that the traffic event detection images collected by drones during cruising are prone to various interferences. For example, when a drone is close to power lines, it is easy to cause electromagnetic interference to the antenna, resulting in missed frames in the video detection image. Missing frames are caused by the network, camera sensors, and transcoding equipment, etc. The video image is not smooth, the picture is lost, and the picture received by the algorithm is empty; in hot summer weather or during the operation of the drone, the temperature of the drone body may be too high, for example, greater than 50°, affecting the image acquisition device of the drone, resulting in distortion of the traffic event detection image of the image acquisition device. In order to improve the accuracy of traffic event detection, the traffic event detection image can be preprocessed before performing traffic event detection based on the traffic event detection image.

[0094] In some embodiments, S102 includes:

[0095] If there are missing frames in the traffic event detection image, a first filtering process is performed on the traffic event detection image to obtain a first image to be processed, and the first image to be processed is input into a deep neural network model to obtain a point set;

[0096] If the fuselage temperature of the drone is greater than or equal to the preset temperature, a second filtering process is performed on the traffic event detection image to obtain a second processed image, and the second image to be processed is input into a deep neural network model to obtain a point set;

[0097] If there is no missing frame in the traffic event detection image and the body temperature of the drone is less than the preset temperature, the traffic event detection image is input into the deep neural network model to obtain a point set.

[0098] In this way, when the traffic incident detection images collected by the drone are affected differently, different filtering processing methods can be used to eliminate the impact on the image and improve the accuracy of traffic incident detection. Since only the traffic incident detection images that are affected are filtered to improve the image quality, for unaffected traffic incident detection images, such as when there are no missing frames or the drone temperature is less than the preset temperature, no filtering processing is required to improve the efficiency of image processing.

[0099] Optionally, after acquiring the traffic event detection image, it can be detected whether the traffic event detection image meets the frame missing condition. If the frame missing condition is met, the traffic event detection image has frame missing; if the frame missing condition is not met, the traffic event detection image does not have frame missing.

[0100] Optionally, the frame missing condition is that if the cumulative number of missed frames in the video frames in one minute is greater than 10 times, then there are missing frames in the traffic event detection image; if the cumulative number of missed frames in the video frames in one minute is less than or equal to 10 times, then there are no missing frames in the traffic event detection image.

[0101] Optionally, the VideoCapture method in the OpenCV library may be used to detect whether a video image is missing a frame.

[0102] Optionally, a temperature sensor is installed on the drone, and the temperature sensor is used to sense the body temperature of the drone and transmit the problem to the traffic event detection device through a communication link.

[0103] Optionally, the first filtering process is mean filtering, and mean filtering is used to solve the problem of detecting missing frames of video images.

[0104] Optionally, the second filtering process is a fast Gaussian filtering operation.

[0105] In the Gaussian filtering process, the two-dimensional Gaussian function is the basis for constructing the Gaussian filter, and its formula is as follows:

[0106]

[0107] Among them, x and y are the coordinates of the center point, and σ is the variance. According to the separation characteristics of the Gaussian function, the two-dimensional Gaussian function is derived into the product of the one-dimensional Gaussian function of the X-axis and the Y-axis, so as to realize the convolution of the entire image matrix in the X-axis direction first, and then convolution of the entire image matrix in the Y-axis direction to realize the rapid processing of the image by Gaussian filtering, as follows:

[0108]

[0109] Since the traffic accident detection image is a visible light image with three RGB channels, the image after Gaussian filtering is as follows:

[0110] G(x,y) R,G,B =∑ i∈(R,G,B) G(x,y) i ;

[0111] Among them, mean filtering replaces the pixel value of each point in the image with the mean of the pixel values ​​of all points, and adds the corresponding pixel values ​​of the three channels to obtain the processed image. The formula is as follows:

[0112]

[0113] Among them, f(x, y) is the original image, A(x, y) R,G，B is the image after mean filtering, s is the set of pixels in the template centered at point (x, y), and the template size is m×n.

[0114] In some embodiments, S103 includes:

[0115] Traverse the points of the point set in the traffic event detection image according to the preset rules to obtain the first point;

[0116] Determine a first neighborhood according to the first point, and determine the points of the point set in the first neighborhood;

[0117] Fitting the points of the point set in the first neighborhood to obtain the sub-lane lines in the first neighborhood, where the sub-lane lines are line segments;

[0118] Delete the points in the point set in the first neighborhood, and determine the second neighborhood according to the tail point of the sub-lane line corresponding to the first neighborhood, where the tail point is the endpoint of the sub-lane line corresponding to the first neighborhood away from the first point;

[0119] The second neighborhood is determined according to the tail point, and the points in the point set in the second neighborhood are determined, the points in the point set in the second neighborhood are fitted, the sub-lane lines in the second neighborhood are obtained, and the points in the point set in the second neighborhood are deleted, and the process is repeated in sequence to obtain multiple sub-lane lines, and the lane lines are determined according to the multiple sub-lane lines. In this way, after determining the point set in the traffic event detection image, the points in the traffic event detection image are traversed according to the preset rules, and after obtaining the first point (i.e., the first point), the first neighborhood is determined according to the first point, and then the points in the point set in the first neighborhood are traversed, and the sub-lane lines in the first neighborhood can be determined by fitting the points in the point set in the first neighborhood, and then all the points in the point set in the first neighborhood are deleted; then the tail point of the sub-lane line in the first neighborhood away from the first point is used as the first point of another neighborhood, and the process is repeated in sequence to obtain the sub-lane lines in multiple neighborhoods. That is, the points in the point set are divided into multiple neighborhoods, and multiple points in the point set in each neighborhood are fitted to obtain the sub-lane lines in multiple neighborhoods. After fitting the sub-lane line through the points in the point set in the neighborhood, delete the points in the point set in the neighborhood. When fitting the sub-lane line in the next neighborhood, there is no need to fit the points that have been used in the previous neighborhood, which improves the fitting efficiency of the sub-lane line. The storage of point information in the point set is reduced, and the data processing efficiency is improved. That is, in the process of fitting the sub-lane lines in multiple neighborhoods of the kernel, delete the points that have been used in the fitting process, and the next neighborhood fitting process does not need to use the point, which improves the fitting efficiency.

[0120] Optionally, if the tail point of one neighborhood is used as the first point of another neighborhood, and the area determined based on the first point does not contain other points in the point set except the tail point, the traversal is stopped, and all lane lines in the traffic event detection image are obtained.

[0121] Optionally, the specific rule may be a specific direction, such as traversing from the lower left corner to the upper right corner of the traffic event detection image.

[0122] Optionally, determining the first neighborhood according to the first point includes: drawing a circle with the first point as the center and a preset length as the radius, and the circle is the first neighborhood. The points in the point set in the first neighborhood are the points that meet the domain threshold of the first neighborhood.

[0123] Optionally, the preset length is determined based on the height of the drone and the angle between the drone and the first point. Since the image taken by the drone will cause image distortion, different heights and angles have different degrees of distortion. Optionally, determining the lane line according to the multiple sub-lane lines includes: determining the target sub-lane line according to the multiple sub-lane lines, at least one endpoint of each target sub-lane line coincides with an endpoint of another sub-lane line.

[0124] For example, see Figure 3 In (A), from the lower left corner to the upper right corner, traverse the points in the point set of the traffic incident detection image to obtain the first point (x1, y1); draw a circle with the first point as the center and the preset length as the radius. This circle is the first neighborhood. Traverse the points in the point set in the first neighborhood, fit the points in the point set in the first neighborhood through RANSAC, obtain the sub-lane line in the first neighborhood, delete the points in the point set in the first neighborhood, and use the endpoint of the sub-lane line in the first neighborhood away from the first point as the tail point, that is, (x2, y2). See Figure 3 In (B), the tail point (x2, y2) is used as the first point (x3, y3) of the second neighborhood. Based on the first point, the second area is determined, and the points in the point set in the second neighborhood are traversed. The points in the point set in the second neighborhood are fitted by RANSAC to obtain the sub-lane line in the second neighborhood. The points in the point set in the second neighborhood are deleted, and the end point of the sub-lane line in the second neighborhood farthest from the first point is used as the tail point, that is, (x4, y4). Repeat in sequence to obtain Figure 3 In (C), multiple sub-lane lines are fitted, and lane lines are formed based on multiple sub-lane lines. By deleting the points that have been used in each neighborhood, it is not necessary to calculate the points in the point set in the previous neighborhood when fitting the sub-lane line in the next neighborhood, thereby reducing the number of fitting points and improving the efficiency of sub-lane line fitting.

[0125] Optionally, in the process of traversing the points in the point set in the traffic event detection image, when the boundary of the traffic event detection image is traversed, the traversal is stopped.

[0126] Easy to understand, in actual scenarios, such as Figure 4 As shown in (A), the lane line is blocked by an obstacle (such as a garbage bag), which makes it impossible to identify the points of the lane line, resulting in the interruption of multiple sub-lane lines of the lane line.

[0127] When it comes to video or image data captured by drones, the phenomenon of “near objects are larger and far objects are smaller” is a common problem. This is because from the drone’s perspective, objects that are closer occupy more pixels in the image. Figure 4As shown in (B), multiple lane lines that are actually parallel intersect at a distance in the perspective of the drone. Due to the distortion of the lane lines, the slopes of the straight lines where the multiple lane lines are located change. Therefore, after determining the lane lines, if there are multiple lanes, since each lane line is formed by sub-lane lines corresponding to multiple neighborhoods, the slope of each lane line can be determined according to the first and last points of each lane line corresponding to each neighborhood. If the slopes of two lane lines are the same or similar, the two lane lines belong to the same target lane line. After S103, if the number of lane lines includes multiple lane lines, the method further includes: determining the slope of the lane line according to the first and last points of the neighborhood corresponding to the lane line; determining the target lane line according to the slopes of the multiple lane lines, wherein the slopes of the multiple lane lines of the same target lane line are the same or similar. For example, if the difference in the slopes of the two lane lines is less than or equal to the preset threshold, the slopes of the two lane lines are similar; if the difference in the slopes of the two lane lines is greater than the preset threshold, the slopes of the two lane lines are not similar.

[0128] like Figure 4 As shown in (C), there are three lane lines, each of which is formed by multiple sub-lane lines corresponding to the neighborhood. The straight line where the three lane lines are located can be determined based on the first and last points of the sub-lane lines corresponding to each neighborhood: y = kx + b, where k is the slope of each lane line. Lanes with similar or identical k values ​​belong to the same lane line, for example Figure 4 In (C), ② and ③ have the same k value, and both belong to the same lane line, while ① belongs to another lane line.

[0129] Since the curvature of the road line becomes less and less as it gets closer to the center of the road, the k value becomes closer. If the slopes of multiple lane lines are similar, the relevant slope can be determined based on the first and last points corresponding to any two lane lines among the multiple lane lines with the same or similar slopes. If the relevant slope is the same or similar to the slope of each lane, it is determined that the two lane lines belong to the same lane line.

[0130] For example, the coordinates of the first and last points of the multiple neighborhoods corresponding to the three lane lines are: [(x a ,y a )......(x b ,y b )]、[(x i ,y i )......(x j ,y j )]、[(x m ,y m )......(x n ,y n)], if the slopes of the three lanes ①, ②, and ③ are similar, that is, k1≈k2≈k3, k1, k2, and k3 are the slopes of the three lanes ①, ②, and ③ respectively, then calculate the relevant slopes of ①②, ②③, and ①③, among which the relevant slopes of ②③ are: The correlation slopes of ①② and ①③ are: That is, the relevant slopes of ②③ are similar to the slopes of ② and ③, that is, lane lines ② and ③ are the same lane line; the relevant slopes of ①② are different from the slopes of ② and ①; the relevant slopes of ①③ are different from the slopes of ① and ①, that is, lane line ① and lane line ② are not the same lane line; lane line ① and lane line ③ are not the same lane line.

[0131] In some embodiments, if there are multiple target lane lines, the method further includes:

[0132] Determine the relevant slope according to the first point and the last point of a plurality of neighborhoods corresponding to any two target lane lines among the plurality of target lane lines;

[0133] If the relevant slope is the same as or similar to the slopes of the two target lane lines, the two target lane lines are the same lane line;

[0134] If the relevant slope is different from or not similar to the slopes of the two target lane lines, the two target lane lines are different lane lines.

[0135] Optionally, if the difference between the relevant slope and the slope of the two target lane lines is less than or equal to a preset threshold, the relevant slope is close to the slope of the two target lane lines; if the difference between the relevant slope and the slope of the two target lane lines is greater than the preset threshold, the relevant slope is not close to the slope of the two target lane lines.

[0136] It is easy to understand that there are usually multiple vehicles running on the expressway at the same time. After identifying multiple targets in the target detection area to obtain target identification information, the multiple targets in the target identification area can be tracked by the ByteTrack method based on the target identification information to obtain the track information of each target. Then the above S106 includes:

[0137] According to the target recognition information, multiple targets in the target recognition area are tracked by ByteTrack to obtain the trajectory information of each target, where the trajectory information includes the target trajectory and target speed.

[0138] Traffic incident detection is performed based on target recognition information and trajectory information.

[0139] For example, the target's coordinate point set is recorded at a frequency of 10 frames within 30 seconds, and the points are sequentially connected into a track using lines; the target's moving speed on the image is recorded at intervals of 1 second.

[0140] According to the target identification information, ByteTrack is used to track multiple targets in the target identification area. The process of obtaining the trajectory information of each target includes the following:

[0141] 1. Detection frame classification: According to the detection score threshold, all detection frames are divided into high-score detection frames (D_high) and low-score detection frames (D_low). The high-score threshold is usually set to a higher value (such as 0.6), while the low-score threshold is lower (such as 0.1).

[0142] 2. Trajectory prediction: Use Kalman filtering to predict the new position of each trajectory in the current frame. Kalman filtering is based on a linear motion model and can predict the position of the target in the next frame.

[0143] 3. Data association:

[0144] First matching: Perform the first association between the high-scoring detection box (D_high) and all trajectories (including the lost trajectory T_lost). The similarity can be calculated by the IOU (intersection over union) between the detection box and the predicted trajectory box. Then, the Hungarian algorithm is used to complete the similarity-based matching.

[0145] Second matching: After the first association, a second association is performed on the low-score detection box (D_low) and the remaining unmatched track (T_remain). The association process is similar to the first one, but usually only IOU is used as the similarity metric, because the low-score detection box may contain severe occlusion or motion blur, and the appearance features are unreliable.

[0146] 4. Track management:

[0147] For unmatched trajectories, a certain life cycle (such as 30 frames) is retained. If no detection box is matched during this period, it is deleted.

[0148] For a high-scoring detection box that is not matched to a track, if its score is higher than a certain threshold (such as 0.7) and can be detected for multiple consecutive frames, a new track is assigned to it.

[0149] In the process of trajectory prediction through ByteTrack, the Kalman filter is used to predict the new position in the current frame of each trajectory. Specifically, the prediction box corresponding to the current position of the target is predicted by the Kalman filter. The target recognition information includes the detection box of the target. The KF state vector of the Kalman filter is:

[0150] x k =[x c (k), y c (k), w(k), h(k), x v (k), y v (k), w′(k), h′(k)] T ;

[0151] Among them, x, y are the coordinates of the detection box, w is the predicted width of the detection box, h is the height of the detection box, and x v is the velocity in the x direction, y v is the speed in the y direction, w′ is the area of ​​the detection frame, and h′ is the height change rate of the detection frame.

[0152] See also Figure 5 , Figure 5 A schematic diagram of the comparison effect of a detection frame provided in an embodiment of the present application, wherein Figure 5 Where A is the existing detection frame, and B is the detection frame determined by the Kalman filter based on this application. Figure 5 It can be seen that by improving the state vector of the Kalman filter, the detection box A is larger than the detection box B, and the trajectory of the target is determined based on the detection box. The position of the target can be accurately defined based on a more accurate detection box.

[0153] See also Figure 6 and Figure 7 , Figure 6 A schematic diagram of the state of the Nth frame provided in an embodiment of the present application, Figure 7 A schematic diagram of the N+50th frame is provided for an embodiment of the present application. Taking the vehicles numbered 87, 88 and 92 in the right lane as an example, by comparing the nth frame with the n+50th frame, the trajectory of each vehicle can be clearly determined, thereby achieving effective tracking of the target vehicle.

[0154] It is easy to understand that since the traffic incident detection images are acquired based on drone collection, the relative motion relationship between different targets (such as vehicles) in the traffic incident detection images can be determined based on the actual speed of the targets (such as vehicles) in the traffic incident detection images, so as to perform traffic incident detection more accurately.

[0155] In some embodiments, S106 includes:

[0156] Get the actual speed of the target:

[0157]

[0158] Among them, v car is the actual speed of the target, (x1, y1) and (x2, y2) are the center coordinates of the starting point and the end point of the target; v uavis the actual speed of the drone; dt is the time difference from the starting point to the end point of the target; h is the flight altitude of the drone, f is the focal length of the drone's camera; α is the angle between the bottom of the drone's camera lens and the vertical direction, α is greater than 0° and less than 90°, β is the field of view angle of the drone's camera lens; p x is the horizontal pixel size of the drone lens, p y is the vertical pixel size. C is the correction function;

[0159] Traffic incident detection is performed based on target recognition information and actual speed.

[0160] See also Figure 8 , Figure 8 is the side view taken by the drone, where β is the angle of view of the drone lens, and a is the angle between the angle of view under the drone lens and the vertical direction line. θ is the angle between the line connecting the center point of the vehicle to the drone lens and the angle of view under the drone lens, θ∈[0,β].

[0161] The same vehicle is in the same lane and moves from position (x1, y1) to position (x2, y2) within time dt.

[0162] For example, the size of the drone sensor is 1 inch (about 22.2mm x 16.6mm) and the resolution is 20MP (5472x3648 pixels), which can be derived from the following:

[0163] Horizontal pixel size = sensor width / horizontal pixel number;

[0164] Vertical pixel size = sensor height / number of vertical pixels;

[0165] The drone’s main camera:

[0166] p y =22.2mm / 5472pix≈0.00406mm / pix;

[0167] p x =16.6mm / 3648pix≈0.00455mm / pix;

[0168] In determining the vehicle direction, if the vehicle meets the following conditions:

[0169]

[0170] This means that the vehicle and the drone are moving in the same direction, otherwise the vehicle and the drone are moving in the same direction.

[0171] in,

[0172]

[0173]

[0174] Among them, y pix is the ordinate of the center point of the target vehicle in the image, Y maxpix is the image height (the image size is 1280*720), the value is 720.

[0175] Where g(y) is:

[0176]

[0177] The correction function C is:

[0178] C(y)=a*y 2 +b*y+c;

[0179] Where a=-0.0005, b=0.6541, c=43.7312. This function is used to eliminate the errors caused by the shaking of the camera angle (<3°) and the deviation of the route (<0.5m) during the flight of the drone.

[0180] For example, the drone is set with constant parameters: the drone's lens field of view angle (β) is 77.25°, the angle (α) between the drone's lens field of view angle and the vertical direction line is 11.50°, the drone's flight speed is 18 km / h, the flight altitude is 50 m, the lens focal length is 50 mm, and the horizontal pixel size p of the drone's lens is x is 0.00455mm / pix, and the vertical pixel size p y It is 0.00406mm / pix, and the time difference dt between the two displacement points is 10. Fig. 9 The engineering vehicle in the figure is a test vehicle, which is parked in the emergency lane, that is, the measured speed is 0, and the actual speed of the vehicle is measured based on the method of the above embodiment.

[0181] Table 2 Experimental results

[0182]

[0183] It can be seen from Table 2 that the speed measurement method can accurately measure the speed on the road, with an average error of 1.324 km / h, which is 0.0367 m / s, approximately 0 m / s, and a small error.

[0184] See also Fig.10, is a schematic diagram of the structure of a traffic incident detection device provided in an embodiment of the present application. As shown in the figure, the traffic incident detection device 100 provided in this embodiment may include: a processor 1040, a memory 1041, and a computer program 1042 stored in the memory 1041 and executable on the processor 1040, such as a program corresponding to the detection method. When the processor 1040 executes the computer program 1042, the steps applied in the detection method embodiment are implemented, such as Figure 1 Steps shown.

[0185] Exemplarily, the computer program 1042 may be divided into one or more modules / units, one or more modules / units are stored in the memory 1041, and are executed by the processor 1040 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 1042 in the traffic event detection device 100.

[0186] Those skilled in the art will understand that Fig.10 This is merely an example of the traffic event detection device 100 and does not constitute a limitation on the traffic event detection device 100 , which may include more or fewer components than those shown in the figure, or a combination of certain components, or different components.

[0187] The processor 1040 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0188] The memory 1041 may be an internal storage unit of the traffic event detection device 100, such as a hard disk or memory of the traffic event detection device 100. The memory 1041 may also be an external storage device of the traffic event detection device 100, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, or a flash card, etc., equipped on an electronic device. Further, the memory 1041 may also include both an internal storage unit of the traffic event detection device 100 and an external storage device.

[0189] The memory 1041 is used to store computer programs and other programs and data required by the electronic device. The memory 1041 can also be used to temporarily store data that has been output or is to be output.

[0190] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure of the power supply control device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0191] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0192] An embodiment of the present application provides a computer program product. When the computer program product is run on a detection device, the detection device implements the steps in the above-mentioned various method embodiments.

[0193] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0194] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0195] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0196] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A traffic incident detection method based on drone, characterized in that: The method comprises: Acquire a traffic event detection image, where the traffic event detection image is acquired by an image acquisition device of a drone; Inputting the traffic event detection image into a deep neural network model to obtain a point set, wherein the point set includes points of lane lines in the traffic event detection image, and the deep neural network model is used to identify points with lane line features in the traffic event detection image; Fitting the point set to obtain lane lines; Segmenting the traffic event detection image based on the lane lines to obtain a target detection area; Identifying the target detection area to obtain target identification information; Traffic incident detection is performed according to the target recognition information.

2. The method according to claim 1, characterized in that The step of fitting the point set to obtain the lane line includes: Traversing the points of the point set in the traffic event detection image according to a preset rule to obtain a first point; Determine a first neighborhood according to the first point, and determine points of the point set in the first neighborhood; Fitting points of the point set in the first neighborhood to obtain sub-lane lines in the first neighborhood; Deleting the points of the point set in the first neighborhood, and determining a second neighborhood according to the tail point of the sub-lane line corresponding to the first neighborhood, where the tail point is the endpoint of the sub-lane line corresponding to the first neighborhood away from the first point; Determine the second neighborhood according to the tail point, determine the points in the point set in the second neighborhood, fit the points in the point set in the second neighborhood, obtain the sub-lane lines in the second neighborhood, repeat in sequence to obtain multiple sub-lane lines, and determine the lane line according to the multiple sub-lane lines.

3. The method according to claim 2, characterized in that After fitting the point set to obtain lane lines, if there are multiple lane lines, the method further includes: Determine the slope of the lane line according to the first point and the last point of a plurality of neighborhoods corresponding to the lane; The target lane line is determined according to the slopes of the multiple lane lines, wherein the slopes of the multiple lane lines corresponding to the same target lane line are the same or similar.

4. The method according to claim 3, characterized in that If there are multiple target lane lines, the method further includes: Determine the relevant slope according to the first point and the last point of a plurality of neighborhoods corresponding to any two target lane lines among the plurality of target lane lines; If the relevant slope is the same as or similar to the slopes of the two target lane lines, the two target lane lines are the same lane line; If the relevant slope is different from or not similar to the slopes of the two target lane lines, the two target lane lines are different lane lines.

5. The method according to claim 1, characterized in that The step of inputting the traffic incident detection image into a deep neural network model to obtain a point set includes: If there are missing frames in the traffic event detection image, performing a first filtering process on the traffic event detection image to obtain a first image to be processed, and inputting the first image to be processed into a deep neural network model to obtain a point set; If the fuselage temperature of the drone is greater than or equal to a preset temperature, performing a second filtering process on the traffic event detection image to obtain a second image to be processed, and inputting the second image to be processed into a deep neural network model to obtain a point set; If there is no missing frame in the traffic event detection image and the fuselage temperature of the drone is lower than a preset temperature, the traffic event detection image is input into a deep neural network model to obtain a point set.

6. The method according to claim 1 or 2, characterized in that: The detecting of traffic incidents according to the target identification information comprises: Based on the target recognition information, tracking the targets in the target detection area by a target tracking method to obtain trajectory information of each target; Traffic incident detection is performed based on the target identification information and the trajectory information of each target.

7. The method according to claim 6, characterized in that The target tracking method is ByteTrack, and based on the target identification information, the target in the target detection area is tracked by the target tracking method to obtain the trajectory information of each target, including: Based on the target recognition information, the prediction box corresponding to the current position of the target is predicted by the Kalman filter, the target recognition information includes the detection box of the target, and the KF state vector of the Kalman filter is: x k＝ [x c (k),y c (k),w(k),h(k),x v (k),y v (k),w′(k),h′(k)] T ; Among them, x, y are the coordinates of the detection box, w is the predicted width of the detection box, h is the height of the detection box, and x v is the velocity in the x direction, y v is the speed in the y direction, w′ is the detection frame area, and h′ is the height change rate of the detection frame; Determine the similarity between the detection frame and the prediction frame; and match the detection frame based on the similarity to obtain trajectory information of the target.

8. The method according to claim 1, characterized in that The detecting of traffic incidents according to the target identification information comprises: Get the actual speed of the target: Among them, v car is the actual speed of the target, (x1, y1) and (x2, y2) are the center coordinates of the starting point and the end point of the target; v uav is the actual speed of the drone; dt is the time difference from the starting point to the end point of the target; h is the flight altitude of the drone, f is the focal length of the drone's camera; α is the angle between the bottom of the drone's camera lens and the vertical direction, α is greater than 0° and less than 90°, β is the field of view angle of the drone's camera lens; p x is the horizontal pixel size of the drone lens, p y is the vertical pixel size, C is the correction function, C(y)=a*y 2 +b*y+c, Traffic incident detection is performed according to the target recognition information and the actual speed.

9. A traffic incident detection device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the detection method according to any one of claims 1 to 8 are implemented.

Images