A vehicle instantaneous speed detection method and device based on drone aerial video

Abstract

The present invention discloses a vehicle instantaneous speed detection method and device based on drone aerial video, which belongs to the field of vehicle speed detection technology and is used to solve the technical problem that the vehicle speed detection accuracy achieved by drones in existing highways is poor and it is difficult to meet the needs of future intelligent operation and management of highways. The method includes: extracting the image of the white dashed lane line area of ​​the highway; performing peripheral rectangle analysis on the binary image under a connected domain set to obtain the connected domain attribute; performing fitting processing between the central pixel position in the connected domain set and the physical distance corresponding to the unit pixel under a nonlinear mechanism to obtain a nonlinear influence mechanism model; based on the interval frame of the rectangular frame of the target vehicle located by the aerial image, the actual physical moving distance output by the nonlinear influence mechanism model and the time difference between the interval frames are used for physical speed estimation to obtain the vehicle instantaneous speed of the target vehicle.

Description

Technical Field

[0001] The present application relates to the field of vehicle speed detection, and in particular to a method and device for detecting vehicle instantaneous speed based on drone aerial video. Background Art

[0002] Traditional methods of highway vehicle speed detection include: speed detection based on gantry radar and speed detection based on gantry camera. However, these methods are only applicable to highway sections where gantries have been deployed, and cannot be used to detect speed on a specific section. In addition, the height limit of the gantry deployment results in a small detection range of the gantry sensor. It is difficult to meet the needs of future highway intelligent operation and management.

[0003] Speed ​​detection by drones along highways is a combination of drone technology, computer vision technology and highway traffic operation management. However, the use of drone inspection videos for vehicle speed detection also faces many problems. The most critical difficulty is that aerial images will produce geometric distortion during the imaging process, which is typically manifested as distortion of objects in the image, inharmonious proportions of near and far, etc. The traditional method is to perform distortion correction on drone aerial images, which involves solving the camera parameters. However, since drones are easily affected by uncontrollable factors such as weather and the stability of the drone flight control system during flight, accurate parameter solution is very difficult. Summary of the invention

[0004] The embodiments of the present application provide a method and device for detecting the instantaneous speed of a vehicle based on drone aerial video, which are used to solve the following technical problems: In existing highways, the accuracy of vehicle speed detection using drones is poor, which is difficult to meet the needs of future intelligent operation and management of highways.

[0005] The present application embodiment adopts the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a method for detecting the instantaneous speed of a vehicle based on aerial video of a drone, comprising: extracting an image of an area of ​​a white dashed lane line of a highway based on an area of ​​a white lane line of a highway and an area of ​​a highway; binarizing the image of the area of ​​the white dashed lane line to obtain a binary image of the white lane line; performing an outer rectangle analysis on the binary image under a connected domain set to obtain connected domain attributes; wherein the connected domain attributes include: the area of ​​the outer rectangle, the height of the outer rectangle, the coordinates of the center pixel of the outer rectangle, the diagonal length of the outer rectangle, and the physical distance corresponding to the unit pixel; performing a two-dimensional coordinate transformation on the connected domain set according to the connected domain attributes, and performing a fitting process under a nonlinear mechanism between the center pixel position in the connected domain set and the physical distance corresponding to the unit pixel based on the two-dimensional coordinate point to obtain a nonlinear influence mechanism model; based on the interval frame of the rectangular frame of the target vehicle located by the aerial image, performing a physical speed estimation on the actual physical moving distance output by the nonlinear influence mechanism model and the time difference between the interval frames to obtain the vehicle instantaneous speed of the target vehicle.

[0007] The embodiment of the present application extracts the white dashed lane line area image based on the specific area of ​​the highway, which provides an accurate analysis object for subsequent speed detection and reduces the interference of irrelevant information. By analyzing the peripheral rectangle under the connected domain set of the binary image, the connected domain attributes including the peripheral rectangle area, height, center pixel coordinates, diagonal length and physical distance corresponding to the unit pixel are obtained, which provides rich information for the subsequent speed calculation. And by using the nonlinear influence mechanism model, it can more accurately reflect the actual situation and improve the accuracy of speed detection. It also realizes the effective detection of the instantaneous speed of the vehicle, providing valuable data for traffic management and research. At the same time, the relationship modeling between the "pixel distance" and the actual "physical distance" in the drone field of view relies on the accurate extraction of the white dashed lane line, which effectively overcomes the interference of illumination, vehicles, highway ground signs and non-highway areas (such as urban roads) in the drone field of view, and greatly improves the accuracy of the instantaneous speed detection of high-speed vehicles.

[0008] In a feasible implementation, based on the lane line area of ​​the highway and the highway area, extracting the image of the white dashed lane line area of ​​the highway, specifically includes: obtaining an aerial image through a drone along the highway; wherein the aerial image includes: aerial image height, aerial image width and aerial image depth; through a preset semantic segmentation technology, performing pixel area detection on the lane line area where the white dashed line is located in the aerial image to obtain a first mask image; performing pixel area detection on the highway area in the aerial image to obtain a second mask image; wherein the first mask image and the second mask image both include a foreground area and a background area; according to Obtain the white dotted lane line area image mask after the first mask image and the second mask image intersect f_w_l ; Among them, mask f_r is the second mask image; mask w_l is the first mask image; h∈{0, 1, ..., H-2, H-1}, and h is the height of the aerial image; w∈{0, 1, ..., W-2, W-1}, and w is the width of the aerial image; 255 and 0 are both grayscale values ​​of the image.

[0009] In a feasible implementation, the white dashed lane line area image is binarized to obtain a binarized image of the white lane line, specifically including: grayscale processing the aerial image collected by the drone; based on the grayscale value of the background area in the white dashed lane line area image, the grayscale values ​​of the background area and the foreground area in the aerial image after grayscale processing are set accordingly to obtain the aerial image to be processed; according to Get the binary image bw based on the white lane line fw _l; among them, gray f _w_l is the aerial image to be processed; 0 <threshold bw is the binarization threshold, and 0 <threshold bw <255; h∈{0, 1, ..., H-2, H-1}, where h is the height of the aerial image; w∈{0, 1, ..., W-2, W-1}, where w is the width of the aerial image.

[0010] In a feasible implementation, the binary image is subjected to an outer rectangle analysis under a connected domain set to obtain connected domain attributes, specifically including: determining a connected area based on a maximum foreground pixel block composed of adjacent foreground pixels in the binary image, and obtaining the connected domain set by a connected domain analysis method; performing an outer rectangle selection process on each connected domain in the connected domain set to determine the diagonal length of the outer rectangle, the area of ​​the outer rectangle, and the height of the outer rectangle; based on the central pixel of the outer rectangle in each connected domain, calculating the horizontal and vertical distance coordinates of the binary image to obtain the coordinates of the central pixel of the outer rectangle; and according to dis_unit_pix m =len dashed / diagonal_len m , get the physical distance dis_unit_pix corresponding to each unit pixel in the connected domain m ; Among them, diagonal_len m is the diagonal length of the outer rectangle; len dashed It is the length of the white dashed line of the highway lane divider.

[0011] In a feasible implementation, before performing outer rectangle analysis on the binary image under a connected domain set to obtain connected domain attributes, the method further includes: performing area constraints and height constraints on noise factors in the highway to determine an area threshold and a height threshold; wherein the noise factors include at least: high-speed vehicles and other road signs; and performing noise removal processing on the connected domain set using the area threshold and the height threshold to obtain an optimized connected domain set.

[0012] In a feasible implementation, according to the connected domain attribute, the connected domain set is transformed into a two-dimensional coordinate, and based on the two-dimensional coordinate point, the central pixel position in the connected domain set and the physical distance corresponding to the unit pixel are fitted under a nonlinear mechanism to obtain a nonlinear influence mechanism model, which specifically includes: extracting the physical distance corresponding to the unit pixel of each connected domain in the connected domain attribute and the vertical distance of the central pixel coordinate of the outer rectangle; determining the physical distance corresponding to the unit pixel as the ordinate of the two-dimensional coordinate system, and determining the vertical distance as the abscissa of the two-dimensional coordinate system; obtaining the two-dimensional coordinate point of each connected domain based on the abscissa and the ordinate; and generating a two-dimensional coordinate point set based on optimizing all the two-dimensional coordinate points in the connected domain set; and fitting the two-dimensional coordinate point set to the two-dimensional coordinate point set. Noise removal is performed to obtain an optimized two-dimensional coordinate point set; all two-dimensional horizontal coordinate sets in the optimized two-dimensional coordinate point set are counted; each horizontal coordinate in the two-dimensional horizontal coordinate set is sorted in terms of numerical value, and all sorted two-dimensional horizontal coordinates are marked with the first and last coordinates of the 5th and 95th percentiles to obtain a 5% segmentation point and a 95% segmentation point; based on the 5% segmentation point and the 95% segmentation point, the corresponding coordinates in the optimized two-dimensional coordinate point set are removed from the first and last coordinates to obtain an accurate two-dimensional coordinate point set; a polynomial fitting process is performed between the central pixel position of the relevant two-dimensional coordinates in the accurate two-dimensional coordinate point set and the corresponding physical distance of the unit pixel through a nonlinear model and the model parameters in the nonlinear model to obtain the nonlinear influence mechanism model.

[0013] In a feasible implementation, the two-dimensional coordinate point set is subjected to noise removal processing to obtain an optimized two-dimensional coordinate point set, specifically including: calculating the average distance between any two-dimensional coordinate point in the two-dimensional coordinate point set and the adjacent two-dimensional coordinate point, and establishing an average distance set; sorting each average distance in the average distance set by numerical value; and marking the 85th percentile based on all the sorted average distances to obtain 85% segmentation points; and performing noise removal processing on the average distances in the sorted average distance set that are greater than 85% of the segmentation points to obtain the optimized two-dimensional coordinate point set based on the remaining two-dimensional coordinate points.

[0014] In a feasible implementation manner, based on the interval frames of the rectangular frame of the target vehicle located by the aerial image, the actual physical moving distance output by the nonlinear influence mechanism model and the time difference of the interval frames are used to perform physical speed estimation to obtain the instantaneous vehicle speed of the target vehicle, specifically including: based on the deep learning-based vehicle detection and tracking algorithm, aerial images of the target vehicle are acquired under interval frames to obtain a first aerial image frame and a second aerial image frame; the first center pixel position of the target positioning rectangular frame in the first aerial image frame is acquired, and the second center pixel position of the target positioning rectangular frame in the second aerial image frame is acquired; based on the first aerial image frame and the first aerial image frame, the time difference of the interval frames is determined; through the nonlinear influence mechanism model, the physical distance calculation based on pixel displacement is performed on the first center pixel position and the second center pixel position to obtain the actual physical moving distance; the physical speed estimation is performed on the actual physical moving distance and the time difference to obtain the instantaneous vehicle speed of the target vehicle.

[0015] In a possible implementation, according to Get the time difference time_dif of the interval frame; frame_index_β is the first aerial image frame; frame_index_α is the first aerial image frame; F is the frame rate of the drone aerial video; according to The actual physical moving distance phy_dis is obtained; f(x) is the nonlinear influence mechanism model; center _ yβ β is the ordinate of the second center pixel position; center_y α is the ordinate of the first central pixel position; Get the instantaneous speed of the vehicle phy_v; where v UAV is the flight speed of the drone.

[0016] On the other hand, an embodiment of the present application also provides a vehicle instantaneous speed detection device based on drone aerial video, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute a vehicle instantaneous speed detection method based on drone aerial video as described in any of the above embodiments.

[0017] The present application provides a method and device for detecting the instantaneous speed of a vehicle based on drone aerial video. Compared with the prior art, the embodiments of the present application have the following beneficial technical effects:

[0018] 1. Targeted image extraction: Extracting white dashed lane line area images based on specific areas of the highway provides accurate analysis objects for subsequent speed detection and reduces interference from irrelevant information.

[0019] 2. Binarization: Binarize the image of the white dashed lane area to make the image simpler, which is conducive to subsequent analysis and processing and improves computational efficiency.

[0020] 3. Connected domain analysis: By analyzing the outer rectangle of the binary image under the connected domain set, the connected domain attributes including the outer rectangle area, height, center pixel coordinates, diagonal length and physical distance corresponding to the unit pixel are obtained, providing rich information for subsequent speed calculation.

[0021] 4. Nonlinear fitting: The two-dimensional coordinates are transformed according to the connected domain attributes, and the central pixel position in the connected domain set and the physical distance corresponding to the unit pixel are fitted under the nonlinear mechanism based on the two-dimensional coordinate points to obtain a nonlinear influence mechanism model, which can more accurately reflect the actual situation and improve the accuracy of speed detection.

[0022] 5. Instantaneous speed detection: Based on the interval frames of the rectangular frame of the target vehicle located by the aerial image, the actual physical moving distance output by the nonlinear influence mechanism model and the time difference of the interval frames are used to estimate the physical speed, and the instantaneous speed of the target vehicle is obtained, thereby realizing the effective detection of the instantaneous speed of the vehicle and providing valuable data for traffic management and research.

[0023] 6. Drone acquisition: The relationship modeling between the "pixel distance" and the actual "physical distance" in the drone's field of view relies on accurate extraction of white dashed lane lines. In order to effectively overcome the interference of lighting, vehicles, highway ground signs, and non-highway areas (such as urban roads) in the drone's field of view, it integrates multiple image processing technologies such as semantic segmentation and morphology, greatly improving the accuracy of high-speed vehicle instantaneous speed detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 A flow chart of a method for detecting the instantaneous speed of a vehicle based on drone aerial video provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of a vehicle instantaneous speed detection device based on drone aerial video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0028] It should be noted that in recent years, drones have been widely used in the field of highway operation and management. Using drones to conduct inspections along specific highway sections has become an important means of monitoring and managing the operation status of highways. In the process of inspections along highways, vehicle speed detection is an important task. On the one hand, it can analyze the basic traffic status of highway sections, and on the other hand, it can detect speeding vehicles, and carry out risk analysis and early warning in turn. This application abandons the traditional speed detection method based on camera parameters and image geometric correction, but fully combines the visual appearance characteristics of highways with computer vision technology to achieve speed assessment. Specifically: the physical length of the white dashed lane line on the highway is known (such as the length of the white dashed line is 6 meters), and the white dashed lane line in the collected picture is accurately extracted, so as to establish the relationship between the "pixel length" of the white dashed lane line in the picture and the "physical length" of the actual color dashed lane line. Taking into account the tilted shooting of drones and the geometric distortion of the lens, a nonlinear model is used to construct the above relationship.

[0029] In addition, the actual physical length corresponding to a pixel at different positions in the drone aerial image is different. Specifically, the closer to the bottom of the aerial image, the closer the presented area is to the drone, resulting in a smaller physical length corresponding to a pixel. Conversely, the closer to the top of the aerial image, the farther away from the drone, resulting in a larger physical length corresponding to an image pixel.

[0030] Therefore, in drone aerial images, the "unit pixel corresponding to physical distance" is not fixed, but is affected by the position of the pixel itself in the image. Therefore, it is necessary to explore the influence mechanism of "pixel position" on "unit pixel corresponding to physical distance".

[0031] Among them, in actual activities, the drone flies along the highway, the flight altitude remains unchanged, and the flight speed v UAV The drone aerial image is image∈RH×W×D . Where H is the height of the aerial image, W is the width of the aerial image, D is the depth of the aerial image, and the frame rate of the drone aerial video is F. The length of the white dashed line of the highway lane is len dashed , the distance between the lines is dis dashed .

[0032] Due to the geometric distortion characteristics of aerial image imaging, the influence mechanism of "pixel position" on "unit pixel corresponding physical distance" is not a simple linear model. Therefore, this application needs to use a deep network to characterize this nonlinear relationship.

[0033] The present application embodiment provides a method for detecting the instantaneous speed of a vehicle based on drone aerial video. Figure 1 As shown, the vehicle instantaneous speed detection method based on drone aerial video specifically includes steps S101-S105:

[0034] S101, extracting an image of a white dashed lane line area of ​​the expressway based on a white lane line area of ​​the expressway and an expressway area.

[0035] Specifically, it is necessary to first obtain aerial images through drones along the highway, wherein the aerial images include: aerial image height, aerial image width, and aerial image depth.

[0036] Furthermore, by using a preset semantic segmentation technology, pixel area detection is performed on the lane line area where the white dotted line in the aerial image is located to obtain a first mask image.

[0037] Furthermore, pixel region detection is performed on the highway region in the aerial image to obtain a second mask image, wherein both the first mask image and the second mask image include a foreground region and a background region.

[0038] In one embodiment, a semantic segmentation method (such as LaneNet) is used to detect the lane line where the white dashed line is located in the aerial image image, and a first mask image is obtained as mask white_lanes ∈R H×W , which is a grayscale image with a depth of 1 and is abbreviated as mask w _l. The foreground area is the lane line area ROI where the white dotted line is located w_l , the grayscale value of the corresponding pixel is 255; the grayscale value of the background area is 0.

[0039] In one embodiment, in order to reduce the interference of non-highway areas (such as urban roads around highways) in the aerial image on the detection of white dashed lanes, this patent uses a semantic segmentation method (such as the DeepLabV3 algorithm) to detect the highway area of ​​the image, and obtains a second mask image as mask freeway_region ∈R H×W , which is a grayscale image with a depth of 1 and is abbreviated as mask f_r The foreground area is the highway area ROI f_r The grayscale value of the corresponding pixel is 255, and the grayscale value of the background area is 0.

[0040] Furthermore, for the second mask image mask f_r And the first mask image mask w_l Perform intersection operation to obtain mask image mask freeway_white_lanes ∈R H×W , that is, the image of the white dashed lane line area, which is a grayscale image and is abbreviated as mask f_w_l The specific operation logic of the intersection operation is:

[0041] according to

[0042] Get the white dashed lane line area image mask after the first mask image and the second mask image intersect f_w_l Among them, mask f_r is the second mask image; mask w_l is the first mask image; h∈{0, 1, ..., H-2, H-1}, and h is the height of the aerial image; w∈{0, 1, ..., W-2, W-1}, and w is the width of the aerial image; 255 and 0 are both grayscale values ​​of the image.

[0043] Among them, mask f_w_l The foreground area (i.e., pixels with a grayscale value of 255) corresponds to the lane line area where the white dotted line of the highway is located. The area outside the foreground area is the background area, and the grayscale value of the pixels is 0.

[0044] S102, binarizing the white dashed lane line area image to obtain a binarized image of the white lane line.

[0045] Specifically, first, the aerial image captured by the drone is gray-processed. Then, based on the gray-scale value of the background area in the white dashed lane line area image, the gray-scale values ​​of the background area and the foreground area in the aerial image after gray-scale processing are set accordingly to obtain the aerial image to be processed.

[0046] In one embodiment, the aerial image image needs to be converted into a grayscale image gray∈R H×W Then based on the mask f_w_l Process the grayscale image gray to get gray f_w_l , that is, the aerial image after grayscale processing, so as to obtain the aerial image gray to be processed f _w_l:

[0047] That is, the grayscale image gray corresponds to the mask f_w_l The grayscale value of the pixel in the background area is set to 0, and the grayscale value of the pixel in the foreground area remains unchanged.

[0048] Further, according to Get the binary image bw based on the white lane line f_w_l Among them, gray f_w_l is the aerial image to be processed; 0 <threshold bw is the binarization threshold, and 0 <threshold bw <255; h∈{0, 1, ..., H-2, H-1}, where h is the height of the aerial image; w∈{0, 1, ..., W-2, W-1}, where w is the width of the aerial image.

[0049] S103. Perform peripheral rectangle analysis on the binary image under the connected domain set to obtain connected domain attributes; wherein the connected domain attributes include: peripheral rectangle area, peripheral rectangle height, peripheral rectangle center pixel coordinates, peripheral rectangle diagonal length, and physical distance corresponding to unit pixel.

[0050] Specifically, based on the largest foreground pixel block composed of adjacent foreground pixels in the binary image, a connected region is determined, and a connected region set is obtained through a connected region analysis method.

[0051] A peripheral rectangle is selected for each connected domain in the connected domain set to determine the diagonal length of the peripheral rectangle, the area of ​​the peripheral rectangle, and the height of the peripheral rectangle.

[0052] Based on the central pixel of the outer rectangle in each connected domain, the horizontal and vertical distance coordinates of the binary image are calculated to obtain the coordinates of the central pixel of the outer rectangle.

[0053] In one embodiment, based on the binary image bw of the white lane line f_w_l Connected region analysis is performed. A connected region is the largest foreground pixel block composed of adjacent foreground pixels in the binary image. Through the connected region analysis method (such as the two-pass scanning method), a connected domain set consisting of M connected regions is finally obtained.

[0054] And, for each connected area, c_component m , computed properties

[0055] area_c_component m 、high_c_component m ,

[0056] center m ={center_x m , center_y m}, and diagonal_len m .in

[0057] area_c_component m Represents the outer rectangular area of ​​the mth connected region, high_c_component m Indicates the height of the outer rectangle of the mth connected region. center m ={img_x m ,img_y m} is the coordinate of the center pixel of the outer rectangle of the mth connected region, that is, the center pixel of the outer rectangle in the image coordinate, img_x m Indicates the horizontal distance of the center pixel relative to the upper left corner of the image, img_y m Indicates the vertical distance of the center pixel relative to the upper left corner of the image. m Represents the diagonal length of the outer rectangle of the mth connected region.

[0058] Furthermore, according to dis_unit_pix m =len dashed / diagonal_len m , get the physical distance dis_unit_pix corresponding to the unit pixel in each connected domain m Among them, diagonal_len m is the diagonal length of the outer rectangle; len dashed It is the length of the white dashed line of the highway lane divider.

[0059] As a feasible implementation method, the noise factors in the highway can also be constrained by area and height, and the area threshold and height threshold can be determined. The noise factors at least include: high-speed vehicles and other road signs. Then, the connected domain set is subjected to noise elimination processing by the area threshold and the height threshold to obtain the optimized connected domain set.

[0060] In one embodiment, due to the influence of vehicles (especially white vehicles) on the highway and other road signs (such as ground traffic guidance arrows, etc.), the above connected domain analysis may contain noise, so it is necessary to remove the noise of the connected region. First, it is necessary to remove the noise of the connected region set c_component_set based on the area constraint and height constraint. The removal rule is: if area_c_component m <thd area or high_c_component m <thd high ; Among them, thd area Indicates the area threshold, thd high Represents the height threshold. Finally, the above elimination rules are used to remove noise from the connected domain set c_component_set, and the remaining N connected regions form the optimized connected domain set. Where N≤M.

[0061] S104, performing two-dimensional coordinate transformation on the connected domain set according to the connected domain attribute, and performing fitting processing under a nonlinear mechanism between the central pixel position in the connected domain set and the physical distance corresponding to the unit pixel based on the two-dimensional coordinate point, to obtain a nonlinear influence mechanism model.

[0062] Specifically, the physical distance corresponding to the unit pixel of each connected domain in the connected domain attribute and the vertical distance of the central pixel coordinates of the peripheral rectangle are extracted, the physical distance corresponding to the unit pixel is determined as the ordinate of the two-dimensional coordinate system, and the vertical distance is determined as the abscissa of the two-dimensional coordinate system.

[0063] Furthermore, based on the abscissa and the ordinate, a two-dimensional coordinate point of each connected domain is obtained; and based on optimizing all the two-dimensional coordinate points in the connected domain set, a two-dimensional coordinate point set is generated.

[0064] In one embodiment, for each connected region c_component in the optimized connected region set c_component_set′ n , the vertical distance img_y of the center pixel coordinates of the outer rectangle n The physical distance corresponding to the unit pixel dis_unit_pix m As the horizontal and vertical coordinates of the two-dimensional coordinate system respectively. In this way, each connected region c_component n Will correspond to a point point in the two-dimensional coordinates n =(img_y n , dis_unit_pix m ). For the convenience of representation, the two-dimensional coordinate pointn Denoted as (x n ,y n ). In the two-dimensional coordinate point set point n In the example, the horizontal axis reflects the horizontal position of the connected region in the image. The vertical axis represents the physical distance corresponding to the unit pixel in the mth connected region. For all connected regions in c_component_set′, N two-dimensional coordinate points can be formed, that is, the two-dimensional coordinate point set

[0065] Furthermore, the two-dimensional coordinate point set is subjected to noise removal processing to obtain an optimized two-dimensional coordinate point set.

[0066] As a feasible implementation method, the average distance between any two-dimensional coordinate point and the adjacent two-dimensional coordinate point in the two-dimensional coordinate point set can be calculated first, and an average distance set can be established. Each average distance in the average distance set is sorted by numerical value. And the 85th percentile is marked based on all the sorted average distances to obtain the 85% segmentation point. Finally, the average distance greater than 85% of the segmentation points in the sorted average distance set is subjected to noise elimination to obtain an optimized two-dimensional coordinate point set based on the remaining two-dimensional coordinate points.

[0067] In one embodiment, due to reasons such as illumination, unclear white dashed lines on highways, or overlap of new and old dashed lines, noise errors appear in the vertical coordinates of some two-dimensional coordinate points. Therefore, it is necessary to remove noise from the two-dimensional coordinate point set point_set. The method is as follows:

[0068] (a) In point_set, for any two-dimensional coordinate point point n , calculate the average distance between the point and its K adjacent points, recorded as K_avg_dis n , and finally get

[0069] (b) Sort K_avg_dis_set from small to large and get the 85th percentile, recorded as K_avg_dis 85％ .

[0070] (c) Set point_set to satisfy K_avg_dis n >K_avg_dis 85 % of the two-dimensional coordinate points are removed. The remaining P two-dimensional coordinate points form the optimized two-dimensional coordinate point set Where P≤N.

[0071] Furthermore, all the two-dimensional horizontal coordinate sets in the two-dimensional coordinate point set are statistically optimized. Each horizontal coordinate in the two-dimensional horizontal coordinate set is sorted in terms of numerical value, and all the sorted two-dimensional horizontal coordinates are marked with the first and last coordinates of the 5th and 95th percentiles to obtain the 5% and 95% split points. Based on the 5% and 95% split points, the corresponding coordinates in the optimized two-dimensional coordinate point set are eliminated from the first and last coordinates to obtain an accurate two-dimensional coordinate point set.

[0072] In one embodiment, since the top and bottom of the aerial image are prone to incomplete display of white dashed lane lines, the connected areas at the top and bottom of the aerial image will be smaller than the nearby connected areas. Since the connected area at the top of the aerial image corresponds to the part of points with the smallest horizontal coordinates in the two-dimensional coordinate point set, and the connected area at the bottom of the aerial image corresponds to the part of points with the largest horizontal coordinates in the two-dimensional coordinate point set, it will eventually lead to position deviations between the two-dimensional coordinate points with the smallest and largest horizontal coordinates in the two-dimensional coordinate space. Therefore, the present application removes the two-dimensional coordinate points with the smallest and largest horizontal coordinates. Specifically:

[0073] (a) Statistically optimize the horizontal coordinates of all two-dimensional coordinate points in the two-dimensional coordinate point set point_set′ to obtain a two-dimensional horizontal coordinate set

[0074] (b)x n _set is sorted from small to large, and the 5th and 95th percentiles are obtained, recorded as x5% and X respectively. 95 %.

[0075] (c) Set point_set′ to satisfy x p ≤x 5％ or x p ≥x 95％ The two-dimensional coordinate points are eliminated.

[0076] (d) The remaining Q two-dimensional coordinate points form a precise two-dimensional coordinate point set Where Q≤P.

[0077] Furthermore, through the nonlinear model and the model parameters in the nonlinear model, a polynomial fitting process is performed between the central pixel position of the relevant two-dimensional coordinates in the precise two-dimensional coordinate point set and the physical distance corresponding to the unit pixel to obtain a nonlinear influence mechanism model.

[0078] In one embodiment, a nonlinear model (such as a polynomial or a convolutional neural network) is used to fit the nonlinear influence mechanism of "pixel position" on "unit pixel corresponding physical distance". Taking a cubic polynomial as an example, the constructed nonlinear influence mechanism model is recorded as: y = f(x) = ax 3 +bx 2 +cx+d; where a, b, c and d are the parameters of the model. The least squares method can also be used to approximate the parameters.

[0079] S105 , based on the interval frames of the rectangular frame of the target vehicle located by the aerial image, the actual physical moving distance output by the nonlinear influence mechanism model and the time difference of the interval frames are used to estimate the physical speed, so as to obtain the instantaneous speed of the target vehicle.

[0080] Specifically, based on the deep learning vehicle detection and tracking algorithm, aerial images of the target vehicle are acquired in interval frames to obtain a first aerial image frame and a second aerial image frame. The first center pixel position of the target positioning rectangular frame in the first aerial image frame is obtained, and the second center pixel position of the target positioning rectangular frame in the second aerial image frame is obtained.

[0081] Further, based on the first aerial image frame and the first aerial image frame, a time difference between the interval frames is determined.

[0082] Furthermore, through the nonlinear influence mechanism model, the physical distance between the first central pixel position and the second central pixel position is calculated based on the pixel displacement to obtain the actual physical movement distance.

[0083] In one embodiment, a vehicle detection and tracking algorithm based on deep learning (such as YOLO+DeepSort) is used to locate and track the vehicle target. Assume that in the first aerial image frame frame_index _ α frame, the center of the target vehicle positioning rectangle (the first center pixel position) is center α ={center_x α , center_y α}, in the second aerial image frame frame_index_β, the center of the target vehicle positioning rectangle (the second center pixel position) is center β ={center_x β , center_y β The pixel displacement of the target vehicle between frame_index_α and frame_index_β is the displacement from center_y α to center_y β Among them, center α / βis the pixel coordinate of the center of the outer rectangle of the connected area.

[0084] Furthermore, the physical speed is estimated based on the actual physical moving distance and the time difference to obtain the instantaneous speed of the target vehicle.

[0085] As a feasible implementation, for the time difference time_dif between the frame_index_α frame and the frame_index_β frame: Get the time difference between frames _ dif. Among them, frame_index_β is the first aerial image frame; frame_index _ α is the first aerial image frame; F is the frame rate of the drone aerial video. α to center_y β The actual physical moving distance is: The actual physical moving distance phy_dis is obtained. f(x) is the nonlinear influence mechanism model; center_y β is the ordinate of the second center pixel position; center_y α is the ordinate of the first central pixel position. Finally, according to Get the instantaneous speed of the vehicle phy_v; where v UAV is the flight speed of the drone.

[0086] In addition, the present application also provides a vehicle instantaneous speed detection device based on drone aerial video, such as Figure 2 As shown, the vehicle instantaneous speed detection device based on drone aerial video specifically includes:

[0087] At least one processor 201. And a memory 202 in communication with the at least one processor 201; wherein the memory 202 stores instructions executable by the at least one processor 201, so that the at least one processor 201 can execute:

[0088] Based on the white lane line area and the highway area, extract the white dashed lane line area image of the highway;

[0089] Binarize the image of the white dashed lane line area to obtain a binary image of the white lane line;

[0090] Performing peripheral rectangle analysis on the binary image under the connected domain set to obtain the connected domain attributes; wherein the connected domain attributes include: peripheral rectangle area, peripheral rectangle height, peripheral rectangle center pixel coordinates, peripheral rectangle diagonal length, and physical distance corresponding to unit pixel;

[0091] According to the connected domain attributes, the connected domain set is transformed into a two-dimensional coordinate, and based on the two-dimensional coordinate points, the central pixel position in the connected domain set and the physical distance corresponding to the unit pixel are fitted under a nonlinear mechanism to obtain a nonlinear influence mechanism model;

[0092] Based on the interval frames of the rectangular frame of the target vehicle located by the aerial image, the actual physical moving distance output by the nonlinear influence mechanism model and the time difference between the interval frames are used to estimate the physical speed, and the instantaneous speed of the target vehicle is obtained.

[0093] The embodiment of the present application extracts the white dashed lane line area image based on the specific area of ​​the highway, which provides an accurate analysis object for subsequent speed detection and reduces the interference of irrelevant information. By analyzing the peripheral rectangle under the connected domain set of the binary image, the connected domain attributes including the peripheral rectangle area, height, center pixel coordinates, diagonal length and physical distance corresponding to the unit pixel are obtained, which provides rich information for the subsequent speed calculation. And by using the nonlinear influence mechanism model, it can more accurately reflect the actual situation and improve the accuracy of speed detection. It also realizes the effective detection of the instantaneous speed of the vehicle, providing valuable data for traffic management and research. At the same time, the relationship modeling between the "pixel distance" and the actual "physical distance" in the drone field of view relies on the accurate extraction of the white dashed lane line, which effectively overcomes the interference of illumination, vehicles, highway ground signs and non-highway areas (such as urban roads) in the drone field of view, and greatly improves the accuracy of the instantaneous speed detection of high-speed vehicles.

[0094] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0095] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0096] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application should be included in the scope of the claims of the present application.

Claims

1. A vehicle instantaneous speed detection method based on drone aerial video, characterized in that: The method comprises: Extracting a white dashed lane line area image of the highway based on the white lane line area of ​​the highway and the highway area; Binarizing the white dashed lane line area image to obtain a binary image of the white lane line; Performing peripheral rectangle analysis on the binary image under a connected domain set to obtain connected domain attributes; wherein the connected domain attributes include: peripheral rectangle area, peripheral rectangle height, peripheral rectangle center pixel coordinates, peripheral rectangle diagonal length, and physical distance corresponding to a unit pixel; According to the connected domain attribute, the connected domain set is transformed into a two-dimensional coordinate, and based on the two-dimensional coordinate point, the central pixel position in the connected domain set and the physical distance corresponding to the unit pixel are fitted under a nonlinear mechanism to obtain a nonlinear influence mechanism model; Based on the interval frames of the rectangular frame of the target vehicle located by the aerial image, the actual physical moving distance output by the nonlinear influence mechanism model and the time difference of the interval frames are used to estimate the physical speed, so as to obtain the instantaneous speed of the target vehicle, which specifically includes: Based on a deep learning vehicle detection and tracking algorithm, aerial images of the target vehicle are acquired in interval frames to obtain a first aerial image frame and a second aerial image frame; Acquire a first central pixel position of a rectangular frame related to the target positioning in the first aerial image frame, and acquire a second central pixel position of a rectangular frame related to the target positioning in the second aerial image frame; Determining a time difference of the interval frames based on the first aerial image frame and the first aerial image frame; By using the nonlinear influence mechanism model, a physical distance calculation based on pixel displacement is performed on the first central pixel position and the second central pixel position to obtain the actual physical movement distance; Performing physical speed estimation on the actual physical moving distance and the time difference to obtain the instantaneous speed of the target vehicle; Among them, according to The time difference time_dif of the interval frame is obtained; wherein frame_index_β is the first aerial image frame; frame_index_α is the first aerial image frame; and F is the frame rate of the drone aerial video; according to The actual physical moving distance phy_dis is obtained; wherein f(x) is the nonlinear influence mechanism model; center_y β is the ordinate of the second center pixel position; center_y α is the ordinate of the first central pixel position; according to Get the instantaneous speed of the vehicle phy_v; where v UAV is the flight speed of the drone.

2. The method for detecting the instantaneous speed of a vehicle based on drone aerial video according to claim 1, characterized in that: Based on the lane line area of ​​the expressway and the expressway area, extracting the white dashed lane line area image of the expressway specifically includes: Acquire aerial images by using drones along the expressway; wherein the aerial images include: aerial image height, aerial image width, and aerial image depth; By using a preset semantic segmentation technology, pixel area detection is performed on the lane line area where the white dotted line in the aerial image is located to obtain a first mask image; Performing pixel area detection on the highway area in the aerial image to obtain a second mask image; wherein the first mask image and the second mask image both include a foreground area and a background area; according to Obtain the white dotted lane line area image mask after the first mask image and the second mask image intersect f_w_l ; Among them, mask f_r is the second mask image; mask w_l is the first mask image; h∈{0,1,...,H-2,H-1}, and h is the height of the aerial image; w∈{0,1,...,W-2,W-1}, and w is the width of the aerial image; 255 and 0 are both grayscale values ​​of the image.

3. The method for detecting vehicle instantaneous speed based on drone aerial video according to claim 1, characterized in that: The white dashed lane line area image is binarized to obtain a binarized image of the white lane line, specifically including: Perform grayscale processing on the aerial images collected by the drone; Based on the grayscale value of the background area in the white dotted lane line area image, the grayscale values ​​of the background area and the foreground area in the aerial image after grayscale processing are set accordingly to obtain the aerial image to be processed; according to Get the binary image bw based on the white lane line f_w_l Among them, gray f_w_l is the aerial image to be processed; <threshold bw is the binarization threshold, and 0 <threshold bw <255; h∈{0,1,...,H-2,H-1}, where h is the height of the aerial image; w∈{0,1,...,W-2,W-1}, where w is the width of the aerial image.

4. The method for detecting vehicle instantaneous speed based on drone aerial video according to claim 1, characterized in that: The binary image is subjected to peripheral rectangle analysis under a connected domain set to obtain connected domain attributes, specifically including: Based on the largest foreground pixel block composed of adjacent foreground pixels in the binary image, a connected region is determined, and the connected region set is obtained by a connected region analysis method; Performing a peripheral rectangle selection process on each connected domain in the connected domain set, and determining the diagonal length of the peripheral rectangle, the area of ​​the peripheral rectangle, and the height of the peripheral rectangle; Based on the central pixel of the peripheral rectangle in each connected domain, the horizontal and vertical distance coordinates of the binary image are calculated to obtain the central pixel coordinates of the peripheral rectangle; According to dis_unit_pix m =len dashed / diagonal_len m , get the physical distance dis_unit_pix corresponding to each unit pixel in the connected domain m ; Among them, diagonal_len m is the diagonal length of the outer rectangle; len dashed It is the length of the white dotted line of the highway lane divider.

5. The method for detecting vehicle instantaneous speed based on drone aerial video according to claim 4, characterized in that: Before performing peripheral rectangle analysis on the binary image under a connected domain set to obtain connected domain attributes, the method further includes: Performing area constraints and height constraints on the noise factors in the highway to determine an area threshold and a height threshold; wherein the noise factors at least include: high-speed vehicles and other road signs; The connected domain set is subjected to noise elimination processing by using the area threshold and the height threshold to obtain an optimized connected domain set.

6. The method for detecting vehicle instantaneous speed based on drone aerial video according to claim 1, characterized in that: According to the connected domain attribute, the connected domain set is transformed into a two-dimensional coordinate, and based on the two-dimensional coordinate point, the central pixel position in the connected domain set and the physical distance corresponding to the unit pixel are fitted under a nonlinear mechanism to obtain a nonlinear influence mechanism model, which specifically includes: Extracting the physical distance corresponding to the unit pixel of each connected domain in the connected domain attribute and the vertical distance of the central pixel coordinates of the outer rectangle; Determine the physical distance corresponding to the unit pixel as the ordinate of a two-dimensional coordinate system, and determine the vertical distance as the abscissa of the two-dimensional coordinate system; Based on the abscissa and the ordinate, the two-dimensional coordinate point of each connected domain is obtained; and based on optimizing all the two-dimensional coordinate points in the connected domain set, a two-dimensional coordinate point set is generated; Performing noise removal processing on the two-dimensional coordinate point set to obtain an optimized two-dimensional coordinate point set; Counting all two-dimensional horizontal coordinate sets in the optimized two-dimensional coordinate point set; Sorting each horizontal coordinate in the two-dimensional horizontal coordinate set according to the numerical value, and marking the first and last coordinates of the 5th and 95th percentiles of all the sorted two-dimensional horizontal coordinates to obtain the 5% segmentation point and the 95% segmentation point; Based on the 5% segmentation point and the 95% segmentation point, the corresponding coordinates in the optimized two-dimensional coordinate point set are removed from the head and the tail to obtain an accurate two-dimensional coordinate point set; Through the nonlinear model and the model parameters in the nonlinear model, a polynomial fitting process is performed between the central pixel position of the relevant two-dimensional coordinates in the precise two-dimensional coordinate point set and the physical distance corresponding to the unit pixel to obtain the nonlinear influence mechanism model.

7. The method for detecting vehicle instantaneous speed based on drone aerial video according to claim 6, characterized in that: The two-dimensional coordinate point set is subjected to noise removal processing to obtain an optimized two-dimensional coordinate point set, specifically comprising: Calculating the average distance between any two-dimensional coordinate point and an adjacent two-dimensional coordinate point in the two-dimensional coordinate point set, and establishing an average distance set; sorting each average distance in the average distance set by numerical value; and performing 85th percentile marking processing based on all sorted average distances to obtain 85% segmentation points; The average distances of the segmentation points that are greater than 85% in the sorted average distance set are subjected to noise elimination processing to obtain the optimized two-dimensional coordinate point set based on the remaining two-dimensional coordinate points.

8. A vehicle instantaneous speed detection device based on drone aerial video, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the vehicle instantaneous speed detection method based on drone aerial video according to any one of claims 1-7.

Images