Road event identification method and system based on real-time video stream of unmanned aerial vehicle

Abstract

The invention belongs to the technical field of unmanned aerial vehicle intelligent traffic, and discloses a road event identification method and system based on an unmanned aerial vehicle real-time video stream. The method comprises the following steps: marking vehicles in collected unmanned aerial vehicle images, and training an accumulated unmanned aerial vehicle vehicle marking data set by using a YOLOv8 algorithm; the trained YOLOv8 model is applied to a real-time video collected by an unmanned aerial vehicle for vehicle identification, and position information of a vehicle in an identification video frame is intercepted; determining a driving area and a driving direction of the vehicle based on vehicle movement data in two unmanned aerial vehicle pictures which are associated front and back in the video; the direction vector of the inspection center line is set as the forward advancing direction of the unmanned aerial vehicle, the included angle between the direction vectors of the vehicle driving on the left lane and the right lane is calculated in the same coordinate system, and the vehicle on the lanes is judged to move reversely according to the included angle. Compared with traditional ground individual road section arrangement monitoring and manual checking, the monitoring efficiency is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV) intelligent transportation technology, and in particular relates to a road event recognition method and system based on real-time video stream of unmanned aerial vehicle (UAV). Background Art

[0002] With the development of drones, their application in highway inspection has been gradually promoted and applied, which can play a greater role in traffic violations. At the same time, vehicles driving in the wrong direction on the road increase the possibility of traffic accidents, reduce road traffic efficiency, and cause chaos in road traffic order, which is a serious traffic violation. By setting up cameras on busy roads to identify vehicles driving in the wrong direction, a certain effect can be achieved, which is mainly suitable for setting up on key roads. However, in the face of huge road mileage, the reverse situation occurs irregularly and the camera monitoring range is limited. Reverse identification on large-scale and long sections has become an urgent problem to be solved. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide a road event recognition method and system based on real-time video stream of unmanned aerial vehicle.

[0004] The technical solution is as follows: A road event recognition method based on real-time video stream of a drone, comprising:

[0005] S1: Label the vehicles in the collected drone images and use the YOLOv8 algorithm to train the accumulated drone vehicle labeling dataset; apply the trained YOLOv8 model to the real-time video collected by the drone for vehicle recognition, and intercept the location information of the vehicle in the recognized video frame;

[0006] S2: Determine the driving area and direction of the vehicle based on the vehicle movement data in the two drone images that are related to each other in the video;

[0007] S3, the direction vector of the inspection centerline is set as the positive direction of the UAV, and the angle between the direction vectors of the vehicle in the left lane and the right lane is calculated in the same coordinate system, and the vehicle on the lane is judged to be in the reverse direction based on the angle.

[0008] In step S2, based on the vehicle movement data in the two drone images that are related to each other in the video, the driving area and driving direction of the vehicle are determined, including:

[0009] S201, identifying the coordinate position (x1, y1) of vehicle A in image P1 at time T1, and identifying the coordinate position (x2, y2) of vehicle A in image P2 at time T2;

[0010] S202, aligning the images P1 and P2, and recording the coordinate position (x1', y1') of the vehicle A at time T1 and the coordinate position (x2', y2') of the vehicle A at time T2 after alignment;

[0011] S203, fitting the driving area in the positive and negative directions of the road for the registered image, obtaining a linear equation of the fitted road in the positive and negative directions, and determining the driving area and driving direction of the vehicle according to the coordinate position (x1', y1') at time T1 and the coordinate position (x2', y2') at time T2.

[0012] In step S202, the image P1 and the image P2 are registered, including: two images acquired at different times, using different sensors and under different conditions are matched and superimposed using an improved time series image registration method, so that the images in the image P1 and the image P2 correspond to points at the same position in space one by one;

[0013] Input two original images P of the same road at different times, and construct two basic isomorphic graphs after denoising, which are expressed as logarithmic ratio isomorphic graphs P e and mean ratio heterogeneous graph P f , respectively for P e and P f Perform neighborhood driving direction feature description to obtain the driving direction feature description matrix P E and P F ; Using similarity comparison, we can get the driving direction feature description matrix P E and P F The corresponding matching matrix H E and H F ; Based on the spectrum classification method of matching matrix superposition, the matching matrix H E and H F Perform classification analysis and obtain the change registration result based on the obtained classification results.

[0014] Furthermore, the spectrum classification method of matching matrix superposition includes:

[0015] 1) The logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The driving direction feature description column vectors of all vehicle image pixels in are arranged in sequence as different columns to obtain the corresponding driving direction feature description matrix P E and P F ;

[0016] 2) The obtained driving direction feature description matrix P E and driving direction feature description matrix P F , use the Euclidean distance of each column vector to measure its similarity, construct the corresponding similarity matrix, and get the matching matrix HE and H F ;

[0017] 3) Initialize the matching matrix H E and H F The weight coefficient q e and q f , calculate the superposition matching matrix, the expression is:

[0018]

[0019] Where H is the superposition matching matrix, k is the kth vehicle image pixel, q k is the k-th vehicle image pixel matching weight coefficient, H k The value of the matching matrix is ​​superimposed on the k-th vehicle image pixel;

[0020] 4) Based on the sampling spectrum classification method, the driving direction characteristic trend vector z and the diagonal matrix U of the superimposed matching matrix H are obtained;

[0021] 5) Verify whether the iteration condition satisfies the error δ=|z now -z old |, if ||δ||≥ε and Iterator≤Iter, then execute step 6), otherwise, the iteration ends and the driving direction characteristic trend vector z is output, where ε is the set allowable error value in degrees; z now is the characteristic trend vector of the current driving direction, z old is the characteristic trend vector of the actual driving direction, Iterator is the iteration value, and Iter is the generated value;

[0022] 6) Update the matching matrix weight coefficient.

[0023] In step 6), the matching matrix weight coefficient is updated, and the expression is:

[0024] η k =z G U k z

[0025]

[0026] Where η k Update the weight coefficients for the k-th vehicle image pixel diagonal matrix once, z G is the characteristic trend vector of the driving direction at the Gth moment, U k is the diagonal matrix of pixels of the k-th vehicle image, is the diagonal matrix and the superposition matching matrix G, the word updates the driving direction feature trend vector offset value at the moment, H kis the value of the matching matrix superimposed on the k-th vehicle image pixel, τ k is the deviation rate in a single update of the weight coefficient;

[0027] According to the one-dimensional search, we can get x e , the one-dimensional search equation expression is:

[0028]

[0029] x e Substitution Get x f , calculate the updated weight coefficient, the expression is:

[0030]

[0031] In the formula, x e Log-ratio heterogeneity Figure 1 The horizontal coordinate value of the vehicle image to be searched, x f Mean ratio heterogeneity Figure 1 The horizontal coordinate value of the vehicle image for dimensional search, q k is the matching weight coefficient of the kth vehicle image pixel.

[0032] Furthermore, the improved time series image registration method includes:

[0033] (1) De-noising two original images P of the same road at different times to obtain denoised images K1 and K2;

[0034] (2) Using the logarithmic ratio operator and the mean ratio operator, the denoised image K1 and the denoised image K2 are operated to obtain the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f , the expression is:

[0035]

[0036] Among them, d1 and d2 are the local means of K1 and K2 respectively;

[0037] (3) Extract the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The e-layer neighborhood vehicle image pixel feature matrix of each vehicle image pixel in , where e ≥ 1, and e is an odd number; the extracted neighborhood feature vehicle image pixel matrix is ​​expanded into columns to obtain the column vector describing the driving direction feature of the vehicle image pixel point; the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P fThe driving direction feature description column vectors of all vehicle image pixels in are arranged in sequence as different columns to obtain the corresponding driving direction feature description matrix P E and driving direction feature description matrix P F ;

[0038] (4) The obtained driving direction feature description matrix P E and driving direction feature description matrix P F , use the Euclidean distance of each column vector to measure its similarity, and construct the corresponding similarity matrix and matching matrix H E and H F ;

[0039] (5) The obtained matching matrix H E and H F , classification is performed using a matching matrix superposition spectrum classification method;

[0040] (6) Use the classification method to classify the output driving direction characteristic trend vector z, and classify the obtained vehicle image pixel classification results according to the logarithmic ratio heterogeneous graph P e Or mean ratio heterogeneous graph P f The arrangement order of the corresponding vehicle image pixels is restored to the size of the original image to obtain the final change registration result map and output the result.

[0041] In step S203, the registered image is fitted with the driving area in the positive and negative directions of the road to obtain a linear equation in the positive and negative directions of the fitted road, including:

[0042] S2031: Use drone highway images as data sets, annotate the acquired drone highway images, and create sample sets according to the U-Net network model semantic segmentation algorithm. By building a U-Net network model, use the created sample sets to train the recognition semantic segmentation model. Use the video stream frame extraction technology to extract the road domain from the frame-extracted drone images of the real-time road video taken by the drone. In the road domain extraction, use the real-time video stream frame extraction technology to extract the sequenced images for processing operations. Realize the road domain and non-road domain extraction based on drone aerial images.

[0043] S2032: Divide the road into positive and negative directions based on the generated road area and non-road area binary map.

[0044] In step S2032, the forward and reverse directions of the road are divided based on the generated road area and non-road area binary map, including:

[0045] Step 1: Get the width and height (w, h) of the binary image, and calculate the maximum x value, maximum x value and average value of the vehicle image pixel with an X-axis value of 1 based on the 30 known Y-axis coordinates; get the coordinates of 30 points on both sides of the road; in the two-dimensional coordinate system formed by the binary image, the linear equation obtained by least squares fitting is:

[0046] Left point set: [(x0 min ,y0),(x1 min ,y1)…(x29 min ,y29),(x30 min ,y30)];

[0047] Right point set: [(x0 max ,y0),(x1 max ,y1)…(x29 max ,y29),(x30 max ,y30)];

[0048] The middle point set: [(x0 avg ,y0),(x1 avg ,y1)…(x29 avg ,y29),(x30 avg ,y30)];

[0049] Step 2: For the 30 points in the left lane, the inspection center line, and the right lane, use the least squares method to perform linear fitting to obtain the respective linear equations, which are expressed as follows:

[0050] y l =k l x+b l

[0051] y c =k c x+b c

[0052] y r =k r x+b r

[0053] In the formula, y l ,y c ,y r are the linear fitting equations for the left lane, inspection center line, and right lane, respectively, k l ,k c ,k r are the slopes of the linear fitting equations for the left lane, inspection center line, and right lane, respectively. l ,b c ,b rThey are the constants of the linear fitting equations of the left lane, inspection center line, and right lane respectively;

[0054] Step 3, for the left lane y l Lower side and inspection center line y c The upper area is the reverse driving area, and the inspection center line y c The area below and above the right lane is the forward driving area.

[0055] In step S3, the angle between the direction vectors of the vehicle traveling in the left lane and the right lane is calculated in the same coordinate system, and the expression is:

[0056]

[0057] θ=arccos(cosθ)

[0058] In the formula, θ is the vector The angle between is the direction vector of the left lane, is the direction vector of the right lane;

[0059] The reverse driving of vehicles on the lane is judged based on the angle, including:

[0060] In the left lane, θ is 0° to 90°, driving in the opposite direction, and θ is 90° to 180°, driving normally;

[0061] In the right lane, when θ is 0° to 90°, the vehicle is driving normally; when θ is 90° to 180°, the vehicle is driving in the opposite direction.

[0062] Another object of the present invention is to provide a road event recognition system based on real-time video stream of unmanned aerial vehicle, which is implemented by the road event recognition method based on real-time video stream of unmanned aerial vehicle, and the system comprises:

[0063] The vehicle identification module is used to annotate the vehicles in the collected drone images and use the YOLOv8 algorithm to train the accumulated drone vehicle annotation data set; the trained YOLOv8 model is applied to the real-time video collected by the drone for vehicle identification, and the location information of the vehicle in the identified video frame is intercepted;

[0064] The vehicle driving direction determination module is used to determine the vehicle's driving area and driving direction based on the vehicle movement data in the two drone images that are related to each other in the video;

[0065] The vehicle reverse traffic judgment module is used to set the direction vector of the inspection center line as the positive direction of the UAV, calculate the angle between the direction vectors of the vehicle in the left lane and the right lane in the same coordinate system, and judge the reverse traffic of the vehicle on the lane based on the angle.

[0066] Combined with all the above technical solutions, the beneficial effects of the present invention are as follows: the present invention performs appropriate training on the highway images taken by the drone to obtain a road classification model and realize a classification binary map of the road and non-road areas; divides the generated binary map into forward and reverse driving areas of the road; marks the vehicles in the drone images, and performs appropriate training to obtain a vehicle recognition model to realize real-time recognition of the vehicle; appropriately aligns the two frames of images taken by the drone at time T1 and time T2, and calculates the distance and direction of vehicle A's movement, sets the direction of the drone to the forward direction, and determines whether the vehicle is going in the reverse direction based on the direction of the drone and the driving direction of the vehicle in the area. Drones can quickly cover a large area, ignoring the limitations of traditional fixed-point monitoring due to factors such as terrain and vegetation, and quickly discover and identify reverse events. Compared with the traditional monitoring of individual sections on the ground and manual viewing, the monitoring efficiency is greatly improved. At the same time, the recognition of other road traffic events can be increased in the future to improve drone cruising. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0068] Figure 1 is a flow chart of a road event recognition method based on real-time video stream of a drone provided by an embodiment of the present invention;

[0069] Figure 2 It is a flow chart of a method for extracting road areas based on drone images provided by an embodiment of the present invention;

[0070] Figure 3 This is a road area recognition effect diagram provided by an embodiment of the present invention;

[0071] Figure 4 A binary road area and non-road area map provided by an embodiment of the present invention;

[0072] Figure 5 is the linear fitting y provided in the embodiment of the present invention l ,y c ,y r Schematic diagram of . DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0074] Embodiment 1, as Figure 1 As shown, the road event recognition method based on the real-time video stream of the UAV includes:

[0075] S1: Label the vehicles in the collected drone images and use the YOLOv8 algorithm to train the accumulated drone vehicle labeling dataset; apply the trained YOLOv8 model to the real-time video collected by the drone for vehicle recognition, and intercept the location information of the vehicle in the recognized video frame;

[0076] S101, annotate the vehicles in the collected drone images, and use the YOLOv8 algorithm to train the accumulated drone vehicle annotation data set; wherein the YOLOv8 algorithm has good characteristics in speed, efficiency, and accuracy recognition in target recognition;

[0077] S102, applying the trained YOLOv8 model to the real-time video of the drone to perform vehicle recognition, and intercepting the position of the vehicle in the recognized video frame; wherein, intercepting the position of the vehicle in the recognized video frame mainly applies the real-time video stream frame extraction technology, uses the model trained by the YOLOv8 algorithm on the extracted frame image, realizes the vehicle recognition function in the image, and records the coordinates of the vehicle in the frame image according to the vehicle recognized by the YOLOv8 model;

[0078] S2: Determine the driving area and direction of the vehicle based on the vehicle movement data in the two drone images that are related to each other in the video;

[0079] S201, identifying the coordinate position (x1, y1) of vehicle A in image P1 at time T1, and identifying the coordinate position (x2, y2) of vehicle A in image P2 at time T2;

[0080] S202, aligning picture P1 and picture P2, recording the coordinate position (x1', y1') of vehicle A at time T1 and the coordinate position (x2', y2') of vehicle A at time T2 after alignment; in aligning picture P1 and picture P2, two images acquired at different times, using different sensors and under different conditions are matched and superimposed using an improved time series image registration method, so that the images in picture P1 and picture P2 correspond one-to-one to points at the same position in space; according to the same scene of the images at different times, the improved time series image registration method is selected to align picture P1 and picture P2, so as to solve the problem of inaccurate images in matching and superimposing multiple images in the prior art.

[0081] Input two original images P of the same road at different times, and construct two basic isomorphic graphs after denoising, which are expressed as logarithmic ratio isomorphic graphs P e and mean ratio heterogeneous graph P f , respectively for P e and P f Perform neighborhood driving direction feature description to obtain the driving direction feature description matrix P E and P F ; Using similarity comparison, we can get the driving direction feature description matrix P E and P F The corresponding matching matrix H E and H F ; Based on the spectrum classification method of matching matrix superposition, the matching matrix H E and H F Perform classification analysis and obtain the change registration result based on the obtained classification results.

[0082] S203, fitting the driving area in the forward and reverse directions of the road for the registered image, obtaining a linear equation of the forward and reverse directions of the road after fitting, and determining the driving area and driving direction of the vehicle according to the coordinate position (x1', y1') at time T1 and the coordinate position (x2', y2') at time T2;

[0083] S2031: Use drone highway images as data sets, annotate the acquired drone highway images, and create sample sets according to the U-Net network model semantic segmentation algorithm. By building a U-Net network model, use the created sample sets to train the recognition semantic segmentation model. Use the video stream frame extraction technology to extract the road domain from the frame-extracted drone images of the real-time road video taken by the drone. In the road domain extraction, use the real-time video stream frame extraction technology to extract the sequenced images for processing operations. Realize the road domain and non-road domain extraction based on drone aerial images.

[0084] S2032: Divide the road into positive and negative directions based on the generated road area and non-road area binary map.

[0085] In step S2032, the forward and reverse directions of the road are divided based on the generated road area and non-road area binary map, including:

[0086] Step 1: Get the width and height (w, h) of the binary image, and calculate the maximum x value, maximum x value and average value of the vehicle image pixel with an X-axis value of 1 based on the 30 known Y-axis coordinates; get the coordinates of 30 points on both sides of the road; in the two-dimensional coordinate system formed by the binary image, the linear equation obtained by least squares fitting is:

[0087] Left point set: [(x0 min ,y0),(x1 min ,y1)…(x29 min ,y29),(x30 min ,y30)];

[0088] Right point set: [(x0 max ,y0),(x1 max ,y1)…(x29 max ,y29),(x30 max ,y30)];

[0089] The middle point set: [(x0 avg ,y0),(x1 avg ,y1)…(x29 avg ,y29),(x30 avg ,y30)];

[0090] Step 2: For the 30 points in the left lane, the inspection center line, and the right lane, use the least squares method to perform linear fitting to obtain the respective linear equations, which are expressed as follows:

[0091] y l =k l x+b l

[0092] y c =k c x+b c

[0093] y r =k r x+b r

[0094] In the formula, y l ,y c ,y r are the linear fitting equations for the left lane, inspection center line, and right lane, respectively, k l ,k c ,k rare the slopes of the linear fitting equations for the left lane, inspection center line, and right lane, respectively. l ,b c ,b r They are the constants of the linear fitting equations of the left lane, inspection center line, and right lane respectively;

[0095] Step 3, for the left lane y l Lower side and inspection center line y c The upper area is the reverse driving area, and the inspection center line y c The area below and above the right lane is the forward driving area.

[0096] S3, the direction vector of the inspection centerline is set as the positive direction of the UAV, and the angle between the direction vectors of the vehicle in the left lane and the right lane is calculated in the same coordinate system, and the vehicle on the lane is judged to be in the reverse direction based on the angle.

[0097] In one embodiment, a spectrum classification method of matching matrix superposition includes:

[0098] 1) The logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The driving direction feature description column vectors of all vehicle image pixels in are arranged in sequence as different columns to obtain the corresponding driving direction feature description matrix P E and P F ;

[0099] 2) The obtained driving direction feature description matrix P E and driving direction feature description matrix P F , use the Euclidean distance of each column vector to measure its similarity, construct the corresponding similarity matrix, and get the matching matrix H E and H F ;

[0100] 3) Initialize the matching matrix H E and H F The weight coefficient q e and q f , calculate the superposition matching matrix, the expression is:

[0101]

[0102] Where H is the superposition matching matrix, k is the kth vehicle image pixel, q k is the k-th vehicle image pixel matching weight coefficient, H k The value of the matching matrix is ​​superimposed on the k-th vehicle image pixel;

[0103] 4) Based on the sampling spectrum classification method, the driving direction characteristic trend vector z and the diagonal matrix U of the superimposed matching matrix H are obtained;

[0104] 5) Verify whether the iteration condition satisfies the error δ=|z now -z old |, if ||δ||≥ε and Iterator≤Iter, then execute step 6), otherwise, the iteration ends and the driving direction characteristic trend vector z is output, where ε is the set allowable error value in degrees; z now is the characteristic trend vector of the current driving direction, z old is the characteristic trend vector of the actual driving direction, Iterator is the iteration value, and Iter is the generated value;

[0105] 6) Update the matching matrix weight coefficient, the expression is:

[0106] η k =z G U k z

[0107]

[0108] Where η k Update the weight coefficients for the k-th vehicle image pixel diagonal matrix once, z G is the characteristic trend vector of the driving direction at the Gth moment, U k is the diagonal matrix of pixels of the k-th vehicle image, is the diagonal matrix and the superposition matching matrix G, the word updates the driving direction feature trend vector offset value at the moment, H k is the value of the matching matrix superimposed on the k-th vehicle image pixel, τ k is the deviation rate in a single update of the weight coefficient;

[0109] According to the one-dimensional search, we can get x e , the one-dimensional search equation expression is:

[0110]

[0111] x e Substitution Get x f , calculate the updated weight coefficient, the expression is:

[0112]

[0113] In the formula, x e Log-ratio heterogeneity Figure 1 The horizontal coordinate value of the vehicle image to be searched, x f Mean ratio heterogeneity Figure 1The horizontal coordinate value of the vehicle image for dimensional search, q k is the matching weight coefficient of the kth vehicle image pixel.

[0114] In one embodiment, an improved time series image registration method includes:

[0115] (1) De-noising two original images P of the same road at different times to obtain denoised images K1 and K2;

[0116] (2) Using the logarithmic ratio operator and the mean ratio operator, the denoised image K1 and the denoised image K2 are operated to obtain the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f , the expression is:

[0117]

[0118] Among them, d1 and d2 are the local means of K1 and K2 respectively;

[0119] (3) Extract the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The e-layer neighborhood vehicle image pixel feature matrix of each vehicle image pixel in , where e ≥ 1, and e is an odd number; the extracted neighborhood feature vehicle image pixel matrix is ​​expanded into columns to obtain the column vector describing the driving direction feature of the vehicle image pixel point; the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The driving direction feature description column vectors of all vehicle image pixels in are arranged in sequence as different columns to obtain the corresponding driving direction feature description matrix P E and driving direction feature description matrix P E ;

[0120] (4) The obtained driving direction feature description matrix P E and driving direction feature description matrix P E , use the Euclidean distance of each column vector to measure its similarity, and construct the corresponding similarity matrix and matching matrix H E and H F ;

[0121] (5) The obtained matching matrix H E and H F , classification is performed using a matching matrix superposition spectrum classification method;

[0122] (6) Use the classification method to classify the output driving direction characteristic trend vector z, and classify the obtained vehicle image pixel classification results according to the logarithmic ratio heterogeneous graph P eOr mean ratio heterogeneous graph P f The arrangement order of the corresponding vehicle image pixels is restored to the size of the original image to obtain the final change registration result map and output the result.

[0123] It can be understood that the driving direction feature description process takes into account the imaging characteristics of image P, and uses the neighborhood information of vehicle image pixels to characterize the vehicle image pixels, which can effectively suppress the image background information while greatly enhancing the area of ​​interest, and can effectively improve the registration accuracy; at the same time, the image superposition process uses the matching matrix superposition spectrum classification method and the idea of ​​optimization solution in mathematics to seek the optimal weight ratio of each heterogeneous graph, which effectively solves the problem of heterogeneous graph selection and can be extended to various types of image change registration scenarios, with clearer physical meaning.

[0124] The present invention adds a vehicle image pixel driving direction feature description process before superimposing the basic heterogeneous graph. Each vehicle image pixel is affected by the scattering of the neighboring vehicle image pixel points, which can be regarded as a certain degree of superposition of its neighboring vehicle image pixel points at this point. The vehicle image pixel neighborhood feature matrix is ​​used to characterize the vehicle image pixel point. Compared with the method of using a single vehicle image pixel grayscale value to describe the vehicle image pixel in the prior art, the information of the original image can be better utilized.

[0125] The present invention utilizes a matching matrix-based superposition spectral classification method in the process of superimposing basic heterogeneous graphs. Compared with the prior art method of superimposing using methods such as wavelet superposition, the present invention can learn the optimal weight ratio of each heterogeneous graph, realize automatic selection of heterogeneous graphs, and has good generalizability. In the learning process, a random sampling spectral classification method is also combined to improve the solution efficiency, overcome the problem that spectral classification cannot solve large-scale data, and realize the analysis and accurate labeling of P image registration change information.

[0126] S203, fitting the positive and negative driving areas of the road for the registered image, obtaining a linear equation straight line that can distinguish the positive and negative directions of the road after fitting, and determining the driving area and driving direction of the vehicle according to the coordinates (x1', y1') at time T1 and the coordinates (x2', y2') at time T2;

[0127] Exemplarily, the registered image is fitted with the driving area in the positive and negative directions of the road, and the linear equation straight line that can distinguish the positive and negative directions of the road after fitting includes:

[0128] S2031: Annotate the acquired drone highway images, use the U-Net network to train the semantic segmentation model for the annotated samples, and realize the extraction of road and non-road areas based on drone aerial images. Specifically include:

[0129] (1) Using UAV highway images as a data set, and creating a sample set according to the U-Net network model semantic segmentation algorithm; the U-Net semantic segmentation algorithm is used to identify the road area of ​​the image, and the classification is only road area and non-road area, which is relatively simple. A model with high recognition accuracy can be trained by simply increasing the number of samples;

[0130] (2) By building a U-Net network model, the prepared sample set is used to train the recognition semantic segmentation model;

[0131] (3) The real-time road video taken by the drone is extracted by using the video stream frame extraction technology, and the road area is extracted from the extracted drone image. The process of the road area extraction method based on drone image is as follows: Figure 2 shown.

[0132] Take one of the frames to extract the effect Figure 3 , and generate binary images of road area and non-road area, setting the black area value to 0 and the white area value to 1; in the road area extraction, the real-time video stream frame extraction technology is used to extract the sequential pictures for processing;

[0133] S2032, dividing the road into positive and negative directions based on the road area and non-road area binary map generated in step S2031.

[0134] When the drone takes off, set the flight route of the drone according to the direction of the inspection road, and the direction of the drone along the road is the positive direction, and the direction facing the drone is the reverse direction. Therefore, if the right lane of the drone is the positive direction of the vehicle, then the reverse driving of the right lane is reverse driving; if the left lane of the drone is the reverse direction of the vehicle, then the positive driving of the left lane is reverse driving; specifically including:

[0135] Step 1: Get the width and height (w, h) of the binary image, and calculate the maximum x value, maximum x value and average value of the vehicle image pixel with an X-axis value of 1 based on the 30 known Y-axis coordinates; get the coordinates of 30 points on both sides of the road; in the two-dimensional coordinate system formed by the binary image, the linear equation obtained by least squares fitting is:

[0136] Left point set: [(x0 min ,y0),(x1 min ,y1)…(x29 min ,y29),(x30 min ,y30)];

[0137] Right point set: [(x0 max ,y0),(x1 max ,y1)…(x29 max ,y29),(x30 max ,y30)];

[0138] The middle point set: [(x0 avg ,y00,(x1 avg ,y1)…(x29 avg ,y29),(x30 avg ,y30)];

[0139] Step 2: For the 30 points in the left lane, the inspection center line, and the right lane, use the least squares method to perform linear fitting to obtain the respective linear equations, which are expressed as follows:

[0140] y l =k l x+b l

[0141] y c =k c x+b c

[0142] y r =k r x+b r

[0143] In the formula, y l ,y c ,y r are the linear fitting equations for the left lane, inspection center line, and right lane, respectively, k l ,k c ,k r are the slopes of the linear fitting equations for the left lane, inspection center line, and right lane, respectively. l ,b c ,b r They are the constants of the linear fitting equations of the left lane, inspection center line, and right lane respectively;

[0144] Step 3, for the left lane y l Lower side and inspection center line y c The upper area is the reverse driving area, and the inspection center line y c The area below and above the right lane is the forward driving area.

[0145] Take the inspection center line y c The direction vector is set as the positive direction of the UAV, and the direction vector of the vehicle in the left lane and the right lane is calculated in the same coordinate system. The angle between the left lane and the right lane is calculated in the same coordinate system. The angle between the direction vectors of the vehicle in the left lane and the right lane is calculated as follows:

[0146]

[0147] θ=arccos(cosθ)

[0148] In the formula, θ is the vector The angle between is the direction vector of the left lane, is the direction vector of the right lane;

[0149] The reverse driving of vehicles on the lane is judged based on the angle, including:

[0150] In the left lane, θ is 0° to 90°, driving in the opposite direction, and θ is 90° to 180°, driving normally;

[0151] In the right lane, when θ is 0° to 90°, the vehicle is driving normally; when θ is 90° to 180°, the vehicle is driving in the opposite direction.

[0152] Table 1. Vehicle reverse driving judgment on the lane

[0153]

[0154]

[0155] Embodiment 2, a road event recognition system based on real-time video stream of a drone provided by an embodiment of the present invention, the system comprises:

[0156] The vehicle identification module is used to annotate the vehicles in the collected drone images and use the YOLOv8 algorithm to train the accumulated drone vehicle annotation data set; the trained YOLOv8 model is applied to the real-time video collected by the drone for vehicle identification, and the location information of the vehicle in the identified video frame is intercepted;

[0157] The vehicle driving direction determination module is used to determine the vehicle's driving area and driving direction based on the vehicle movement data in the two drone images that are related to each other in the video;

[0158] The vehicle reverse traffic judgment module is used to set the direction vector of the inspection center line as the positive direction of the UAV, calculate the angle between the direction vectors of the vehicle in the left lane and the right lane in the same coordinate system, and judge the reverse traffic of the vehicle on the lane based on the angle.

[0159] The vehicle identification module comprises:

[0160] The vehicle annotation submodule is used to annotate the vehicles in the collected drone images and use the YOLOv8 algorithm to train the accumulated drone vehicle annotation dataset;

[0161] The vehicle position recognition submodule is used to apply the trained YOLOv8 model to the real-time video of the drone for vehicle recognition and to intercept and identify the position of the vehicle in the video frame;

[0162] The vehicle driving direction determination module includes: a different picture vehicle recognition submodule, which is used to recognize the coordinate position (x1, y1) of vehicle A in picture P1 at time T1, and recognize the coordinate position (x2, y2) of vehicle A in picture P2 at time T2;

[0163] Different picture registration submodules are used to register the pictures P1 and P2, and record the coordinate position (x1', y1') of the vehicle A at time T1 and the coordinate position (x2', y2') of the vehicle A at time T2 after registration;

[0164] The vehicle driving area and driving direction determination submodule is used to fit the driving area in the positive and negative directions of the road for the registered image, obtain the first-order equation of the positive and negative directions of the road after fitting, and determine the vehicle's driving area and driving direction according to the coordinate position (x1', y1') at time T1 and the coordinate position (x2', y2') at time T2;

[0165] The vehicle driving area and driving direction determination submodule includes:

[0166] The road and non-road area extraction module based on UAV aerial images is used to annotate the acquired UAV highway images, and use the U-Net network to train the semantic segmentation model for the annotated samples to realize the road and non-road area extraction based on UAV aerial images.

[0167] The module for dividing the positive and negative directions of the road is used to divide the positive and negative directions of the road based on the generated road area and non-road area binary map, and to divide the vehicle reverse driving area and the forward driving area.

[0168] The vehicle reverse driving judgment module comprises:

[0169] Angle calculation submodule, used to inspect the center line y c The direction vector is set as the positive direction of the UAV, and the direction vector of the vehicle in the left lane and the right lane is calculated in the same coordinate system. The angle between

[0170] The reverse traffic judgment submodule is used to judge whether the vehicle on the lane is reverse traffic based on the angle.

[0171] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A road event recognition method based on real-time video stream of unmanned aerial vehicle, characterized in that: The method includes: S1: Label the vehicles in the collected drone images and use the YOLOv8 algorithm to train the accumulated drone vehicle labeling dataset; apply the trained YOLOv8 model to the real-time video collected by the drone for vehicle recognition, and intercept the location information of the vehicle in the recognized video frame; S2: Determine the driving area and direction of the vehicle based on the vehicle movement data in the two drone images that are related to each other in the video; S3, the direction vector of the inspection centerline is set as the positive direction of the UAV, and the angle between the direction vectors of the vehicle in the left lane and the right lane is calculated in the same coordinate system, and the vehicle on the lane is judged to be in the reverse direction based on the angle.

2. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 1 is characterized in that: In step S2, based on the vehicle movement data in the two drone images that are related to each other in the video, the driving area and driving direction of the vehicle are determined, including: S201, identifying the coordinate position (x1, y1) of vehicle A in image P1 at time T1, and identifying the coordinate position (x2, y2) of vehicle A in image P2 at time T2; S202, aligning the images P1 and P2, and recording the coordinate position (x1′, y1′) of the vehicle A at time T1 and the coordinate position (x2′, y2′) of the vehicle A at time T2 after alignment; S203, fitting the driving area in the positive and negative directions of the road for the registered image, obtaining a linear equation of the fitted road in the positive and negative directions, and determining the driving area and driving direction of the vehicle according to the coordinate position (x1′, y1′) at time T1 and the coordinate position (x2′, y2′) at time T2.

3. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 2 is characterized in that: In step S202, the image P1 and the image P2 are registered, including: two images acquired at different times, using different sensors and under different conditions are matched and superimposed using an improved time series image registration method, so that the images in the image P1 and the image P2 correspond to points at the same position in space one by one; Input two original images P of the same road at different times, and construct two basic isomorphic graphs after denoising, which are expressed as logarithmic ratio isomorphic graphs P e and mean ratio heterogeneous graph P f , respectively for P e and P f Perform neighborhood driving direction feature description to obtain the driving direction feature description matrix P E and P F ; Using similarity comparison, we can get the driving direction feature description matrix P E and P F The corresponding matching matrix H E and H F ; Based on the spectrum classification method of matching matrix superposition, the matching matrix H E and H F Perform classification analysis and obtain the change registration result based on the obtained classification results.

4. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 3 is characterized in that: The spectrum classification method of matching matrix superposition includes: 1) The logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The driving direction feature description column vectors of all vehicle image pixels in are arranged in sequence as different columns to obtain the corresponding driving direction feature description matrix P E and P F ; 2) The obtained driving direction feature description matrix P E and driving direction feature description matrix P F , use the Euclidean distance of each column vector to measure its similarity, construct the corresponding similarity matrix, and get the matching matrix H E and H F ; 3) Initialize the matching matrix H E and H F The weight coefficient q e and q f , calculate the superposition matching matrix, the expression is: Where H is the superposition matching matrix, k is the kth vehicle image pixel, q k is the k-th vehicle image pixel matching weight coefficient, H k The value of the matching matrix is ​​superimposed on the k-th vehicle image pixel; 4) Based on the sampling spectrum classification method, the driving direction characteristic trend vector z and the diagonal matrix U of the superimposed matching matrix H are obtained; 5) Verify whether the iteration condition satisfies the error δ=|z now -z old |, if ||δ||≥ε and Iterator≤Iter, then execute step 6), otherwise, the iteration ends and the driving direction characteristic trend vector z is output, where ε is the set allowable error value in degrees; z now is the characteristic trend vector of the current driving direction, z old is the characteristic trend vector of the actual driving direction, Iterator is the iteration value, and Iter is the generated value; 6) Update the matching matrix weight coefficient.

5. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 4 is characterized in that: In step 6), the matching matrix weight coefficient is updated, and the expression is: η k =from G AT k With Where η k Update the weight coefficients for the k-th vehicle image pixel diagonal matrix once, z G is the characteristic trend vector of the driving direction at the Gth moment, U k is the diagonal matrix of pixels of the k-th vehicle image, is the diagonal matrix and the superposition matching matrix G, the word updates the driving direction feature trend vector offset value at the moment, H k is the value of the matching matrix superimposed on the k-th vehicle image pixel, τ k is the deviation rate in a single update of the weight coefficient; According to the one-dimensional search, x e , the one-dimensional search equation expression is: x e Substitution Get x f , calculate the updated weight coefficient, the expression is: In the formula, x e is the horizontal coordinate value of the vehicle image in the one-dimensional search of the logarithmic ratio heterogeneous graph, x f is the horizontal coordinate value of the vehicle image in the one-dimensional search of the mean ratio heterogeneous graph, q k is the matching weight coefficient of the kth vehicle image pixel.

6. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 3 is characterized in that: Improved time series image registration method, including: (1) De-noising two original images P of the same road at different times to obtain denoised images K1 and K2; (2) Using the logarithmic ratio operator and the mean ratio operator, the denoised image K1 and the denoised image K2 are operated to obtain the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f , the expression is: Among them, d1 and d2 are the local means of K1 and K2 respectively; (3) Extract the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The e-layer neighborhood vehicle image pixel feature matrix of each vehicle image pixel in , where e ≥ 1, and e is an odd number; the extracted neighborhood feature vehicle image pixel matrix is ​​expanded into columns to obtain the column vector describing the driving direction feature of the vehicle image pixel point; the logarithmic ratio heterogeneous graph P e and mean ratio heterogeneous graph P f The driving direction feature description column vectors of all vehicle image pixels in are arranged in sequence as different columns to obtain the corresponding driving direction feature description matrix P E and driving direction feature description matrix P F ; (4) The obtained driving direction feature description matrix P E and driving direction feature description matrix P F , use the Euclidean distance of each column vector to measure its similarity, and construct the corresponding similarity matrix and matching matrix H E and H F ; (5) The obtained matching matrix H E and H F , classification is performed using a matching matrix superposition spectrum classification method; (6) Use the classification method to classify the output driving direction characteristic trend vector z, and classify the obtained vehicle image pixel classification results according to the logarithmic ratio heterogeneous graph P e Or mean ratio heterogeneous graph P f The arrangement order of the corresponding vehicle image pixels is restored to the size of the original image, the final change registration result map is obtained, and the result is output.

7. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 2 is characterized in that: In step S203, the registered image is fitted with the driving area in the positive and negative directions of the road to obtain a linear equation in the positive and negative directions of the fitted road, including: S2031: Use drone highway images as data sets, annotate the acquired drone highway images, and create sample sets according to the U-Net network model semantic segmentation algorithm. By building a U-Net network model, use the created sample sets to train the recognition semantic segmentation model. Use the video stream frame extraction technology to extract the road domain from the frame-extracted drone images of the real-time road video taken by the drone. In the road domain extraction, use the real-time video stream frame extraction technology to extract the sequenced images for processing operations. Realize the road domain and non-road domain extraction based on drone aerial images. S2032: Divide the road into positive and negative directions based on the generated road area and non-road area binary map.

8. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 7 is characterized in that: In step S2032, the forward and reverse directions of the road are divided based on the generated road area and non-road area binary map, including: Step 1, obtain the width and height (w, h) of the binary image, and calculate the maximum x value, maximum x value and average value of the vehicle image pixel with an X-axis value of 1 based on the 30 known Y-axis coordinates; obtain the coordinates of 30 points on both sides of the road; in the two-dimensional coordinate system formed by the binary image, the linear equation obtained by least squares fitting is: Left point set: [(x0 min , y0), (x1 min ,y1)…(x29 min , y29), (x30 min ,y30)]; Right point set: [(x0 max , y0), (x1 max ,y1)…(x29 max , y29), (x30 max ,y30)]; The middle point set: [(x0 avg , y0), (x1 avg ,y1)…(x29 avg , y29), (x30 avg ,y30)]; Step 2: For the 30 points in the left lane, the inspection center line, and the right lane, use the least squares method to perform linear fitting to obtain the respective linear equations, which are expressed as follows: y l =k l x+b l y c =k c x+b c y r =k r x+b r In the formula, y l ,y c ,y r are the linear fitting equations for the left lane, inspection center line, and right lane, respectively, k l , k c , k r are the slopes of the linear fitting equations for the left lane, inspection center line, and right lane, respectively. l , b c , b r They are the constants of the linear fitting equations of the left lane, inspection center line, and right lane respectively; Step 3, for the left lane y l Lower side and inspection center line y c The upper area is the reverse driving area, and the inspection center line y c The area below and above the right lane is the forward driving area.

9. The road event recognition method based on real-time video stream of unmanned aerial vehicle according to claim 1 is characterized in that: In step S3, the angle between the direction vectors of the vehicle traveling in the left lane and the right lane is calculated in the same coordinate system, and the expression is: θ=arccos(cosθ) In the formula, θ is the vector The angle between is the direction vector of the left lane, is the direction vector of the right lane; The reverse driving of vehicles on the lane is judged based on the angle, including: In the left lane, θ is 0° to 90°, driving in the opposite direction, and θ is 90° to 180°, driving normally; In the right lane, when θ is 0° to 90°, the vehicle is driving normally; when θ is 90° to 180°, the vehicle is driving in the opposite direction.

10. A road event recognition system based on real-time video stream of unmanned aerial vehicle, characterized in that: The system is implemented by the road event recognition method based on the real-time video stream of a drone according to any one of claims 1 to 9, and the system comprises: The vehicle identification module is used to annotate the vehicles in the collected drone images and use the YOLOv8 algorithm to train the accumulated drone vehicle annotation data set; the trained YOLOv8 model is applied to the real-time video collected by the drone for vehicle identification, and the location information of the vehicle in the identified video frame is intercepted; The vehicle driving direction determination module is used to determine the vehicle's driving area and driving direction based on the vehicle movement data in the two drone images that are related to each other in the video; The vehicle reverse traffic judgment module is used to set the direction vector of the inspection center line as the positive direction of the UAV, calculate the angle between the direction vectors of the vehicle in the left lane and the right lane in the same coordinate system, and judge the reverse traffic of the vehicle on the lane based on the angle.

Images