Multi-dimensional intelligent identification method and system for dynamic violation behavior of unmanned aerial vehicle road vehicle

Abstract

The application discloses a kind of unmanned aerial vehicle road vehicle dynamic rule violation multi-dimensional intelligent identification method and system.The method comprises: obtaining road vehicle video image sequence;Vehicle confidence feature map detected in previous frame is fused to current frame, and vehicle position and confidence are output after multi-scale feature extraction;The detected vehicle is associated and matched with the existing track twice according to confidence height, and vehicle track is generated;Whether the track continuously occupies the specified lane area is determined according to lane line model, and the labeled rule violation track is returned to tracking step;When the labeled vehicle drives to the near end of field of view, its image is cropped, and if it is identified as social vehicle by classification network, the label is maintained, and if it is special vehicle, the label is removed;Vehicle image and track information maintaining label are extracted and output.The problems of difficult detection of small target at long distance, occlusion leading to track interruption and misjudgment of special vehicle are solved, and the accuracy and tracking continuity of rule violation vehicle identification are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a multi-dimensional intelligent identification method and system for dynamic violations of road vehicles based on UAV vision. Background Technology

[0002] With the continuous growth of car ownership in my country, road traffic congestion has become increasingly serious, and violations such as illegally occupying emergency lanes occur frequently, posing significant road safety hazards. Emergency lanes, known as "lifelines," are dedicated lanes for emergency passage by fire trucks, ambulances, and other special vehicles. Illegally occupying emergency lanes not only violates traffic regulations but can also delay emergency rescues and cause serious consequences.

[0003] Currently, traffic management departments mainly rely on fixed cameras to capture and penalize violations of emergency lane usage. However, existing technical solutions have the following shortcomings:

[0004] First, fixed cameras have a limited field of view, typically covering only vehicles within a few dozen meters in front of the camera. For vehicles at a distance, the detection rate is low due to the small pixel ratio and weak features, making it difficult to achieve long-distance continuous monitoring of vehicles violating regulations.

[0005] Second, in road scenarios, vehicles are severely obscured. Due to low detection confidence, small target vehicles at long distances are often directly discarded during multi-target tracking, leading to premature termination of the trajectory and identity change, making it impossible to continuously track illegal vehicles to the effective evidence collection area.

[0006] Third, there is an extreme imbalance between the number of social vehicles and special vehicles (such as police cars, ambulances, and fire trucks). Traditional classification models tend to predict the sample as the majority class, which can easily misjudge legally occupied special vehicles as illegal vehicles, resulting in wrongful penalties.

[0007] Fourth, existing solutions only utilize single-frame image information for target detection, failing to fully leverage the temporal correlation between adjacent frames in a video sequence, thus limiting detection accuracy.

[0008] Therefore, improving the detection capability of small target vehicles at long distances, ensuring the continuity of tracking illegal vehicles under obstructed conditions, and accurately distinguishing between social vehicles and special vehicles are technical problems that urgently need to be solved in this field. Summary of the Invention

[0009] This invention proposes solutions to the following problems existing in the prior art: fixed cameras have limited field of view and low detection rate of small target vehicles at long distances; vehicle occlusion leads to interruption of trajectory and identity change of low-confidence targets; extreme imbalance between the number of social vehicles and special vehicles leads to high misjudgment rate of special vehicles; and target detection does not make full use of the temporal information between video frames.

[0010] In a first aspect, embodiments of this application provide a multi-dimensional intelligent identification method for dynamic violations of road vehicle behavior by unmanned aerial vehicles, the method comprising:

[0011] Acquire a sequence of road vehicle video images continuously captured by a camera mounted on a drone;

[0012] For the current frame in the video image sequence, the vehicle confidence feature map detected in the previous frame is filtered and upsampled and then fused with the current frame image. After multi-scale feature extraction, the position and confidence of all vehicles in the current frame are output.

[0013] The vehicles detected in each frame are matched with existing trajectories twice according to their confidence level, to generate vehicle trajectories that run through the video image sequence.

[0014] Based on the lane line model, determine whether each vehicle trajectory continuously occupies the designated lane area. Add a violation label to the trajectory of the vehicle that is determined to be in violation, and send the trajectory with the violation label back to the association matching step until the vehicle moves to the near end of the field of vision.

[0015] When the trajectory of a vehicle with a violation label indicates that it has traveled to the near end of the field of vision, the image of the vehicle is cropped from the current frame using the vehicle's position information in the current frame. After being identified by a classification network, if it is identified as a public vehicle, the violation label is maintained; if it is identified as a special vehicle, the violation label is removed.

[0016] Extract and output the images and trajectory information corresponding to the vehicles that maintain the violation label from the video image sequence.

[0017] Secondly, embodiments of this application provide a multi-dimensional intelligent recognition system for dynamic violations of road vehicle behavior by unmanned aerial vehicles, applied to the method described in the first aspect, the system comprising:

[0018] The image acquisition module is used to acquire a sequence of road vehicle video images continuously captured by the camera mounted on the drone;

[0019] The vehicle detection module is used to take the vehicle confidence feature map detected in the previous frame in the current frame of the video image sequence, filter and upsample it, fuse it with the current frame image, and output the position and confidence of all vehicles in the current frame after multi-scale feature extraction.

[0020] The multi-target tracking module is used to match the vehicles detected in each frame with existing trajectories twice according to their confidence level, to generate vehicle trajectories that run through the video image sequence.

[0021] The violation determination and tag management module is used to determine whether each vehicle's trajectory continuously occupies the designated lane area based on the lane line model, attach a violation tag to the trajectory of the vehicle determined to be in violation, and send the trajectory with the violation tag back to the multi-target tracking module until the vehicle moves to the near end of the field of vision.

[0022] The near-end classification and confirmation module is used to crop the image of a vehicle with a violation label from the current frame when the trajectory of the vehicle indicates that it has traveled to the near end of the field of vision. The image is then identified by the classification network. If the vehicle is identified as a public vehicle, the violation label is maintained; if it is identified as a special vehicle, the violation label is removed.

[0023] The output module is used to extract and output the image and trajectory information corresponding to the vehicles that maintain the violation label from the video image sequence.

[0024] Thirdly, embodiments of this application provide an electronic device, including:

[0025] processor;

[0026] Memory used to store processor-executable instructions;

[0027] The processor is configured to implement the multi-dimensional intelligent identification method for dynamic violations of road vehicles by unmanned aerial vehicles as described in the first aspect when executing the instructions.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the multi-dimensional intelligent identification method for dynamic violations of road vehicle behavior by unmanned aerial vehicles as described in the first aspect.

[0029] This invention improves the detection rate of small target vehicles at long distances by fusing inter-frame confidence features and using the detection results of the previous frame to assist the detection of the current frame. Through secondary trajectory association matching with confidence levels, low-confidence detection boxes are matched a second time, improving the problems of trajectory interruption and identity jump under occlusion. A closed loop of detection-tracking-judgment-backhaul is formed by transmitting violation labels, ensuring that violating vehicles are continuously tracked to the near end. Near-end triggered pruning and a classification network trained with FocalLoss and resampling strategies accurately distinguish between social vehicles and special vehicles, avoiding false penalties for legally occupied vehicles. Compared with existing technologies, this invention has the advantages of high detection accuracy, good tracking continuity, and high classification accuracy. Attached Figure Description

[0030] Figure 1 This is a schematic flowchart of a multi-dimensional intelligent identification method for dynamic violations of road vehicles by unmanned aerial vehicles provided in an embodiment of this application.

[0031] Figure 2This is an architecture diagram of a multi-dimensional intelligent recognition system for dynamic violations of road vehicles by unmanned aerial vehicles provided in an embodiment of this application.

[0032] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0034] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0035] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Example 1

[0037] Figure 1 This is a schematic flowchart illustrating a multi-dimensional intelligent identification method for dynamic road vehicle violations by an unmanned aerial vehicle (UAV) according to an embodiment of this application. Figure 1 As shown, a multi-dimensional intelligent identification method for dynamic road vehicle violations using unmanned aerial vehicles includes:

[0038] S1. Acquire a sequence of road vehicle video images continuously captured by the camera mounted on the drone. This continuous sequence of road vehicle video images, acquired via the drone's camera, provides the raw data source for subsequent vehicle detection, tracking, and violation identification.

[0039] Specifically, in this embodiment, a DJI Matrice M300 series drone is used, equipped with a Zenmuse H20T high-definition zoom camera. The drone is hovered approximately 50 meters above and to the side of the road, with the camera fixedly mounted facing the road at a downward angle of 30 degrees. After the drone is activated, the camera continuously captures video images of road vehicles at a frame rate of 25 frames per second. Each frame in the captured video image sequence has an original resolution of 1920×1080 pixels, clearly showing the road surface, white dashed lane markings, yellow solid emergency lane markings, and various vehicles in motion. The captured video stream is transmitted in real-time to the ground workstation via the drone's built-in 5G communication module, serving as input data for subsequent vehicle detection, tracking, and violation identification. Upon receiving the video stream, the ground workstation sequentially performs the subsequent processing steps for each frame.

[0040] S2. For the current frame in the video image sequence, the vehicle confidence feature map detected in the previous frame is filtered and upsampled, then fused with the current frame image. After multi-scale feature extraction, the positions and confidence scores of all vehicles in the current frame are output. The vehicle confidence feature map detected in the previous frame is fused with the current frame image, and after multi-scale feature extraction, the position coordinates and confidence scores of all vehicles in the current frame are output.

[0041] Specifically, in this embodiment, the step of fusing the vehicle confidence feature map detected in the previous frame with the current frame image after filtering and upsampling includes:

[0042] S2.1. The multi-channel confidence feature map output by the target detection network in the previous frame is maximized along the channel dimension to generate a single-channel confidence feature map. In this embodiment, CenterNet is used as the target detection network. For the previous frame image in the video sequence, the confidence feature map output by this network is in multi-channel form, with the number of channels equal to the total number of detected categories (in this embodiment, vehicles are a single category, so the number of channels is 1, but for general illustration, it is still processed as multi-channel). A multi-channel confidence feature map refers to a feature map where each spatial location corresponds to multiple channels, and each channel represents the confidence score of that location belonging to a certain category. To simplify subsequent processing, the multi-channel feature map is maximized along the channel dimension; that is, for each spatial location in the feature map, the maximum value is selected from the confidence scores of all channels as the final confidence score for that location, thus generating a single-channel confidence feature map. In this single-channel feature map, the value of each pixel represents the probability that a vehicle exists at the corresponding location in the previous frame image.

[0043] S2.2 Filter low-confidence responses in the single-channel confidence feature map using a set threshold, setting feature points below the threshold to zero and retaining salient target heatmaps above the threshold. Compare each pixel value in the single-channel confidence feature map generated in the previous step with a pre-set threshold. In this embodiment, the threshold is set to 0.3. Low-confidence responses refer to feature points with confidence scores below the threshold; these points typically correspond to background areas or detection noise in the image. For feature points with confidence scores below 0.3, their values ​​are directly set to zero; for feature points with confidence scores above or equal to 0.3, their original values ​​are retained. After this step, a heatmap is obtained that retains only high-confidence salient targets, where non-zero areas indicate the possible locations of vehicles in the previous frame.

[0044] S2.3. Upsample the filtered single-channel confidence feature map to the same size as the current frame image using bilinear interpolation. Since the feature map size output by the object detection network is usually smaller than the original input image size (in this embodiment, the feature map size is one-quarter of the input image size), it cannot be directly fused with the current frame image. Therefore, bilinear interpolation is used to upsample the filtered single-channel confidence feature map generated in the second step. Bilinear interpolation is an image scaling algorithm that determines the value of a target pixel by calculating the weighted average of the values ​​of its four neighboring pixels, effectively maintaining the spatial continuity of the heatmap. After upsampling, the size of the heatmap is enlarged to be exactly the same as the original image of the current frame.

[0045] S2.4. The upsampled feature map and the current frame image are mapped to the same feature space through convolution operations and then added bit-by-bit. The upsampled heatmap and the original image of the current frame are input into two convolutional layers with identical structures and independent parameters for processing. Each convolutional layer contains a 7×7 kernel with a stride of 1, padding of 3, and 16 output channels. Bit-by-bit addition means that the two feature maps with the same size and number of channels after convolution mapping are directly added at each spatial location and each channel to obtain the fused result for that location and channel. After bit-by-bit addition, a fused feature map with the same size as the current frame and 16 channels is generated. This feature map contains both the original image information of the current frame and the prior information of the vehicle positions detected in the previous frame. This fused feature map is then fed into the subsequent multi-scale feature extraction network to output the positions and confidence scores of all vehicles in the current frame.

[0046] In this embodiment, the complete process of inter-frame feature fusion can be represented by the following mathematical form:

[0047] ,

[0048] in, The feature map after fusion in frame t has a size of H×W×16; The convolution mapping function uses a 7×7 convolution kernel with a stride of 1 and 16 output channels. The original image of frame t has dimensions H×W×3; This represents the bitwise addition operator; This is a bilinear interpolation upsampling function that upsamples the input to H×W; This is a threshold filtering function with a threshold θ = 0.3, which sets elements below the threshold to zero. This is an operation to retrieve the maximum value along the channel dimension; This is the confidence feature map of the c-th channel output by the target detection network in the (t-1)-th frame; For channel indexing, , H represents the category set; H is the image height, which is 1080 pixels in this embodiment; W is the image width, which is 1920 pixels in this embodiment.

[0049] For the current frame image First, it is processed through a convolution mapping function. Mapped to a 16-dimensional feature space. Simultaneously, the multi-channel confidence feature map output from the previous frame's object detection network is extracted. The confidence score of a single channel is obtained by taking the maximum value along the channel dimension, and then subjected to a threshold function. Low-confidence responses are filtered out, and salient target heatmaps above a threshold of 0.3 are retained. These heatmaps are then upsampled to the same spatial size as the current frame using bilinear interpolation, and finally processed by the same convolutional mapping function. After mapping, the feature map is added bit by bit to the current frame feature map to obtain the fused feature map.

[0050] Before filtering low-confidence responses in the single-channel confidence feature map using a set threshold, the method further includes data augmentation training of the single-channel confidence feature map. This embodiment performs data augmentation processing on the standard heatmap during the model training phase, as detailed below.

[0051] First, a standard heatmap is generated based on the actual annotation information of vehicles in the training samples. Specifically, for each labeled vehicle in each training image, a Gaussian response region is generated using a two-dimensional Gaussian kernel function, centered on the vehicle's center point. The Gaussian response regions of all vehicles are superimposed to form the standard heatmap of the image. In this standard heatmap, the value of each pixel represents the actual probability of a vehicle existing at that location, with the highest value at the center point and gradually decreasing to zero towards the outside.

[0052] After the standard heatmap is generated but before it is input into the network for training, the following three enhancement processes are applied to the standard heatmap to simulate detection errors that may occur in actual prediction:

[0053] The first step is to simulate target mislocalization. For a marked vehicle target in a standard heatmap, when rendering it onto the heatmap, the true center point coordinates are not used directly. Instead, a random offset is superimposed on the true center point coordinates. In this embodiment, this random offset is generated independently in the horizontal and vertical axes, with a value ranging from -16 to +16 pixels. Simultaneously, the center point after the superimposed offset is limited to remain within the original image range. After this enhancement process, the center of the Gaussian response region in the heatmap will deviate from the true center of the vehicle, simulating the deviation between the target position predicted by the network and the true position in actual detection.

[0054] The second step simulates background misidentification as a target. False peaks are randomly generated in the target-free regions of the standard heatmap. Target-free regions refer to areas in the heatmap where the original values ​​are zero or close to zero; these regions correspond to the background in the original image. In this embodiment, several locations are randomly selected across the entire heatmap, and a false Gaussian peak is generated at each selected location in the same manner as the Gaussian response generated at that location. The center coordinates of this false peak are randomly generated, and the Gaussian kernel radius is consistent with the kernel radius of the real target. After this enhancement process, a target response that was not originally present appears in the heatmap, simulating the false detection situation where the background is misidentified as a vehicle in actual detection.

[0055] The third step simulates the missed detection of small targets. The response region corresponding to the target in the standard heatmap is randomly deleted with a probability positively correlated with the target size. Specifically, for each labeled vehicle in the training image, the area of ​​its detection box is calculated, which is the width multiplied by the height. The larger the area value, the more pixels the target occupies in the image, indicating a large target; the smaller the area value, the farther and smaller the target, indicating a small target. In this embodiment, the deletion probability is negatively correlated with the target area; that is, the smaller the target area, the higher the probability of deletion. In practice, a random number between zero and one is first generated. If this random number is less than the deletion probability of the target, the entire Gaussian response region of the target in the standard heatmap is set to zero. After this enhancement process, the thermal responses corresponding to some small targets are completely removed, simulating the situation where distant small targets are missed in actual detection.

[0056] During training, the three enhancement processes described above can be used individually or in any combination. Before each training image is input into the network, the application of each enhancement process is randomly determined according to a preset probability. The enhanced heatmap serves as the training label for the feature input head in the inter-frame feature fusion module, guiding the network to learn how to accurately detect vehicles even in the presence of detection errors.

[0057] S3. The vehicles detected in each frame are divided into two groups according to their confidence level, and then matched with existing trajectories twice to generate a continuous vehicle trajectory throughout the video image sequence.

[0058] Specifically, in this embodiment, the step of associating and matching the detected vehicles in each frame with existing trajectories twice, according to their confidence levels, to generate vehicle trajectories throughout the video image sequence, specifically includes:

[0059] S3.1. Set high-confidence and low-confidence thresholds to divide the detection results of each frame into a high-confidence detection box set and a low-confidence detection box set according to their confidence scores. In this embodiment, the target detection network outputs several vehicle detection boxes for each frame of image. Each detection box is accompanied by a confidence score, which is between 0 and 1, indicating the degree of confidence that a vehicle exists within the detection box. Two thresholds are preset: the high-confidence threshold is set to 0.5, and the low-confidence threshold is set to 0.1. For each detection box output in the current frame, its confidence score is compared with these two thresholds. If the confidence score is greater than or equal to 0.5, the detection box is assigned to the high-confidence detection box set; if the confidence score is between 0.1 (inclusive) and 0.5 (exclusive), the detection box is assigned to the low-confidence detection box set; if the confidence score is less than 0.1, the detection box is discarded as background and does not participate in any subsequent trajectory association matching.

[0060] As an alternative to the fixed threshold scheme described above, this embodiment can also employ the following dynamic thresholding method. This method adaptively calculates the threshold based on the detection results of each frame of the image, eliminating the need for manually setting a fixed threshold. The calculation formula is as follows:

[0061] ,

[0062] ,

[0063] ,

[0064] ,

[0065] in, Let t be the set of high-confidence detection boxes; Let t be the set of low-confidence detection boxes; For the first Each detection box contains coordinate and size information; For the first The confidence score of each detection box, with a value ranging from 0 to 1; Let be the mean confidence score of all detection boxes in frame t; Let be the standard deviation of the confidence scores of all detection boxes in frame t; t represents the total number of detection boxes in frame t.

[0066] For each frame of the image, first calculate the mean confidence score of all detection boxes in that frame. and standard deviation Then, the mean plus standard deviation is used as the high-confidence threshold, and the mean minus standard deviation is used as the low-confidence threshold. Detection boxes with a confidence level greater than or equal to the mean plus standard deviation are assigned to the high-confidence set; detection boxes with a confidence level between the mean minus standard deviation and the mean plus standard deviation are assigned to the low-confidence set; and detection boxes with a confidence level lower than the mean minus standard deviation are discarded as background. When using a dynamic thresholding scheme, there is no need to set fixed high-confidence thresholds of 0.5 and low-confidence thresholds of 0.1; the threshold for each frame is determined by the statistical distribution of the detection results within that frame. In actual implementation, either a fixed thresholding scheme or a dynamic thresholding scheme can be selected depending on the application scenario.

[0067] S3.2 Perform the first trajectory association. Use the inverse of the intersection-union ratio between the detection box and the trajectory prediction box as the risk cost matrix to match the set of high-confidence detection boxes with all existing trajectories. The successfully matched trajectories are updated according to the detection results. Unmatched detection boxes are added to the set of remaining detection boxes, and unmatched trajectories are added to the set of remaining trajectories.

[0068] In this embodiment, a Kalman filter is first used to predict the position of each existing trajectory in the current frame, resulting in a trajectory prediction box. The Intersection over Union (IoU) ratio is calculated by dividing the area of ​​the overlapping region of two bounding boxes by the total area of ​​their merged region; its value ranges from 0 to 1, with a higher value indicating a greater degree of overlap. This embodiment uses the reciprocal of the IoU ratio as the risk cost of matching; a smaller IoU ratio corresponds to a larger reciprocal, indicating a higher matching cost, and vice versa. Specifically, for each bounding box in the high-confidence detection box set and for each trajectory prediction box corresponding to all existing trajectories, the IoU ratio is calculated, and then 1 is divided by the IoU ratio to obtain the risk cost. The risk costs of all bounding boxes and all trajectory prediction boxes together constitute a risk cost matrix. Then, the Hungarian algorithm is used to solve this risk cost matrix to find the matching result that minimizes the total matching cost. For successfully matched bounding boxes and trajectories, the position information of the bounding box is used to update the Kalman filter of the corresponding trajectory, thus correcting the trajectory's state. For high-confidence detection boxes that do not match any trajectory, they are added to the remaining detection box set for further processing. For trajectories that do not match any high-confidence detection boxes, they are added to the remaining trajectory set to participate in the next round of matching.

[0069] To more accurately measure the matching degree between the detection box and the trajectory prediction box, this embodiment proposes a weighted intersection-union ratio (CIU), which is defined as follows:

[0070] ,

[0071] ,

[0072] in, The weighted intersection-union ratio (IoU) between the i-th detection box and the j-th trajectory prediction box; For the i-th detection box, the coordinates of its center point are... ,width ,high definition; For the j-th trajectory prediction box, the coordinates of the center point are... ,width ,high definition; This represents the area of ​​the overlapping region between the detection box and the prediction box. This represents the total area of ​​the merged region between the detection box and the prediction box. It is an exponential function with the natural constant e as its base; The distance between the center of the detection box and the center of the prediction box is Euclidean distance; This is the distance attenuation factor, which is set to 50 pixels in this embodiment; This is the size similarity weighting coefficient, which is set to 0.5 in this embodiment; The smaller of the detection box width and the predicted box width; The larger of the detection box width and the predicted box width is used.

[0073] Weighted Intersection over Union (IoU) introduces two adjustment factors into the traditional IoU: a Gaussian decay factor for center point distance, which decreases exponentially as the center distance between the detection box and the predicted box increases, reducing the likelihood of long-distance matching; and a width similarity factor, which increases as the widths of the detection box and the predicted box become closer, thus increasing the weight of size matching. Weighted IoU integrates spatial overlap, center distance, and size similarity dimensions, providing a more accurate measure of the matching degree between the detection box and the trajectory prediction box compared to traditional IoU.

[0074] In practical implementation, to avoid excessively large values ​​in the cost matrix due to excessively small weighted intersection-union ratios (ECRs) (for example, when the detection box and the prediction box hardly overlap, the weighted ECR approaches 0, and its reciprocal approaches infinity), this embodiment performs numerical pruning on the weighted ECR, limiting its value range to [0.1, 1]. That is, when the calculated weighted ECR is less than 0.1, it is set to 0.1. After pruning, the value range of elements in the risk cost matrix is ​​limited to [1, 10], preventing a few excessively large cost values ​​from dominating the matching results of the Hungarian algorithm and improving matching stability.

[0075] Taking into account multiple matching factors, the comprehensive matching cost matrix defined in this embodiment is as follows:

[0076] ,

[0077] in, The comprehensive matching cost between the i-th detection box and the j-th trajectory prediction box; It is a very small positive number, with a value of 0.0001, used to prevent the denominator from being zero; The high similarity weighting coefficient is set to 0.3 in this embodiment; For the first The height of each detection frame; For the first The height of each detection frame; The smaller of the detection box height and the predicted box height; The larger of the detection box height and the predicted box height; The weighted intersection-union ratio (IUU) threshold is set to 0.2 in this embodiment; A value of positive infinity indicates a refusal to match.

[0078] The comprehensive matching cost consists of three parts: the reciprocal of the weighted intersection-union ratio (the smaller the value, the lower the matching cost), a penalty term for height similarity (the greater the height difference, the greater the penalty), and an infinite rejection term when the weighted intersection-union ratio is below a threshold. This cost matrix comprehensively considers spatial overlap, center distance, size similarity, and height consistency, and can more robustly handle matching problems involving occlusion and distant small targets.

[0079] S3.3 Perform the second trajectory association, matching the low-confidence detection box set with the trajectories in the remaining trajectory set; the successfully matched trajectories are updated according to the low-confidence detection boxes, the low-confidence detection boxes that are still not matched are deleted as background, and the trajectories that are still not matched are added to the lost trajectory set.

[0080] In this embodiment, after the first trajectory association is completed, the trajectories in the remaining trajectory set are associated with the detection boxes in the low-confidence detection box set for a second trajectory association. The matching method also uses the inverse of the intersection-union ratio (IUGR) as the risk cost matrix and solves it using the Hungarian algorithm, with the specific calculation method being the same as the first trajectory association. For successfully matched low-confidence detection boxes and remaining trajectories, the position information of the low-confidence detection boxes is used to update the Kalman filter of the corresponding trajectory. It is worth noting here that even if the confidence of the detection box is low (between 0.1 and 0.5), as long as its IUGR with the trajectory prediction box meets the matching condition, it is still considered a reliable observation of the corresponding vehicle and used to update the trajectory state. For low-confidence detection boxes that fail to match, they are directly deleted as background false detections and do not participate in any subsequent processing. For remaining trajectories that fail to match, they are moved to the lost trajectory set. Trajectories in the lost trajectory set will undergo additional management in subsequent frames; for example, if no detection box can be matched within several consecutive frames, the trajectory will be finally deleted.

[0081] In this embodiment, the Hungarian algorithm is used to match detection boxes with trajectories during multi-target tracking. The Hungarian algorithm is a combinatorial optimization algorithm used to solve for maximum or minimum cost matching in a bipartite graph. In this embodiment, the set of detection boxes is considered as one side of the bipartite graph, and the set of trajectories is considered as the other side. The inverse of the intersection-union ratio (IUU) between the detection boxes and trajectories is used as the weight of the connecting edge. The Hungarian algorithm iteratively searches for the set of matching results that minimizes the total weight from all possible matching combinations, i.e., the detection box-trajectory correspondence with the lowest overall matching cost.

[0082] For the first trajectory association, the set of high-confidence detection boxes and all existing trajectories are used as two sides of a bipartite graph. After constructing a risk cost matrix, it is input into the Hungarian algorithm, which outputs the optimal matching result between the high-confidence detection boxes and the trajectories. For the second trajectory association, the set of low-confidence detection boxes and the set of remaining trajectories that did not match after the first association are used as two sides of a bipartite graph. Similarly, the Hungarian algorithm is used to output the optimal matching result between the low-confidence detection boxes and the remaining trajectories.

[0083] This embodiment employs a Kalman filter to predict and update the trajectory state. The Kalman filter is a recursive state estimation algorithm suitable for linear systems. It can predict the state at the next moment based on the system's historical state and correct the prediction based on observations. For each trajectory, the Kalman filter maintains a state vector, which includes the x-coordinate and y-coordinate of the detection box center point, the detection box width, the detection box height, and the rate of change of these parameters in pixel space. Before processing each frame of image, the Kalman filter predicts the position of the vehicle corresponding to the trajectory in the current frame based on the trajectory state of the previous frame, generating a trajectory prediction box. When the trajectory successfully matches a detection box, the actual position of that detection box is used as an observation to update the Kalman filter's state vector, enabling the filter to more accurately predict the trajectory's position in the next frame.

[0084] To more accurately predict the vehicle's motion state during acceleration, deceleration, and lane changes, this embodiment employs a 10-dimensional state vector Kalman filter, with the following prediction and update formulas:

[0085] State prediction:

[0086] ,

[0087] ,

[0088] Kalman gain calculation:

[0089] ,

[0090] Status Update:

[0091] ,

[0092] ,

[0093] State vector definition:

[0094] ,

[0095] in, This represents the state vector of the k-th frame predicted based on the (k-1)-th frame. This is the updated state vector after the (k-1)th frame; Let be the state transition matrix for the k-th frame, with dimensions 10×10; This is the control input matrix for the k-th frame; This is the control vector for the k-th frame, containing acceleration information; The covariance matrix for the predicted state vector has a dimension of 10×10; This is the updated covariance matrix for the (k-1)th frame; Let be the system noise covariance matrix of the k-th frame; Let K be the Kalman gain matrix for the k-th frame; Let be the observation matrix of the k-th frame, with dimensions 4×10; Let be the observation noise covariance matrix of the k-th frame; Let k be the observation vector of the k-th frame. ; It is a 10-dimensional identity matrix; The coordinates of the center point of the detection box in the k-th frame; Let be the width and height of the detection box in the k-th frame; The rate of change of the coordinates of the center point (first derivative); The rate of change of width and height (first derivative); The acceleration (second derivative) is the change in the coordinates of the center point.

[0096] This embodiment uses a 10-dimensional state vector Kalman filter, which adds an acceleration term compared to the traditional 8-dimensional state vector. This allows for better prediction of the nonlinear motion of vehicles during acceleration, deceleration, and lane changes. (State transition matrix) A uniformly accelerated motion model is adopted. Once the trajectory successfully matches the detection box, the detection box parameters are used as observation values. Update the state vector.

[0097] This embodiment also includes a management mechanism for the lost trajectory set. The lost trajectory set stores trajectories that have not matched any detection boxes after the second trajectory association. For each trajectory in the lost trajectory set, the system records the number of consecutive unmatched frames. This embodiment sets the survival frame count for lost trajectories to 30 frames. If a trajectory fails to match any detection box within 30 consecutive frames, it is determined that the vehicle has left the camera's field of view or has been completely obscured and cannot be recovered. The system permanently deletes the trajectory from the trajectory set and releases the corresponding storage resources. If, during the 30-frame period, the trajectory re-matches a detection box (regardless of whether the detection box belongs to the high-confidence set or the low-confidence set), it is moved back from the lost trajectory set to the normal trajectory set, and its consecutive unmatched frame count counter is reset.

[0098] This embodiment also includes a new trajectory initialization mechanism. After the first trajectory association is completed, for high-confidence detection boxes in the remaining detection box set that still do not match any trajectory, it is further determined whether their confidence score is higher than the tracking threshold. In this embodiment, the tracking threshold is set to 0.4. If the confidence score of the high-confidence detection box is higher than 0.4, it is determined that the detection box corresponds to a new vehicle entering the camera's field of view for the first time. The system uses the position information of the detection box as the initial state, creates a new Kalman filter, generates a new trajectory for the vehicle, and adds the trajectory to the trajectory set to participate in the tracking of subsequent frames. If the confidence score of the high-confidence detection box is lower than or equal to 0.4, it is not initialized and is discarded as a false detection or low-quality detection box.

[0099] S4. Based on the lane line model, determine whether each vehicle's trajectory continuously occupies the designated lane area. Add a violation label to the trajectory of a vehicle deemed to be in violation, and return the tagged trajectory to the association and matching step until the vehicle reaches the near end of the field of vision. Based on the pre-established lane line model, determine whether each vehicle's trajectory continuously occupies the designated lane area. Add a violation label to the trajectory of a vehicle deemed to be in violation, and return the tagged trajectory to the tracking step, continuously tracking until the vehicle reaches the near end of the field of vision.

[0100] Specifically, in this embodiment, the step of determining whether each vehicle trajectory continuously occupies the designated lane area based on the lane line model, attaching a violation label to the trajectory of the vehicle determined to be in violation, and sending the trajectory with the violation label back to the association matching step until the vehicle moves to the near end of the field of vision, specifically includes:

[0101] S4.1. A road lane line model is pre-constructed, comprising a defined lane space range represented by a polygonal region. In this embodiment, a road lane line model is pre-constructed for road scenes monitored by the UAV. Specifically, the operator manually marks the boundary position of the defined lane (i.e., the emergency lane) in the first frame image captured by the UAV. The defined lane space range refers to the road surface area occupied by the emergency lane. Since the emergency lane is usually elongated, its boundary in the image is enclosed by two lane lines on the left and right and two endpoints at the front and rear. In this embodiment, this space range is represented as a quadrilateral polygonal region, which is formed by connecting the coordinates of its four vertices sequentially to form a closed geometric shape. All pixels within this polygonal region belong to the emergency lane space range. In practical applications, since the UAV camera angle is fixed, the polygonal region remains in the same position in each frame of the video sequence, without the need for recalibration for each frame.

[0102] S4.2 For each detected vehicle target, calculate the intersection-union ratio (IUGR) or overlap area between its detection frame and the polygonal region of the designated lane. If the IUGR exceeds a set threshold, the vehicle is determined to be illegally occupying the designated lane, and a violation label is attached to it. In this embodiment, for each vehicle target detected in step two, the four boundary coordinates of its detection frame are obtained. The IUGR is the area of ​​the overlapping portion between the detection frame and the polygonal region of the designated lane, divided by the total area of ​​the merged portion, and its value is between 0 and 1. The overlap area is the area of ​​the intersection portion between the detection frame and the polygonal region of the designated lane, not divided by the total merged area, and is an absolute value. This embodiment uses the overlap area as the judgment index, and sets the overlap area threshold to 50% of the area of ​​the detection frame itself. The specific judgment process is as follows: First, calculate the overlap area between the detection box and the polygonal area of ​​the emergency lane; then calculate the proportion of this overlap area to the total area of ​​the detection box itself; if this proportion is greater than or equal to 50%, it is determined that the vehicle is occupying the emergency lane, and the degree of occupation has reached the violation standard. At this time, the system adds a violation label to the trajectory corresponding to the vehicle; if the proportion is less than 50%, it is determined that the vehicle is not illegally occupying the emergency lane, and no label is added. Adding a violation label essentially involves adding a marker field to the trajectory's data structure and setting the value of this field to "violation" for subsequent steps to identify and process.

[0103] To avoid misjudgments caused by momentary occlusion or detection fluctuations, this embodiment adopts a violation detection method based on cumulative judgment within a time window, the mathematical expression of which is as follows:

[0104] ,

[0105] ,

[0106] in, This is an indicator function for whether the i-th vehicle in the t-th frame violates a rule, and its value is either 0 or 1; This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise; Let i be the detection bounding box for the i-th vehicle; The emergency lane is a polygonal area; This refers to the area where the detection frame overlaps with the emergency lane area; The total area of ​​the detection frame itself; The threshold for the area occupied by a single frame is set to 0.5 (50%) in this embodiment. This is an indicator function for whether the k-th trajectory ultimately violates the rules. This is the length of the time window, in frames; in this embodiment, it is set to 10 frames. This is the starting frame number of the time window; The threshold for the violation rate within the time window is set to 0.7 (70%) in this embodiment. First, the overlap area between the vehicle detection frame and the polygonal region of the emergency lane is calculated as the ratio of the total area of ​​the detection frame itself. If this ratio reaches 50% or more, the vehicle is determined to be illegally occupying the emergency lane in the current frame. To avoid instantaneous false detections, a cumulative time window determination is used: within a consecutive 10-frame time window, if the proportion of violation frames reaches 70% or more, the vehicle is ultimately determined to be in violation, and a violation label is added to it.

[0107] S4.3. The trajectory of the vehicle with the attached violation tag is sent back to the association matching step until the vehicle is in the near field of vision.

[0108] In this embodiment, once a violation label is attached to a trajectory, the label is transmitted back to the association matching stage in step three along with the trajectory. The specific implementation of this transmission is as follows: In the trajectory management module of step three, an attribute list is maintained for each trajectory, with the violation label as a Boolean attribute in this list. When step four determines that a vehicle has violated the rules and attaches a label, the attribute list of that trajectory is updated. Step three reads the attribute list of each trajectory during trajectory association matching in subsequent frames. For trajectories with violation labels, the association matching step employs a special processing strategy: even if the vehicle corresponding to the trajectory has a low detection box confidence in a frame due to occlusion or excessive distance, the trajectory will not be easily moved into the lost trajectory set, but will be forcibly retained and continuously participate in the association matching of each frame. Step three continues to track the violating vehicle until the vehicle's trajectory information indicates that it has traveled to the near-field region of the drone camera's view. The so-called near-field region refers to an area close to the drone where the vehicle occupies a large pixel area in the image, typically determined by the vehicle detection box size being greater than a set threshold. When the vehicle moves to the near end of the field of vision, the image information of the vehicle is clear enough to proceed to step five for classification and recognition.

[0109] The step of transmitting the vehicle trajectory with the attached violation tag back to the association matching step until the vehicle moves to the near end of the field of vision specifically includes:

[0110] S4.3.1 In the association matching step, trajectories with additional violation labels are given higher priority matching rights than ordinary trajectories.

[0111] In this embodiment, when the association matching step matches all existing trajectories, the trajectories are prioritized based on whether they have violation labels. Specifically, normal trajectories refer to the trajectories of normal vehicles without violation labels. Priority matching means that before performing Hungarian algorithm matching, trajectories with violation labels are matched in a different order than normal trajectories. In this embodiment, all trajectories with violation labels are arranged at the front of the matching queue, and these trajectories are matched with detection boxes first. When constructing the risk cost matrix, the matching relationship between violation-labeled trajectories and detection boxes is processed first to ensure that violation-labeled trajectories have the greatest chance of matching with a suitable detection box. Normal trajectories are matched only after the violation-labeled trajectories have been matched. This priority matching mechanism ensures that even when there are insufficient high-confidence detection boxes, violating vehicles can still obtain detection box resources first, thereby maintaining the continuity of their trajectories.

[0112] To prioritize the matching of violation trajectories, this embodiment defines the following priority score calculation formula:

[0113] ,

[0114] ,

[0115] in, Let be the matching priority score for the j-th trajectory; This is the basic matching priority for the j-th trajectory; the value for a normal trajectory is 1. The priority enhancement coefficient for violation trajectories is set to 2 in this embodiment; This is an indicator function for whether the j-th trajectory violates the rules; it takes a value of 1 if it violates the rules, and 0 otherwise. Let j be the time interval between the j-th trajectory and the last successful match; This is the time decay constant, which is set to 5 frames in this embodiment; The current moment; Let be the time when the j-th trajectory last successfully matched a detection box. During trajectory association matching, a matching priority score is calculated for each trajectory. The priority score of a violation trajectory is multiplied by an enhancement factor, which decays exponentially with the time interval since the last match; that is, the priority of violation trajectories that have not matched for a long time gradually decreases. The priority score of normal trajectories remains at the base value of 1. During matching, trajectories are processed in descending order of priority score to ensure that violation trajectories receive priority matching rights.

[0116] S4.3.2 When the vehicle corresponding to the trajectory has a detection confidence level lower than the high confidence threshold in a certain frame, it is still forcibly included in the candidate set for the first trajectory association matching. In this embodiment, the conventional first trajectory association matching only matches detection boxes in the high confidence detection box set (confidence level greater than or equal to 0.5). For trajectories with additional violation labels, even if the confidence level of the vehicle detection box corresponding to the trajectory in the current frame is lower than the high confidence threshold of 0.5, the system still forcibly includes the trajectory in the candidate set for the first trajectory association matching for evaluation. In specific implementation, for trajectories with violation labels, the system does not require them to match a high confidence detection box, but retains the trajectory in the candidate set for the first trajectory association matching, allowing it to attempt to match with detection boxes in the high confidence detection box set. If the trajectory can successfully match a high confidence detection box, the trajectory is updated according to the normal process; if it cannot match any high confidence detection box, the trajectory enters the remaining trajectory set and continues to participate in the second trajectory association matching. This mechanism effectively avoids situations where violation tracks are prematurely excluded due to a temporary decrease in detection confidence (such as when a vehicle is partially obscured).

[0117] S4.3.3 When the trajectory does not match any detection box in multiple consecutive frames, it will not be moved into the lost trajectory set. Instead, the trajectory will remain active for the next set number of frames until the vehicle is detected again or it is confirmed that it has left the field of view.

[0118] In this embodiment, for ordinary tracks, if they fail to match any detection box within three consecutive frames, they are immediately moved to the lost track set and deleted if they still fail to match within the next 30 frames. However, for tracks with attached violation labels, the system adopts a more lenient management strategy. Specifically, when a track with a violation label fails to match any detection box in multiple consecutive frames, the system does not immediately move it to the lost track set but keeps the track active. In this embodiment, the survival frame count for violation tracks is set to 60 frames, twice that of ordinary tracks. During this 60-frame grace period, the track remains in the active track set and participates in the first and second track association matching attempts in each frame. If, in any frame within the 60-frame period, the track successfully matches a detection box (regardless of whether the detection box belongs to the high-confidence set or the low-confidence set), its consecutive unmatched frame count counter is reset to zero, and normal tracking is restored. If the trajectory still fails to match any detection box after the 60-frame grace period, it is determined that the vehicle has left the drone camera's field of view and can no longer be recaptured. At this point, the system removes the trajectory from the active trajectory set and outputs its complete historical information as a violation record. This extended survival time mechanism ensures that even if a violating vehicle reappears after a prolonged period of obstruction or after entering a distant area, its violation tag and trajectory identity are retained, preventing identity changes and loss of violation records.

[0119] S5. When the trajectory of a vehicle with a violation label indicates that it has reached the near end of the field of vision, its image is cropped from the current frame using the vehicle's position information. This image is then processed by a classification network. If the vehicle is identified as a civilian vehicle, the violation label is retained; otherwise, it is removed.

[0120] Specifically, in this embodiment, S5 includes:

[0121] S5.1 Construct a lightweight image classification network to distinguish between social vehicles and special vehicles that can legally occupy designated lanes.

[0122] In this embodiment, a lightweight image classification network is constructed for binary classification of cropped vehicle images. A lightweight image classification network refers to a neural network model with a relatively simple structure, few parameters, and low computational complexity, suitable for real-time operation on UAV-borne edge computing devices. This embodiment uses MobileNetV3 as the basic architecture of the classification network. This network uses depthwise separable convolutions instead of ordinary convolutions, significantly reducing computation; it also introduces a channel attention mechanism, enabling the network to automatically focus on feature channels more important for the classification task. The input to the network is the cropped vehicle image, uniformly scaled to 128×128 pixels. The output layer of the network contains two neurons, corresponding to the probability scores of civilian vehicles and special vehicles, respectively. Civilian vehicles refer to ordinary civilian vehicles that are not authorized to use emergency lanes; special vehicles refer to vehicles that are legally authorized to use emergency lanes, including police cars, ambulances, fire trucks, and emergency rescue vehicles. The sum of the two probability scores is 1, and the network uses the higher probability score as the final classification result.

[0123] S5.2 When training the classification network, Focal Loss is used as the classification loss function, and a resampling strategy is adopted, that is, social vehicle samples and special vehicle samples are stored separately, and samples are randomly drawn from the two sample sets without replacement in each training batch according to a set ratio; when the special vehicle samples are exhausted in advance, difficult samples are selected from the special vehicle samples that were misclassified in the historical training to continue training.

[0124] In this embodiment, the training of the classification network faces the problem of class imbalance, where the number of social vehicle samples far exceeds the number of special vehicle samples. To address this issue, the following two strategies are employed.

[0125] First, Focal Loss is used as the classification loss function. Focal Loss is an improved cross-entropy loss function, the core idea of ​​which is to reduce the weight of easily classified samples and increase the weight of difficult-to-classify samples in the loss. In this embodiment, special vehicle samples are difficult to classify due to their small number and diverse appearances; social vehicle samples are numerous and have obvious features, making them easy to classify. Focal Loss introduces a modulation factor. When a sample is correctly classified and the confidence level is high, this modulation factor significantly reduces the loss value; when a sample is misclassified or the confidence level is low, the modulation factor keeps the loss value at a high level. This makes the network pay more attention to the classification performance of special vehicle samples during training, thereby alleviating the problem of majority bias caused by class imbalance.

[0126] To further enhance the classification network's ability to distinguish special vehicles, this embodiment proposes an improved FocalLoss, defined as follows:

[0127] ,

[0128] ,

[0129] in, This is the improved Focal Loss value; The total number of samples in the training batch is 64 in this embodiment; For the nth sample, the true class Category weight coefficients; The true category of the nth sample is 0 (social vehicles) or 1 (special vehicles). This represents the model's predicted probability of the true class of the nth sample. As a focusing parameter, it is set to 2 in this embodiment; This is the weight coefficient for the regularization term, which is set to 0.1 in this embodiment; This is a regularization function that suppresses over-prediction of non-true categories; Predict the probability that the nth sample belongs to the mth class for the model; For category indexing, Indicates categories other than the true category; To find the maximum value function, comparing it with 0 ensures that it is non-negative; The tolerance threshold for non-target categories is 0.3 in this embodiment.

[0130] The improved Focal Loss adds a regularization term to the traditional Focal Loss. This regularization term penalizes samples with excessively high predicted probabilities for non-true categories; specifically, it imposes an additional penalty when the model's predicted probabilities for other categories exceed a tolerance threshold δ=0.3. This forces the model to focus more on distinguishing between social vehicles and special vehicles, reducing confusion between the two categories. This loss function is used for backpropagation optimization when training the classification network.

[0131] To dynamically adapt to changes in the difficulty of identifying special vehicles during model training, this embodiment adopts the following adaptive resampling ratio calculation formula:

[0132] ,

[0133] ,

[0134] in, The sampling ratio of special vehicle samples in the training epoch; The minimum sampling ratio for special vehicles is set to 0.1 in this embodiment; The maximum sampling ratio for special vehicles is set to 0.5 in this embodiment; The initial sampling ratio is set to 0.25 in this embodiment; The adjustment coefficient is set to 2 in this embodiment; The pseudo-error rate for the epoch reflects the difficulty of classifying special vehicles. The target pseudo-error rate is set to 0.3 in this embodiment; To verify the number of special vehicle samples in the collection; This represents the probability that the model predicts the k-th special vehicle sample in the validation set as positive. During the classification network training process, after each training epoch, the pseudo-error rate for special vehicles is calculated on the validation set. This value reflects the difficulty the current model faces in identifying special vehicles. If the pseudo-error rate is higher than the target value of 0.3, it indicates that special vehicle classification is still difficult, and the sampling ratio of special vehicles in the next training cycle needs to be increased; otherwise, the sampling ratio should be decreased. The sampling ratio is dynamically adjusted to ensure that the network is always trained on the most challenging sample distribution.

[0135] Second, a resampling strategy is employed. Social vehicle samples and special vehicle samples are stored in two separate folders. In each training batch, sampling is performed at a ratio of three to one for social vehicles and special vehicles. Specifically, the total number of samples in each batch is set to 64 images, with 48 randomly selected without replacement from the social vehicle sample set and 16 randomly selected without replacement from the special vehicle sample set. Once all samples in the special vehicle sample set have been sampled once, ordinary special vehicle samples are not repeatedly sampled. Instead, difficult-to-classify samples are selected from the misclassified special vehicle samples recorded during historical training for further training. Difficult-to-classify samples refer to samples that the network mispredicted or had low prediction confidence in previous training cycles. The system maintains a queue of difficult-to-classify samples. Whenever a special vehicle sample is misclassified, the sample and its confidence score are added to the queue and sorted from low to high confidence. When the normal special vehicle samples are exhausted, the difficult-to-classify samples with the lowest confidence are preferentially selected from the queue for subsequent training batches. In this way, the network can repeatedly learn from special vehicle samples that are most easily misjudged, continuously improving the accuracy of identifying special vehicles.

[0136] S5.3. Input the vehicle body image cropped from the current frame when the vehicle with the violation label drives to the near end of the field of vision into the classification network. If the classification result is a social vehicle, the violation label of the vehicle is maintained. If the classification result is a special vehicle, the violation label of the vehicle is removed.

[0137] In this embodiment, when a vehicle with a violation tag, tracked collaboratively in steps three and four, moves to the near end of the drone's field of view, the image cropping and classification process is triggered. "Near end of the field of view" refers to a vehicle occupying a certain threshold pixel area in the image. In this embodiment, a vehicle is considered to have moved to the near end of the field of view when the area of ​​its detection box is greater than 40,000 pixels (approximately 200×200 pixels). This threshold can be calibrated and adjusted according to the drone's flight altitude, camera focal length, and actual scene requirements: the lower the flight altitude and the longer the focal length, the higher the threshold; conversely, the higher the altitude and the longer the focal length, the lower the threshold. In actual deployment, operators can collect several frames of images of the near-end vehicle and calculate the average area of ​​its detection box as the judgment threshold for that scene.

[0138] At this point, using the vehicle's bounding box coordinates in the current frame output from step two, the vehicle image is cropped from the original frame and scaled to a uniform size of 128×128 pixels. The cropped and scaled image is then input into the MobileNetV3 classification network constructed in S5.1. The network outputs two probability values, representing the probability that the vehicle belongs to a civilian vehicle and the probability that it belongs to a special vehicle, respectively. If the probability of a civilian vehicle is greater than the probability of a special vehicle, the classification result is determined to be a civilian vehicle, the system maintains the vehicle's original violation label, and the vehicle is ultimately confirmed as a civilian vehicle illegally occupying the emergency lane. If the probability of a special vehicle is greater than or equal to the probability of a civilian vehicle, the classification result is determined to be a special vehicle, the system removes the violation label from the vehicle, that is, the violation label field in the trajectory data structure is reset to "no violation". After the label is removed, the vehicle is no longer considered a violation vehicle and will not be output to the final result. This mechanism ensures that special vehicles such as fire trucks and ambulances, which legally have the right to use emergency lanes, are not misjudged as violation vehicles, avoiding invalid enforcement and wrongful penalties.

[0139] S6. Extract and output the image and trajectory information corresponding to the vehicles whose violation labels are maintained from the video image sequence. In this embodiment, after step five completes the classification and recognition, the violation labels of some vehicles are maintained, and the violation labels of some vehicles are removed. For the vehicles whose violation labels are ultimately maintained, the system executes the following information extraction and output process.

[0140] First, image information is extracted. The system extracts all relevant frames from the video image sequence, from the moment the violation is first detected until the moment it is finally confirmed as a violating vehicle. Specifically, for each trajectory maintaining a violation label, the system records the start and end frames of the trajectory's lifecycle. Each frame within that time period is extracted from the video image sequence, and based on the coordinates of the detection box corresponding to the vehicle in each frame, the vehicle area is cropped from the original image to form an image sequence of the vehicle's driving process. Simultaneously, several frames containing the original complete image of the vehicle are also retained to demonstrate the relative positional relationship between the vehicle and the lane lines, serving as corroborating evidence of the violation.

[0141] Second, the system extracts trajectory information. From the trajectory data structure of the violating vehicle, the system extracts the following information: the sequence of coordinates of the center point of the detection box in each frame, used to reconstruct the vehicle's driving path; the sequence of width and height of the detection box in each frame, used to reflect the size changes of the vehicle in the image; the time when the vehicle is first identified as violating the rules and the corresponding frame number; the total number of frames or the total duration of the vehicle continuously occupying the emergency lane; the time when the vehicle reaches the near end of the field of view and completes the classification confirmation and the corresponding frame number; the vehicle's violation label maintenance status and the final confirmation time.

[0142] Third, information output. The system organizes the extracted image and trajectory information to form a complete violation record. Specific output includes: a continuous motion trajectory map of the violating vehicle, showing its path on the emergency lane as lines connecting trajectory points; a keyframe image sequence of the violating vehicle, showing the process of the vehicle occupying the emergency lane as thumbnails or video clips; a final confirmed cropped image of the vehicle body, used to display the vehicle's appearance and license plate information; a violation timestamp, including the start time, duration, and confirmation time of the violation; and vehicle type information, i.e., the identification result confirming it as a civilian vehicle. This information, along with the image files, is saved locally in a structured data format (such as JSON or XML) or uploaded to the ground monitoring platform via the communication module. Supervisory personnel can review the violation record through the platform as a basis for enforcement.

[0143] Fourth, real-time early warning output. For real-time monitoring scenarios, the system immediately generates early warning information upon confirming a violating vehicle. This information includes the vehicle's real-time location coordinates, its position in the image, the type of violation (occupation of the emergency lane), and a cropped image of the vehicle. This warning information is pushed in real-time to the ground monitoring terminal via an onboard edge computing device on the drone, enabling monitoring personnel to promptly grasp the occurrence of violations and, if necessary, remotely communicate with the violator or notify nearby law enforcement personnel for handling.

[0144] Example 2

[0145] like Figure 2As shown in the diagram, this application provides an architecture diagram of a multi-dimensional intelligent recognition system for dynamic road vehicle violations by unmanned aerial vehicles (UAVs), which is applied to the multi-dimensional intelligent recognition method for dynamic road vehicle violations by UAVs as described in Embodiment 1, including:

[0146] Image acquisition module 210 is used to acquire a sequence of road vehicle video images continuously captured by the camera mounted on the UAV.

[0147] The vehicle detection module 220 is used to fuse the vehicle confidence feature map detected in the previous frame with the current frame image after filtering and upsampling, and output the position and confidence of all vehicles in the current frame after multi-scale feature extraction.

[0148] The multi-target tracking module 230 is used to associate and match the vehicles detected in each frame with existing trajectories in two steps according to their confidence levels, thereby generating vehicle trajectories that run through the video image sequence.

[0149] The violation determination and label management module 240 is used to determine whether each vehicle trajectory continuously occupies the designated lane area based on the lane line model, attach a violation label to the trajectory of the vehicle determined to be in violation, and send the trajectory with the violation label back to the multi-target tracking module until the vehicle moves to the near end of the field of vision.

[0150] The near-end classification confirmation module 250 is used to crop the image of a vehicle with a violation label from the current frame when the trajectory of the vehicle indicates that it has traveled to the near end of the field of vision. The image is then identified by the classification network. If the vehicle is identified as a public vehicle, the violation label is maintained; if it is identified as a special vehicle, the violation label is removed.

[0151] The output module 260 is used to extract and output the image and trajectory information corresponding to the vehicle that maintains the violation label from the video image sequence.

[0152] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0153] In this embodiment of the application, memory 300 is used to store executable instructions of processor 301, which, when configured to execute instructions, implements the method as described in the first aspect.

[0154] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0155] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0156] It should be noted that a portion of the electronic device described in the above embodiments can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0157] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0158] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0159] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0160] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A multi-dimensional intelligent identification method for dynamic road vehicle violations by unmanned aerial vehicles, characterized in that, Includes the following steps: Acquire a sequence of road vehicle video images continuously captured by a camera mounted on a drone; For the current frame in the video image sequence, the vehicle confidence feature map detected in the previous frame is filtered and upsampled and then fused with the current frame image. After multi-scale feature extraction, the position and confidence of all vehicles in the current frame are output. The vehicles detected in each frame are matched with existing trajectories twice according to their confidence level, to generate vehicle trajectories that run through the video image sequence. Based on the lane line model, it is determined whether each vehicle's trajectory continuously occupies the designated lane area. A violation label is added to the trajectory of vehicles deemed to be in violation, and the tagged trajectory is sent back to the association and matching step until the vehicle reaches the near end of the field of vision. Specifically, this includes: A road lane line model is pre-constructed, the road lane line model including a defined lane space range represented by polygonal regions; For each detected vehicle target, calculate the intersection-union ratio or overlap area between its detection frame and the polygonal region of the specified lane. If it exceeds a set threshold, determine that the vehicle is illegally occupying the specified lane and attach a violation label to it. The vehicle trajectory with the attached violation tag is sent back to the matching step until the vehicle is in the near field of vision; The step of transmitting the vehicle trajectory with the attached violation tag back to the association matching step until the vehicle moves to the near end of the field of vision specifically includes: In the association matching step, trajectories with additional violation tags are given higher priority matching rights than ordinary trajectories; If the vehicle corresponding to the trajectory has a detection confidence score lower than the high confidence threshold in a certain frame, it will still be forcibly included in the candidate set for the first trajectory association matching. If the trajectory does not match any detection box in multiple consecutive frames, it will not be moved into the lost trajectory set. Instead, the trajectory will remain active for a set number of subsequent frames until the vehicle is detected again or it is confirmed that it has left the field of view. When the trajectory of a vehicle with a violation label indicates that it has traveled to the near end of the field of vision, the image of the vehicle is cropped from the current frame using the vehicle's position information in the current frame. After being identified by a classification network, if it is identified as a public vehicle, the violation label is maintained; if it is identified as a special vehicle, the violation label is removed. Extract and output the images and trajectory information corresponding to the vehicles that maintain the violation label from the video image sequence.

2. The method according to claim 1, characterized in that, The acquisition of a sequence of road vehicle video images continuously collected by a camera mounted on a drone specifically includes: the camera mounted on the drone is installed at a fixed angle facing the road and continuously collects road vehicle video images at a frame rate of not less than 15 frames per second; each frame in the video image sequence includes the road surface, lane markings, and moving vehicles.

3. The method according to claim 1, characterized in that, The process of fusing the vehicle confidence feature map detected in the previous frame with the current frame image after filtering and upsampling specifically includes: Take the maximum value of the multi-channel confidence feature map output by the target detection network in the previous frame according to the channel dimension to generate a single-channel confidence feature map; Low confidence responses in the single-channel confidence feature map are filtered out by setting a threshold, feature points below the threshold are set to zero, and significant target heatmaps above the threshold are retained. The filtered single-channel confidence feature map is upsampled to the same size as the current frame image using bilinear interpolation. The upsampled feature map and the current frame image are mapped to the same feature space through convolution operations and then added bit by bit.

4. The method according to claim 3, characterized in that, Before filtering low-confidence responses in the single-channel confidence feature map with a set threshold, the method further includes data augmentation training of the single-channel confidence feature map: during the model training phase, the following augmentation processes are applied to the standard heatmap generated based on real labeled data to simulate detection errors in actual prediction. The enhancement process includes: Random offsets are superimposed on the coordinates of the target's center point to simulate target mislocation; In the heat map, false peaks are randomly generated in areas without targets to simulate background and misidentify them as targets. The response region corresponding to the target in the heatmap is randomly deleted with a probability positively correlated with the target size to simulate the missed detection of small targets.

5. The method according to claim 1, characterized in that, The step of associating and matching the detected vehicles in each frame with existing trajectories twice, according to their confidence levels, to generate vehicle trajectories throughout the video image sequence, specifically includes: Set high confidence thresholds and low confidence thresholds, and divide the detection results of each frame into a set of high confidence detection boxes and a set of low confidence detection boxes according to the confidence score; The first trajectory association is performed, using the inverse of the intersection-union ratio between the detection box and the trajectory prediction box as the risk cost matrix. The set of high-confidence detection boxes is matched with all existing trajectories. The successfully matched trajectories are updated according to the detection results, and the unmatched detection boxes are added to the set of remaining detection boxes and the unmatched trajectories are added to the set of remaining trajectories. A second trajectory association is performed, matching the low-confidence detection box set with the trajectories in the remaining trajectory set; the successfully matched trajectories are updated according to the low-confidence detection boxes, the low-confidence detection boxes that are still not matched are deleted as background, and the trajectories that are still not matched are added to the lost trajectory set.

6. The method according to claim 5, characterized in that, The matching of the high-confidence detection box set with all existing trajectories, and the matching of the low-confidence detection box set with the trajectories in the remaining trajectory set, are both implemented using the Hungarian algorithm; the trajectory is updated according to the detection results, specifically by using a Kalman filter to predict and update the state of the corresponding trajectory based on the successfully matched detection boxes; The method further includes: if a trajectory in the lost trajectory set fails to be rematched within a set number of consecutive frames, it is deleted from the trajectory set; and high-confidence detection boxes in the remaining detection box set that are still unmatched and have a confidence level higher than the tracking threshold are initialized as new trajectories.

7. The method according to claim 1, characterized in that, When the trajectory of a vehicle with a violation label indicates that it has reached the near end of the field of vision, its image is cropped from the current frame using the vehicle's position information. After classification by a network, if it is identified as a civilian vehicle, the violation label is maintained; if it is identified as a special vehicle, the violation label is removed. Specifically, this includes: Construct a lightweight image classification network to distinguish between social vehicles and special vehicles that can legally occupy designated lanes; When training the classification network, Focal Loss is used as the classification loss function, and a resampling strategy is adopted, that is, social vehicle samples and special vehicle samples are stored separately, and each training batch randomly draws samples from the two sample sets without replacement according to a set ratio; when the special vehicle samples are exhausted in advance, difficult samples are selected from the special vehicle samples that were misclassified in the historical training to continue training. When a vehicle with a violation label moves to the near end of the field of vision, the vehicle body image cropped from the current frame is input into the classification network. If the classification result is a social vehicle, the violation label of the vehicle is maintained; if the classification result is a special vehicle, the violation label of the vehicle is removed.

8. A multi-dimensional intelligent recognition system for dynamic violations of road vehicle behavior by unmanned aerial vehicles, applied to the method described in any one of claims 1 to 7, characterized in that, The system includes: The image acquisition module is used to acquire a sequence of road vehicle video images continuously captured by the camera mounted on the drone; The vehicle detection module is used to take the vehicle confidence feature map detected in the previous frame in the current frame of the video image sequence, filter and upsample it, fuse it with the current frame image, and output the position and confidence of all vehicles in the current frame after multi-scale feature extraction. The multi-target tracking module is used to match the vehicles detected in each frame with existing trajectories twice according to their confidence level, to generate vehicle trajectories that run through the video image sequence. The violation determination and tag management module is used to determine whether each vehicle's trajectory continuously occupies the designated lane area based on the lane line model, attach a violation tag to the trajectory of the vehicle determined to be in violation, and send the trajectory with the violation tag back to the multi-target tracking module until the vehicle moves to the near end of the field of vision. The near-end classification and confirmation module is used to crop the image of a vehicle with a violation label from the current frame when the trajectory of the vehicle indicates that it has traveled to the near end of the field of vision. The image is then identified by the classification network. If the vehicle is identified as a public vehicle, the violation label is maintained; if it is identified as a special vehicle, the violation label is removed. The output module is used to extract and output the image and trajectory information corresponding to the vehicles that maintain the violation label from the video image sequence.

Images