Vehicle illegal parking detection method based on urban high complexity detection scene

Abstract

The application particularly relates to a vehicle illegal parking detection method based on a high-complexity detection scene in a city, which comprises the following steps: collecting a vehicle-mounted video through a task vehicle; detecting a target vehicle in each video frame of the vehicle-mounted video, tracking the target vehicle by combining the motion features and appearance features of the target vehicle, and then outputting a vehicle image of the target vehicle tracked in continuous video frames; calculating the relative distance change between the target vehicle and the task vehicle by combining the dynamic visual ranging and the vehicle image, and judging whether the target vehicle is in a parking state by combining a parking judgment threshold; judging whether the target vehicle is located in a no-parking area according to the real-time position of the task vehicle, and taking the target vehicle in the parking state and located in the no-parking area as an illegal parking vehicle; and performing license plate recognition on the illegal parking vehicle. The application can adapt to the high-loss of vehicle tracking targets and the high dynamics of illegal parking judgment scenes in the high-complexity detection scene in the city, and can simultaneously ensure the accuracy and stability of the illegal parking detection of urban roads.

Description

Technical Field

[0001] This invention relates to the fields of vehicle networking and big data, specifically to a method for detecting illegally parked vehicles in highly complex urban detection scenarios. Background Technology

[0002] With the sustained and rapid development of the economy and society and the continuous improvement of national income levels, the number of motor vehicles in my country is increasing year by year. Currently, more than one million parking lots have been built domestically, but compared to the huge number of cars, parking spaces remain scarce. Urban parking conflicts are becoming increasingly prominent, leading to a more widespread phenomenon of illegal parking, which poses a significant safety hazard to urban traffic. For example, it causes traffic congestion and hinders emergency rescue. Therefore, how to effectively detect illegally parked vehicles and eliminate traffic safety hazards has become one of the urgent problems to be solved in urban development.

[0003] Currently, there are two main methods for detecting illegal parking. The first is through patrols by law enforcement officers, which requires significant manpower and resources, is inefficient, and prone to missed detections. The second is through monitoring with fixed cameras (such as electronic eyes) deployed on major traffic arteries. However, this method generally has blind spots and limited monitoring range, and the deployment and maintenance of city-level monitoring equipment is extremely expensive. Therefore, traditional methods for detecting illegal parking are no longer sufficient to meet the needs of large-scale urban road illegal parking detection.

[0004] Group vehicle perception, as an emerging perception paradigm, can provide an important means for large-scale, low-cost detection of illegal parking on urban roads. On the one hand, cameras universally installed in vehicles can provide a wider and more flexible monitoring method than fixed cameras; on the other hand, using road videos taken by users while driving to detect illegal parking can greatly reduce detection costs. However, when group vehicle perception methods are applied to highly complex urban detection scenarios, the following problems still exist:

[0005] 1) High rate of target loss when tracking vehicles. Unlike video footage captured by traditional fixed cameras (with fixed shooting positions and angles), in-vehicle video footage is very complex: interference from factors such as camera shake, shadows of moving targets, and mutual occlusion between vehicles can all affect the accuracy of vehicle detection, which can easily lead to target tracking failure and result in low accuracy of illegal parking detection.

[0006] 2) The scenarios for illegal parking judgment are highly dynamic. In urban roads, the movement patterns of vehicles themselves and between vehicles are in a highly dynamic state. Therefore, illegal parking analysis needs to consider the movement patterns (such as speed, road changes, etc.) of both the task vehicle and the target vehicle. In addition, illegal parking judgment also depends on the effectiveness of target detection. When the detection effect is poor, the accuracy of illegal parking judgment will also decrease.

[0007] In summary, urban high-complexity detection scenarios are characterized by a high rate of target loss in vehicle tracking and a high degree of dynamism in illegal parking detection, leading to poor accuracy and stability in illegal parking detection. Therefore, designing a method that can adapt to the high rate of target loss in vehicle tracking and the high dynamism of illegal parking detection in urban high-complexity detection scenarios is an urgent technical problem to be solved. Summary of the Invention

[0008] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a vehicle illegal parking detection method based on highly complex urban detection scenarios, which can adapt to the high loss of vehicle tracking targets and the high dynamism of illegal parking judgment scenarios in highly complex urban detection scenarios, thereby ensuring the accuracy and stability of illegal parking detection on urban roads at the same time, and providing a new approach for illegal parking detection on roads based on group vehicle perception.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0010] Vehicle illegal parking detection methods based on highly complex urban detection scenarios include:

[0011] S1: Collect onboard video from the mission vehicle;

[0012] S2: Detect the target vehicle in each frame of the vehicle video, and track it by combining the motion and appearance features of the target vehicle, and then output the vehicle image of the tracked target vehicle in continuous video frames.

[0013] S3: Calculate the relative distance change between the target vehicle and the task vehicle by combining dynamic visual ranging with vehicle images, and determine whether the target vehicle is in a parked state by combining the parking judgment threshold.

[0014] S4: Determine whether the target vehicle is located in a no-parking zone based on the real-time location of the task vehicle, and classify the target vehicle that is parked and located in a no-parking zone as an illegally parked vehicle.

[0015] S5: Perform license plate recognition on illegally parked vehicles to detect illegal parking.

[0016] Preferably, the target vehicle is tracked through the following steps:

[0017] S201: Perform vehicle target detection on each frame of the in-vehicle video and generate a detection box for the target vehicle;

[0018] S202: Predict the vehicle trajectory of the target vehicle and associate the detection box of the target vehicle with the corresponding vehicle trajectory to achieve trajectory tracking of the target vehicle;

[0019] S203: Repeat step S202 until the video frame of the vehicle video ends, and output the vehicle image of the target vehicle tracked in the continuous video frames.

[0020] Preferably, the trajectory tracking of the target vehicle is achieved through the following steps:

[0021] S2021: Create vehicle trajectories for target vehicles detected in the initial video frames of the in-vehicle video;

[0022] S2022: Use Kalman filtering to predict the state vectors of all vehicle trajectories in the next video frame and obtain the prediction boxes;

[0023] S2023: When reading the current video frame, the Hungarian algorithm is used to match the detection box of the target vehicle in the current video frame with the predicted box of the vehicle trajectory in the previous video frame based on the motion and appearance features of the target vehicle. The target vehicle is then associated with the corresponding vehicle trajectory, and the matching result is updated by Kalman filtering.

[0024] S2024: Repeat steps S2022 to S2023 to achieve trajectory tracking of the target vehicle.

[0025] Preferably, vehicle trajectories are divided into deterministic and uncertain states. The vehicle trajectories in the initial video frame and newly appearing vehicle trajectories in subsequent video frames are uncertain states. Uncertain vehicle trajectories become deterministic after their prediction boxes match the detection boxes of the target vehicles a certain number of times. Deterministic vehicle trajectories are deleted after their prediction boxes mismatch a certain number of times.

[0026] Preferably, for a deterministic vehicle trajectory, the predicted bounding box of the previous video frame and the detection bounding box of the target vehicle in the current video frame are concatenated and matched to obtain three results: unmatched vehicle trajectory, unmatched detection bounding box, and matched vehicle trajectory.

[0027] For uncertain vehicle trajectories, first merge them with unmatched vehicle trajectories, then perform IOU matching with unmatched detection boxes, and finally use the obtained IOU result as the basis for the next round of trajectory tracking.

[0028] Preferably, the matching degree between the predicted bounding box of the vehicle trajectory and the detection bounding box of the target vehicle is evaluated by calculating the cost matrix between the predicted bounding box and the detection bounding box of the target vehicle through the motion features and appearance features of the target vehicle.

[0029] Preferably, the cost matrix between the predicted bounding box and the detected bounding box is calculated through the following steps:

[0030] 1) Calculate the motion features between the predicted bounding box and the detected bounding box;

[0031] The formula is described as: d (1)(i,j)=(d j -y i ) T S i -1 (d j -y i );

[0032] In the formula: d (1) (i,j) represents the Mahalanobis distance between the i-th predicted bounding box and the j-th detected bounding box, i.e., the motion feature; d j y represents the state vector of the j-th detection box; i S represents the state vector of the i-th prediction box; i This represents the average covariance matrix between the detection box and the predicted box;

[0033] 2) Calculate the appearance features between the predicted bounding box and the detected bounding box;

[0034] The formula is described as follows:

[0035] In the formula: d (2) (i,j) represents the minimum cosine distance between the i-th predicted bounding box and the j-th detected bounding box, i.e., the appearance feature; r j The r represents the appearance feature vector extracted from the detection box j; k (i) The r represents the appearance feature vectors of the k most recently successfully associated trajectories of the i-th vehicle trajectory; j T r k (i) Indicates cosine similarity;

[0036] 3) Calculate the cost matrix between the predicted bounding box and the detected bounding box;

[0037] The formula is described as: C i,j =λd (1) (i,j)+(1-λ)d (2) (i,j);

[0038] In the formula: C i,j d represents the cost matrix between the i-th predicted bounding box and the j-th detected bounding box; (1) (i,j) represents the Mahalanobis distance; d (2) (i,j) represents the minimum cosine distance; λ represents the hyperparameter set.

[0039] Preferably, the target vehicle is parked by the following steps:

[0040] S301: Calculate the relative distance between the target vehicle and the mission vehicle by combining dynamic visual ranging with vehicle images;

[0041] S302: Convert the relative distance between the target vehicle and the task vehicle into the moving distance of the target vehicle, and then determine whether the target vehicle is in a parked state based on the moving distance of the target vehicle and the parking judgment threshold.

[0042] Preferably, the relative distance between the target vehicle and the mission vehicle is calculated through the following steps:

[0043] 1) With the optical center of the camera as the origin O of the coordinate axis C Establish camera coordinate system O C -X C -Y C -Z C Z C The Y axis is perpendicular to the imaging plane and points in front of the camera. C The axis is perpendicular to the ground plane, X C Perpendicular to Z C and Y C The plane that forms the composition;

[0044] 2) Establish an image coordinate system ouv with the top left corner of the vehicle image as the origin o of the coordinate axis, where u is parallel to the length direction of the vehicle image and v is parallel to the width direction of the vehicle image.

[0045] 3) Establish the following coordinate system transformation formula between the camera coordinate system and the image coordinate system;

[0046]

[0047] In the formula: K represents the intrinsic parameter matrix of the vehicle-mounted camera, f x f y c x c y This represents the parameters in the intrinsic parameter matrix K; let P be a point in real space, and let the coordinates of P in the camera coordinate system be [X...]. C ,Y C Z C ] T P is projected through the optical center onto pixel p with pixel coordinates [u,v] in the image coordinate system;

[0048] 4) Set the coordinates of the target vehicle detection box in the vehicle image to {p i =(u i ,v i Substituting |i=1,...,4} into the coordinate system transformation formula, we obtain the coordinates of the four points of the detection box in the camera coordinate system {P i =(X i ,Y i Z i Let )|i=1,...,4}, and X i Y i and Z iThe following relationship exists:

[0049]

[0050]

[0051] 5) P1, P2, P3, and P4 are the real-world coordinates of the target vehicle in the vehicle image. i (i = 1, 2, 3, 4) Coordinates are determined by Z i This indicates that Z1 = Z2 = Z3 = Z4; then, according to different vehicle types, the corresponding actual height H is set, and the distance Z is obtained by combining the following formula. i As the relative distance between the target vehicle and the mission vehicle;

[0052]

[0053] Preferably, the relative distances D1 and D2 between the target vehicle and the mission vehicle in two onboard video frames are first obtained through dynamic visual ranging; then, the distance S traveled by the mission vehicle in the two onboard video frames is obtained using the GPS data of the mission vehicle and the geodesic distance formula. cam The motion of the target vehicle relative to the mission vehicle is converted into its real-world linear distance S using the following formula. car Finally, a stopping judgment is achieved by setting a stopping judgment threshold s, that is, when the target vehicle has moved a distance S. car When the value is less than the threshold s, the target vehicle is determined to be in a parked state;

[0054] S car =S cam +D2-D1.

[0055] Compared with existing technologies, the vehicle illegal parking detection method based on highly complex urban detection scenarios in this invention has the following advantages:

[0056] This invention first calculates the movement distance changes of target vehicles in the onboard video collected by the task vehicle through multi-target tracking and dynamic visual ranging to determine whether the target vehicle is parked. Then, based on the real-time position of the task vehicle, it determines whether the target vehicle is located in a no-parking zone, classifying parked vehicles located in no-parking zones as illegally parked vehicles. Finally, it performs license plate recognition on illegally parked vehicles to achieve vehicle illegal parking detection. This invention further combines edge-cloud collaboration technology, utilizing road videos captured by users during driving and the powerful computing capabilities of cloud servers to achieve large-scale, low-cost detection of illegal parking on urban roads, greatly ensuring urban road traffic safety. Furthermore, it leverages the advantages of group vehicle perception without requiring additional dedicated equipment, exhibiting greater versatility and effectively enabling accurate vehicle illegal parking detection in highly complex urban detection scenarios.

[0057] This invention detects target vehicles in each frame of an in-vehicle video and tracks them by combining their motion and appearance features. It then outputs vehicle images of the tracked target vehicles across consecutive video frames, enabling cross-video frame tracking and acquisition of target vehicle image information. Furthermore, by comprehensively considering the motion and appearance features of the target vehicle during the vehicle detection and tracking process, it effectively avoids the problem of missed detection caused by mutual occlusion between multiple vehicles. In other words, it can adapt to the high loss rate of vehicle tracking targets in highly complex urban detection scenarios, thereby ensuring the accuracy and robustness of illegal parking detection on urban roads.

[0058] This invention calculates the relative distance change between the target vehicle and the task vehicle by combining dynamic visual ranging with vehicle images and determines whether the target vehicle is in a parked state. This allows the tracking information of the target vehicle (vehicle image and vehicle ID) to be cleverly transformed into the distance change in the real scene (relative to the task vehicle) using dynamic visual ranging. It can capture the motion information of the target vehicle in highly dynamic vehicle scenarios, that is, it can adapt to the high dynamism of illegal parking judgment scenarios in highly complex urban detection scenarios, thereby ensuring the stability of illegal parking detection on urban roads and providing a new approach to illegal parking detection on roads based on group vehicle perception. Attached Figure Description

[0059] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0060] Figure 1 This is a diagram illustrating the algorithm framework for detecting illegal parking on urban roads.

[0061] Figure 2 This is a schematic diagram of an in-vehicle video scene model;

[0062] Figure 3 The impact of different traffic flows on algorithm performance;

[0063] Figure 4 The impact of different vehicle types on algorithm performance;

[0064] Figure 5 The impact of different numbers of illegally parked vehicles on algorithm performance;

[0065] Figure 6 A schematic diagram illustrating the implementation of a prototype system for detecting illegal parking on large-scale urban roads based on edge-cloud collaboration;

[0066] Figure 7 This is a diagram illustrating the detection results of illegally parked vehicles. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0068] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not mean that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0069] The following detailed explanation illustrates the specific implementation methods:

[0070] Example:

[0071] This embodiment discloses a method for detecting illegally parked vehicles in highly complex urban detection scenarios.

[0072] like Figure 1 and Figure 1 As shown, a vehicle illegal parking detection method based on highly complex urban detection scenarios includes:

[0073] S1: Collect onboard video from the mission vehicle;

[0074] S2: Detect the target vehicle in each frame of the vehicle video, and track it by combining the motion and appearance features of the target vehicle, and then output the vehicle image of the tracked target vehicle in continuous video frames.

[0075] S3: Calculate the relative distance change between the target vehicle and the task vehicle by combining dynamic visual ranging with vehicle images, and determine whether the target vehicle is in a parked state by combining the parking judgment threshold.

[0076] S4: Determine whether the target vehicle is located in a no-parking zone based on the real-time location of the task vehicle, and classify the target vehicle that is parked and located in a no-parking zone as an illegally parked vehicle.

[0077] In this embodiment, the real-time location of the mission vehicle can be obtained through the positioning device on the mission vehicle itself or through the user's smartphone on the mission vehicle.

[0078] S5: Perform license plate recognition on illegally parked vehicles to detect illegal parking.

[0079] This invention first calculates the movement distance changes of target vehicles in the onboard video collected by the task vehicle through multi-target tracking and dynamic visual ranging to determine whether the target vehicle is parked. Then, based on the real-time position of the task vehicle, it determines whether the target vehicle is located in a no-parking zone, classifying parked vehicles located in no-parking zones as illegally parked vehicles. Finally, it performs license plate recognition on illegally parked vehicles to achieve vehicle illegal parking detection. This invention further combines edge-cloud collaboration technology, utilizing road videos captured by users during driving and the powerful computing capabilities of cloud servers to achieve large-scale, low-cost detection of illegal parking on urban roads, greatly ensuring urban road traffic safety. Furthermore, it leverages the advantages of group vehicle perception without requiring additional dedicated equipment, exhibiting greater versatility and effectively enabling accurate vehicle illegal parking detection in highly complex urban detection scenarios.

[0080] This invention detects target vehicles in each frame of an in-vehicle video and tracks them by combining their motion and appearance features. It then outputs vehicle images of the tracked target vehicles across consecutive video frames, enabling cross-video frame tracking and acquisition of target vehicle image information. Furthermore, by comprehensively considering the motion and appearance features of the target vehicle during the vehicle detection and tracking process, it effectively avoids the problem of missed detection caused by mutual occlusion between multiple vehicles. In other words, it can adapt to the high loss rate of vehicle tracking targets in highly complex urban detection scenarios, thereby ensuring the accuracy and robustness of illegal parking detection on urban roads.

[0081] This invention calculates the relative distance change between the target vehicle and the task vehicle by combining dynamic visual ranging with vehicle images and determines whether the target vehicle is in a parked state. This allows the tracking information of the target vehicle (vehicle image and vehicle ID) to be cleverly transformed into the distance change in the real scene (relative to the task vehicle) using dynamic visual ranging. It can capture the motion information of the target vehicle in highly dynamic vehicle scenarios, that is, it can adapt to the high dynamism of illegal parking judgment scenarios in highly complex urban detection scenarios, thereby ensuring the stability of illegal parking detection on urban roads and providing a new approach to illegal parking detection on roads based on group vehicle perception.

[0082] Experimental results show that the detection accuracy of the proposed method for illegally parked vehicles is on average 22.2% higher than that of the three existing comparative algorithms, and the detection accuracy is improved by an average of 16.1% and 23.3% for different types and quantities of illegally parked vehicles. Furthermore, this invention further constructs a large-scale urban road illegal parking detection prototype system based on edge-cloud collaboration, verifying the feasibility and practical applicability of the proposed method.

[0083] To better understand the technical solution of this invention, the main mathematical symbols involved in this invention are first given in Table 1.

[0084] Table 1 Explanation of Main Mathematical Symbols

[0085]

[0086] In practical implementation, to determine the vehicle status, multi-target vehicle tracking is required on the in-vehicle video to obtain continuous motion trajectories. However, the scenes in in-vehicle videos are very complex. When vehicles occlude each other or their appearance changes, traditional target detection algorithms (such as SORT) are prone to losing their own IDs, leading to vehicle tracking failure. Therefore, this invention employs a multi-target tracking algorithm based on Deep SORT to track vehicles in in-vehicle videos. This algorithm is a detection-by-detection-based target tracking algorithm, meaning it first performs target detection and then associates the detection results with different tracking trajectories. Because Deep SORT adds cascade matching, it effectively solves the problem of targets being occluded for extended periods, reducing the probability of missed detections of illegal parking events.

[0087] Specifically, the target vehicle is tracked through the following steps:

[0088] S201: Perform vehicle target detection on each frame of the vehicle video using a target detector, generate a detection box for the target vehicle, and assign a corresponding vehicle ID to the target vehicle.

[0089] In this embodiment, the YOLOv4 network model is used as an object detector for vehicle target detection. YOLOv4 is an advanced object detection algorithm that can be used to detect various objects, including vehicles. The following are the steps of how the YOLOv4 model performs vehicle target detection:

[0090] Data preprocessing: The Yolov4 model needs to convert the input data into a set of numbers (i.e., tensors) for computation within the model. For vehicle object detection, the input data can be images or videos. During the preprocessing stage, the model resizes the input data to the same size and performs standardization and normalization.

[0091] Building Feature Maps: In the Yolov4 model, the input data first passes through a series of convolutional and pooling layers, generating multiple feature maps. These feature maps contain image features at different levels, helping the model better understand objects in the image.

[0092] Predicted bounding boxes and categories: In Yolov4, each feature map generates a set of predicted bounding boxes. These boxes are rectangular bounding boxes used to locate objects in the image. Each box also predicts a category to indicate what object is contained within it.

[0093] Redundant Box Filtering: Since each feature map generates a set of predicted boxes, multiple boxes may overlap. To avoid repeatedly detecting the same target, the Yolov4 model uses Non-Maximum Suppression (NMS) to filter redundant boxes. This algorithm calculates a confidence score for each box and selects the box with the highest score as the final detection result.

[0094] Output detection results: Finally, the Yolov4 model outputs the bounding boxes and category information of the detected vehicles. This information can be used in applications such as vehicle tracking and traffic monitoring.

[0095] Specifically, the Yolov4 network model of this invention includes an Input layer, a Backbone layer, a Neck layer, and a Head layer;

[0096] After receiving the input vehicle images, the Input layer expands the vehicle image set using Mosaic data augmentation to enrich the samples and improve training speed.

[0097] In the backbone layer, the CSPDarknet53 structure is used as the backbone network for feature extraction. The input vehicle image is downsampled to extract high-level semantic feature information of the image.

[0098] The Neck layer fuses feature information from feature maps of different sizes, and further fuses the backbone features through a combination of upsampling and downsampling.

[0099] The extracted image features are fed into the Head layer to obtain the target category and bounding box information.

[0100] S202: Predict the vehicle trajectory of the target vehicle and associate the detection box of the target vehicle with the corresponding vehicle trajectory to achieve trajectory tracking of the target vehicle;

[0101] S203: Repeat step S202 until the video frame of the vehicle video ends, and output the vehicle ID and vehicle image of the target vehicle tracked in the continuous video frames.

[0102] This invention first uses a target detector to detect vehicles in each frame of the vehicle-mounted video, generating bounding boxes for the target vehicles. Then, it predicts the vehicle trajectory and associates the bounding boxes with the corresponding trajectories to achieve trajectory tracking. This allows for cross-video frame tracking to acquire image information of the target vehicles by first detecting the targets and then associating the detection results with different tracking trajectories. Furthermore, the invention comprehensively considers the motion and appearance features of the target vehicles during vehicle detection and tracking, thus better avoiding missed detections caused by occlusion between vehicles. This makes it more suitable for the high loss rate of vehicle tracking targets in complex urban detection scenarios. Simultaneously, the Deep SORT method used in this invention incorporates cascaded matching, which effectively solves the problem of target vehicles being occluded for extended periods, reducing the probability of missed detections of illegal parking incidents.

[0103] In the specific implementation process, the trajectory tracking of the target vehicle is achieved through the following steps:

[0104] S2021: Create vehicle trajectories for target vehicles detected in the initial video frames of the in-vehicle video;

[0105] S2022: Predict the state vector of all vehicle trajectories in the next video frame using Kalman filtering (using an 8-dimensional vector). (To characterize) and obtain the prediction box;

[0106] In this embodiment, Kalman filtering is an algorithm for estimating system states, which can be used to predict the state vector of a trajectory. The prediction process of Kalman filtering can be divided into two steps:

[0107] 1) Predicting the state vector

[0108] First, we need to predict the state vector for the next time step based on the current state vector and the state transition matrix. The state vector includes information such as the target's position and velocity, while the state transition matrix describes how the state vector changes over time.

[0109] 2) Covariance matrix of the predicted state vector

[0110] Secondly, we need to predict the covariance matrix of the state vector, which describes the uncertainty of the state vector. The change of the covariance matrix over time can be calculated using the state transition matrix and the process noise covariance matrix.

[0111] Through these two steps, we can obtain the predicted state vector and covariance matrix for the next time step. These predictions can be used to update the state estimate of the Kalman filter and for prediction in the next time step.

[0112] In other preferred embodiments, Kalman filtering can also perform prediction through the following steps:

[0113] Define state variables: First, you need to define the vehicle's state variables, such as position, speed, and acceleration. These variables are usually difficult to measure directly, but can be measured indirectly through sensors or other means.

[0114] Establish a dynamic model: Establish a dynamic model of the vehicle, that is, how the vehicle's state changes over time. Generally, Newton's kinematic equations can be used to establish the vehicle's motion model, including the changing patterns of state variables such as position, velocity, and acceleration.

[0115] Measurement Update: Based on measurement data obtained from sensors and other devices, the vehicle's state variables are updated. Since measurement data may contain errors, Kalman filtering is used to filter the data and reduce the impact of these errors on the state variables.

[0116] Prediction Update: Based on the current state variables and the dynamic model, predict the future state variables of the vehicle. During the prediction process, Kalman filtering is also used to correct the predicted values ​​to improve the accuracy and reliability of the prediction.

[0117] Iterative loop: By continuously repeating the above process and iteratively optimizing the measurement data and prediction results, the accuracy of vehicle trajectory prediction is gradually improved.

[0118] S2023: When reading the current video frame, the Hungarian algorithm is used to match the detection box of the target vehicle in the current video frame with the predicted box of the vehicle trajectory in the previous video frame based on the motion and appearance features of the target vehicle. The target vehicle is then associated with the corresponding vehicle trajectory, and the matching result is updated by Kalman filtering.

[0119] In this embodiment, the Hungarian algorithm is a classic optimal matching algorithm that can be used to match detection boxes with prediction boxes. The specific steps are as follows:

[0120] 1) Calculate the distance matrix between all detection boxes and prediction boxes. Distance can be measured by metrics such as the distance between the center points of two boxes or the area difference.

[0121] 2) Normalize the distance matrix by rows and columns so that the minimum value of each row and column is 0.

[0122] 3) Find the minimum weight in the distance matrix and mark the corresponding row and column as matched.

[0123] 4) For unmatched rows and columns, find the smallest weight among them and mark the corresponding row and column as matched.

[0124] 5) Repeat step 4) until all rows and columns are matched.

[0125] 6) For each matching pair, associate the detection box and the prediction box, and calculate the IoU (Intersection over Union) value between them.

[0126] 7) For each detection box and prediction box, if they are not matched, they are considered to be new targets or targets that have disappeared.

[0127] The Hungarian algorithm can achieve optimal matching between detection boxes and prediction boxes, thereby improving the accuracy and stability of target tracking.

[0128] In other preferred implementations, the Hungarian algorithm first calculates the distance between the predicted bounding box and the detection bounding box. This distance can be measured using different metrics, such as Euclidean distance or intersection-over-union (IoU). Then, these distances are used to construct a bipartite graph, where the predicted bounding boxes and detection bounding boxes form the left and right sets of nodes, respectively. The distance between each predicted bounding box and detection bounding box can be considered as an edge between a left and right node, and a corresponding weight is assigned to each edge to represent the confidence level of their match. Next, the Hungarian algorithm is used to solve for the maximum weighted matching in this bipartite graph, finding the optimal matching scheme between all left and right nodes. In this matching scheme, each predicted bounding box is matched with an optimal detection bounding box, and each detection bounding box is matched with an optimal predicted bounding box.

[0129] S2024: Repeat steps S2022 to S2023 to achieve trajectory tracking of the target vehicle.

[0130] Specifically, vehicle trajectories are divided into deterministic and uncertain states. The vehicle trajectory in the initial video frame and the newly appearing vehicle trajectory in subsequent video frames are uncertain. Uncertain vehicle trajectories become deterministic after their prediction boxes match the detection boxes of the target vehicles a certain number of times (default is 3 times). Deterministic vehicle trajectories are deleted after their prediction boxes mismatch a certain number of times (default is 30 times).

[0131] In the specific implementation process, for a vehicle trajectory in a deterministic state, the predicted bounding box of the previous video frame and the detection bounding box of the target vehicle in the current video frame are concatenated and matched to obtain three results: unmatched vehicle trajectory, unmatched detection bounding box, and matched vehicle trajectory.

[0132] For uncertain vehicle trajectories, they are first merged with unmatched vehicle trajectories, then matched with unmatched detection boxes using IOU (intersection-over-union), and finally the obtained IOU result is used as the basis for the next round of trajectory tracking.

[0133] In this embodiment, because cascaded matching prioritizes matching the detection box with the trajectory with the shorter disappearance time, the vehicle ID assigned to the occluded vehicle remains unchanged when it reappears, reducing the occurrence of vehicle ID jumps.

[0134] This invention first creates vehicle trajectories for target vehicles detected in the initial video frame. Then, it uses Kalman filtering to predict the state vectors of all vehicle trajectories in the next video frame and obtains the predicted bounding boxes. Finally, it uses the Hungarian algorithm to match the detection bounding boxes of target vehicles in the current video frame with the predicted bounding boxes of vehicle trajectories in the previous video frame and associates the target vehicles with the corresponding vehicle trajectories. This effectively achieves target detection first and then associates the detection results with different tracking trajectories, thereby enabling cross-video frame tracking to obtain the image information of target vehicles. Furthermore, it comprehensively considers the motion and appearance features of target vehicles during vehicle detection and tracking, which can better avoid the problem of missed detection caused by mutual occlusion between multiple vehicles. This makes it better suited to the high loss rate of vehicle tracking targets in highly complex urban detection scenarios.

[0135] In practice, the matching degree between the predicted bounding box of the vehicle trajectory and the detection bounding box of the target vehicle is evaluated by calculating the cost matrix between the predicted bounding box and the detection bounding box of the target vehicle through the motion features and appearance features of the target vehicle.

[0136] In this embodiment, the smaller the cost matrix, the higher the matching degree between the predicted box and the detected box.

[0137] The cost matrix between the predicted bounding box and the detected bounding box is calculated using the following steps:

[0138] 1) Calculate the motion features between the predicted bounding box and the detection bounding box; the motion features refer to the Mahalanobis distance between the predicted bounding box of the vehicle trajectory and the detection bounding box of the target vehicle.

[0139] The formula is described as: d (1) (i,j)=(d j -y i ) T S i -1 (d j -y i );

[0140] In the formula: d (1) (i,j) represents the Mahalanobis distance between the i-th predicted bounding box and the j-th detected bounding box, i.e., the motion feature; dj y represents the state vector of the j-th detection box; i S represents the state vector of the i-th prediction box; i This represents the average covariance matrix between the detection box and the predicted box; a threshold is also set for the Mahalanobis distance. To exclude those targets that are not relevant.

[0141] 2) Calculate the appearance features between the predicted bounding box and the detection bounding box; the appearance features refer to the minimum cosine distance between the predicted bounding box of the vehicle trajectory and the detection bounding box of the target vehicle.

[0142] The formula is described as follows:

[0143] In the formula: d (2) (i,j) represents the minimum cosine distance between the i-th predicted bounding box and the j-th detected bounding box, i.e., the appearance feature; r j The r represents the appearance feature vector extracted from the detection box j; k (i) This represents the appearance feature vectors of the k most recently successfully associated vehicle trajectories (trajectory trackers), and a repository is set up for each vehicle trajectory tracker k. Used to store the appearance feature vectors from the previous 100 frames at the current time; r j T r k (i) This represents the cosine similarity, therefore the cosine distance is 1-r. j T r k (i) Similarly, a threshold is also used for the cosine distance part. Used to distinguish whether or not there is a relationship.

[0144] In this embodiment, the ReID feature extraction network (from YE M, SHEN J, LIN G, et al. Deep learning for person re-identification: A survey and outlook) is used to extract appearance features. Since the original DeepSORT algorithm is only suitable for pedestrian identification and tracking, and the images input to the feature extraction network are scaled to 128×64, which does not conform to the aspect ratio of the target vehicle, this invention adjusts the size of the input image to 128×128 to better match vehicle characteristics.

[0145] 3) Calculate the cost matrix between the predicted bounding box and the detected bounding box. On the one hand, Mahalanobis distance provides information about the possible object positions, which is very effective for short-term predictions. On the other hand, cosine distance takes into account appearance information, which is particularly effective for recovering vehicle IDs after long-term occlusion. To combine these two features, this invention performs a weighted operation on both to obtain the cost matrix.

[0146] The formula is described as: C i,j =λd (1) (i,j)+(1-λ)d (2) (i,j);

[0147] In the formula: C i,j d represents the cost matrix between the i-th predicted bounding box and the j-th detected bounding box; (1) (i,j) represents the Mahalanobis distance; d (2) (i,j) represents the minimum cosine distance; λ represents the hyperparameter that can be set, and the influence of each index can be controlled by the parameter λ.

[0148] This invention calculates the cost matrix between the predicted bounding box of the vehicle trajectory and the detection bounding box of the target vehicle by using the motion and appearance features of the target vehicle. This matrix is ​​used to evaluate the matching degree between the predicted bounding box and the detection bounding box. This allows for the comprehensive consideration of the motion and appearance features of the target vehicle during the vehicle detection and tracking process, thereby better avoiding the problem of missed detection caused by mutual occlusion between multiple vehicles. This makes it better suited to the high loss rate of vehicle tracking targets in highly complex urban detection scenarios.

[0149] In the specific implementation process, the following steps are used to determine whether the target vehicle is parked:

[0150] S301: Calculate the relative distance between the target vehicle and the mission vehicle by combining dynamic visual ranging with vehicle images;

[0151] S302: Convert the relative distance between the target vehicle and the task vehicle into the moving distance of the target vehicle, and then determine whether the target vehicle is in a parked state based on the moving distance of the target vehicle and the parking judgment threshold.

[0152] The relative distance between the target vehicle and the mission vehicle is calculated using the following steps:

[0153] 1) Establish the following coordinate system transformation formula between the camera coordinate system and the image coordinate system;

[0154] Photos taken by mainstream smartphones on the market today lack depth information and cannot provide accurate 3D stereo perception or scene reconstruction. Therefore, to obtain the motion information of the target vehicle, dynamic visual ranging is used to transform image information into distance changes in the real scene.

[0155] A model is built for the in-vehicle video scene captured by the user. The optical center of the camera is considered as a sufficiently small aperture, and the internal photosensitive element is considered as the imaging plane. The vertical distance of the optical center from this plane is the focal length. Specifically:

[0156] 1) such as Figure 2 As shown, the origin O of the coordinate axis is the optical center of the camera. C Establish camera coordinate system O C -X C -Y C -Z C Z C The Y axis is perpendicular to the imaging plane and points in front of the camera. C The axis is perpendicular to the ground plane, X C Perpendicular to Z C and Y C The plane that forms the composition;

[0157] 2) such as Figure 2 As shown, with the top left corner of the vehicle image as the origin o of the coordinate axis, an image coordinate system ouv is established, where u is parallel to the length direction of the vehicle image and v is parallel to the width direction of the vehicle image.

[0158] 3) Establish the following coordinate system transformation formula between the camera coordinate system and the image coordinate system;

[0159] Let P be a point in real space, and its coordinates in the camera coordinate system be [X...]. C ,Y C Z C ] T It is projected onto p in the image coordinate system through the optical center, and the corresponding pixel coordinates are [u,v]. Based on the object-image similarity relationship, the transformation relationship is as follows:

[0160]

[0161] In the formula: K represents the intrinsic parameter matrix of the vehicle-mounted camera, f x f y c x c y This represents the parameters in the intrinsic parameter matrix K; let P be a point in real space, and let the coordinates of P in the camera coordinate system be [X...]. C ,Y C Z C ] T P is projected through the optical center onto pixel p with pixel coordinates [u,v] in the image coordinate system;

[0162] 4) Set the coordinates of the target vehicle detection box in the vehicle image to {p i =(u i ,v iSubstituting |i=1,...,4} into the coordinate system transformation formula, we obtain the coordinates of the four points of the detection box in the camera coordinate system {P i =(X i ,Y i Z i Let )|i=1,...,4}, and X i Y i and Z i The following relationship exists:

[0163]

[0164]

[0165] 5) P1, P2, P3, and P4 are the real-world coordinates of the target vehicle in the vehicle image. i (i = 1, 2, 3, 4) Coordinates are determined by Z i This indicates that Z1 = Z2 = Z3 = Z4; then, according to different vehicle types, the corresponding actual height H is set, and the distance Z is obtained by combining the following formula. i As the relative distance between the target vehicle and the mission vehicle;

[0166]

[0167] The following steps are used to determine whether the target vehicle is parked: First, obtain the relative distances D1 and D2 between the target vehicle and the task vehicle in two onboard video frames (e.g., when it appears and disappears) using dynamic visual ranging; then, use the GPS data of the task vehicle (recorded by the user's smartphone) and the geodesic distance formula to obtain the distance S that the task vehicle has moved in the two onboard video frames. cam The motion of the target vehicle relative to the mission vehicle is converted into its real-world linear distance S using the following formula. car Finally, a stopping judgment is achieved by setting a stopping judgment threshold s, that is, when the target vehicle has moved a distance S. car When the value is less than the threshold s, the target vehicle is determined to be in a parked state;

[0168] S car =S cam +D2-D1.

[0169] In this embodiment, the geodesic distance formula is a formula used to calculate the geodesic distance (also known as the great circle distance) between two points on Earth. It represents the shortest distance along the Earth's surface between the two points and is commonly used for navigation and mapmaking. The geodesic distance formula is based on spherical geometry, taking into account that the Earth is approximately spherical. According to this formula, the geodesic distance depends on the longitude and latitude between the two points, as well as the Earth's radius.

[0170] Common geodetic distance formulas include the Haversine formula, the Vincenty formula, and the Spherical Law of Cosines formula. Among them, the Haversine formula is the most commonly used and is suitable for short-distance calculations, while the Vincenty formula is suitable for long-distance calculations and has higher accuracy.

[0171] This invention utilizes a monocular vision model to model the vehicle-mounted shooting scene (establishing a coordinate system transformation formula between the camera coordinate system and the image coordinate system). It can calculate the relative distance change between the target vehicle and the task vehicle by combining dynamic visual ranging with vehicle images and determine whether the target vehicle is in a parked state. In other words, it can cleverly use dynamic visual ranging to transform the tracking information of the target vehicle into the distance change in the real scene, which can better achieve the capture of target vehicle motion information in highly dynamic vehicle-mounted scenes. This makes it better suited to the high dynamism of illegal parking judgment scenarios in highly complex urban detection scenarios.

[0172] In this embodiment, the LPRNet algorithm (from ZHANG W, LU J, ZHANG J, et al. Research on the algorithm of license plate recognition based on MPGAN haze weather) is used to recognize license plate characters in images of illegally parked vehicles. LPRNet is a deep learning-based license plate recognition algorithm that can achieve fast and accurate license plate recognition through a combination of image processing techniques and neural networks. LPRNet consists of lightweight convolutional neural networks and does not use an RNN (Recurrent Neural Network) structure. It extracts image features through a series of convolution and pooling operations. Because this network is fast and has excellent performance in recognizing Chinese license plates, it is suitable for the large-scale, high-efficiency urban illegal parking detection system proposed in this invention.

[0173] Specifically, the LPRNet algorithm first preprocesses the input license plate image using image processing techniques such as edge detection and color space conversion. Then, it uses a convolutional neural network (CNN) to extract and recognize features from the license plate image. This network, composed of multiple convolutional and pooling layers, automatically learns key features of the license plate image, such as numbers, letters, symbols, and the license plate border. Finally, the LPRNet algorithm outputs the recognition results as a character sequence, including the license plate number and region code. Due to the efficiency and accuracy of deep learning algorithms, LPRNet has been widely applied in the field of license plate recognition, such as in intelligent transportation, vehicle management, and security.

[0174] In this invention, the Yolov4 network model is first extended to enable it to detect and locate license plates in images of illegally parked vehicles. The license plate image is then input into the LPRNet backbone network, which consists of Small basic block sub-modules, convolutional layers, pooling layers, etc. After feature extraction by the LPRNet backbone network, a sequence representing the probability of the corresponding license plate characters is obtained. Finally, the output probability sequence is inferred and decoded using a greedy search method to select the character with the highest probability at each position, thus obtaining the license plate character recognition result for the illegally parked vehicle. The Small basic block module uses 1×1 convolutions to reduce the dimensionality of the input image, thereby reducing the number of network parameters. Then, 3×1 and 1×3 convolutions are used for image feature extraction, making the model more lightweight. Finally, the dimensionality is increased to obtain the feature map of the original image.

[0175] Furthermore, since the network output sequence is of variable length, LPRNet introduces the CTC (Connectionist temporal classification) loss function (from GRAVES A, FERNáNDEZ S, GOMEZ F, et al. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks) to enable the model to be trained end-to-end.

[0176] To better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.

[0177] 1. Introduction to Dataset and Experimental Methods

[0178] 1) Introduction to the experimental dataset

[0179] The dataset used in this experiment consisted of self-collected vehicle-mounted road video data records from Chongqing. Each record contained a 30fps video and a corresponding log file (timestamps at 1-second intervals, GPS coordinates, acceleration, and gyroscope data). The dataset covered an area of ​​approximately 11.7 km², with a total travel distance exceeding 34.3 km. A total of 113 vehicles were found to be violating traffic regulations in the video data.

[0180] 2) Introduction to Experimental Methods

[0181] To evaluate the performance of urban road illegal parking detection algorithms suitable for high dynamic range video scenarios, this experiment primarily assesses the algorithm's detection accuracy and recall. The evaluation considers different traffic flows, vehicle types, and numbers of illegally parked vehicles. Secondly, to evaluate the impact of the algorithm's parking judgment threshold on illegal parking detection, this experiment conducts evaluations using the same dataset under different threshold settings.

[0182] Deep network model training. The experimental model was trained using PyTorch, utilizing the PASCAL VOC 2007 and 2012 datasets (from EVERINGHAM M, ESLAMI S, VAN GOOL L, et al. The pascal visual objectclasses challenge: A retrospective) plus 900 supplementary blue-green Chinese license plate images to train the Yolov4 model. The VOC training set contains 1874 images of cars, showing a total of 3267 cars with different appearances. This experiment used the VeRi-776 vehicle re-identification dataset (LIU X, LIU W, MEI T, et al. A deep learning-based approach to progressive vehicle re-identification for urban surveillance) to train the appearance feature extraction model in Deep SORT, which includes images of 776 different cars taken at different times and locations in the city from different angles. For the LPRNet license plate recognition model, this experiment uses weights obtained from training on the Chinese License Plate General Dataset (CCPD) (from XU Z, YANG W, MENG A, et al. Towards end-to-end license plate detection and recognition: A large dataset and baseline).

[0183] To comprehensively evaluate the performance of the proposed illegal parking detection algorithm, this experiment compares it with three other illegal parking vehicle detection algorithms with different performance levels: 1) ParkMaster (from GRASSI G, JAMIESON K, BAHL P, et al. Parkmaster: An in-vehicle, edge-based video analytics service for detecting open parking spaces in urban environments): Based on a self-collected dataset of illegally parked vehicle images, a Viola-Jones cascade classifier is trained to detect illegal parking events in video images; 2) Faster RCNN (from REN S, HE K, GIRSHICK R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks): A typical two-stage neural network object detection algorithm, which has better detection accuracy and robustness than the traditional recognition algorithm used in ParkMaster; 3) SSD (from LIU W, ANGUELOV D, ERHAN D, et al. Ssd: Single shot multibox detector): A typical single-stage object detection algorithm, which is simpler than the two-stage Faster RCNN algorithm model.

[0184] In the experimental evaluation, this experiment adopted two performance evaluation indicators: 1) Precision (P), which is the proportion of correctly predicted positive out of all correctly predicted positive results, as shown in formula (1); 2) Recall (R), which is the proportion of correctly predicted positive out of all actually positive results, as shown in formula (2). Wherein, TP represents a true label value and a true predicted value, FP represents a false label value and a true predicted value, and FN represents a false label value and a false predicted value.

[0185]

[0186]

[0187] 2. Experimental Results

[0188] 2.1 Performance Evaluation of Illegally Parked Vehicle Detection

[0189] First, the impact of different traffic conditions on the performance of the illegal parking detection algorithm is evaluated. Traffic flow conditions are categorized into three types based on traffic volume: open, normal, and congested. For example... Figure 3As shown, experimental results demonstrate that the detection method proposed in this invention outperforms the other three algorithms under all three traffic flow conditions. Specifically, as... Figure 3 As shown in (a), the proposed method for detecting illegally parked vehicles achieves a detection accuracy of 87.1% under normal traffic flow conditions. Compared to this method, the ParkMaster algorithm, Faster R-CNN algorithm, and SSD algorithm show detection accuracies reductions of 23.5%, 22.7%, and 20.5%, respectively. Figure 3 As shown in (b), the recall rate of the proposed illegal parking detection method gradually decreases with increasing road congestion. The recall rates of the proposed method are 79.5%, 76.0%, and 73.5% under empty, normal, and congested road conditions, respectively. Under normal traffic conditions, the recall rates of ParkMaster, Faster RCNN, and SSD algorithms are reduced by 32.0%, 10.5%, and 14.0% compared to the proposed method, respectively. Furthermore, experimental results show that the ParkMaster algorithm has the lowest recall rate because it uses a traditional image detection classifier, while Faster RCNN, SSD, and the proposed method use deep learning detection models, achieving better detection results.

[0190] Next, the detection performance of the illegal parking detection algorithm was evaluated for different vehicle types (including cars, tricycles, and buses). For example... Figure 4 As shown in (a) and (b), the detection method proposed in this invention performs best in detecting illegally parked cars, with an average detection precision and recall of 88.3% and 81.6%, respectively. However, the detection precision for illegally parked buses decreases, reaching only 65.5%. This is because buses are often parked in a row on the roadside, easily causing mutual obstruction. Furthermore, the high similarity in appearance of buses across different cities confuses vehicle tracking in the detection method of this invention, reducing its effectiveness. Compared to the method of this invention, the ParkMaster, Faster RCNN, and SSD algorithms show average detection precision reductions of 23.2%, 10.6%, and 14.4%, respectively, and average recall reductions of 33.1%, 20.3%, and 17.3%, respectively. Moreover, the ParkMaster training dataset does not include tricycles and buses, preventing the algorithm from detecting illegally parked vehicles of these two types. Furthermore, all algorithms showed the best detection performance when detecting illegally parked cars, while their performance dropped significantly when detecting tricycles and buses. This is because the dataset used for model training contained more images of illegally parked cars than tricycles and buses, resulting in poor feature learning for the algorithm models of tricycles and buses.

[0191] Finally, this experiment evaluated the performance of the illegal parking detection algorithm on detecting different numbers of illegally parked vehicles. In the urban road dataset collected for this experiment, the number of illegally parked vehicles on the roadside was generally 2-3; therefore, the algorithm was set to detect three scenarios: fewer than 3, 3 or more, and more than 3 illegally parked vehicles. Figure 5 As shown in (a) and (b), the precision and recall of each algorithm gradually decrease as the number of illegally parked vehicles increases. When the number of illegally parked vehicles on the road is less than 3, the algorithm proposed in this invention exhibits the best performance, with an average detection precision of 84% and a recall of 75.8%. Compared to the method in this invention, the ParkMaster, Faster RCNN, and SSD algorithms show a decrease in average detection precision of 12.5%, 10.6%, and 12.1%, respectively, and a decrease in average recall of 22.0%, 8.0%, and 8.9%, respectively. Experimental results show that the detection performance of the ParkMaster, Faster RCNN, and SSD algorithms rapidly declines when the number of illegally parked vehicles on the roadside increases. The method in this invention introduces cascaded matching in the vehicle tracking part, reducing the occurrence of vehicle ID jumps during tracking and ensuring the accuracy and robustness of the algorithm.

[0192] 2.2 Impact Assessment of Parking Judgment Threshold

[0193] To investigate the optimal illegal parking detection threshold of this invention, experiments were conducted by increasing the threshold from 3 meters to 9 meters, obtaining the changes in precision and recall of the illegal parking detection algorithm under different conditions. As shown in Table 2, as the parking threshold increases, the precision of the method gradually decreases, but the recall decreases slightly after reaching a certain value. When the parking threshold is set to 3m, the highest precision of 88.0% is achieved; when the threshold is set to 7m, the algorithm's recall is the highest, reaching 76.1%. This indicates that setting the parking threshold to 7-8m can, to some extent, balance the detection precision and recall of the algorithm, resulting in better performance of the illegal parking detection method of this invention.

[0194] Table 2 Algorithm performance under different illegal parking judgment threshold settings

[0195]

[0196] 3. Prototype System for Large-Scale Urban Road Illegal Parking Detection Based on Edge-Cloud Collaboration

[0197] To verify the feasibility and practical applicability of the urban road illegal parking detection algorithm designed in this invention, an application was implemented on vehicle-mounted devices, and a large-scale urban road illegal parking detection prototype system was built in the cloud.

[0198] The system framework for monitoring illegal parking on urban roads mainly includes the following steps: 1. Users participating in the sensing process receive illegal parking detection tasks issued by the cloud, mainly including basic information about no-parking zones (such as zone name, location, and length); 2. When a user drives a car to the corresponding task zone, such as... Figure 6 As shown, a smartphone application is used to capture video of the road ahead, simultaneously recording vehicle driving information such as GPS, acceleration, and camera-built-in parameters (obtained via the Android Camera API). The collected data is then uploaded to a cloud server, and an algorithm suitable for high-dynamic-range video scenarios is used to detect illegally parked vehicles in no-parking zones, thereby achieving urban road parking violation monitoring. Specifically, for example... Figure 6 As shown, a large-scale prototype system for detecting illegal parking on urban roads was built on a cloud server. This system can dynamically display the real-time situation of no-parking sections captured by the task vehicle, the vehicle's driving status, and the results of illegal parking detection (geographical location, detection time, image of the illegally parked vehicle, and license plate information).

[0199] Finally, to verify the system's feasibility, illegal parking detection was performed on vehicle-mounted videos of roads with varying traffic volumes. For example... Figure 7 As shown, the first column represents open roads, columns 2 and 3 represent normal roads, the fourth column represents congested roads, and the last row represents the actual illegally parked vehicles in the current image. The remaining rows display the illegal parking detection results of each algorithm. The results show that when vehicles are mutually obstructed, SSD and ParkMaster cannot correctly identify all illegally parked vehicles on the roadside, and may even misidentify normally driving vehicles as illegally parked vehicles. In contrast, the illegal parking detection results of the method of this invention are basically consistent with the actual number of illegally parked vehicles.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A vehicle illegal parking detection method based on a high-complexity detection scene in a city, characterized in that, Comprise: S1: collecting vehicle-mounted video by task vehicle; S2: detecting target vehicle in each frame of video frame of vehicle-mounted video, and tracking target vehicle by combining motion feature and appearance feature of target vehicle, and outputting vehicle image of target vehicle tracked under continuous video frame; S3: calculating relative distance change of target vehicle and task vehicle by dynamic visual ranging combined with vehicle image, and judging whether target vehicle is in parking state by combining parking judgment threshold value; In step S3, target vehicle is judged by parking by the following steps: S301: calculating relative distance of target vehicle and task vehicle by dynamic visual ranging combined with vehicle image; In step S301, relative distance of target vehicle and task vehicle is calculated by the following steps: 1) with the optical center of the camera as the origin of the coordinate axis O C , a camera coordinate system O C is established C -X C -Y C -Z C , the Z C axis points to the front of the camera, the Y C axis is perpendicular to the ground plane, and the X C axis is perpendicular to the plane formed by Z C and Y 2) establishing image coordinate system o-u-v with upper left corner of vehicle image as coordinate axis origin o, u being parallel to length direction of vehicle image, and v being parallel to width direction of vehicle image; 3) establishing coordinate system conversion formula of camera coordinate system and image coordinate system as follows: wherein: K represents an intrinsic matrix of the vehicle-mounted camera, f x , c y , c x , c y represents a parameter in the intrinsic matrix K; let P be a point in a real space, the coordinates of P in a camera coordinate system are [X C , Y C , Z C ] T , and P is projected to a pixel coordinate p of [u, v] in an image coordinate system through a principal point. 4) The coordinates of the target vehicle detection box in the vehicle image {p i i i | i = 1,..., 4} are brought into the coordinate system conversion formula to obtain the coordinates of the four points of the detection box in the camera coordinate system {P i i i i | i = 1,..., 4}, and X i , Y i and Z i exist the following relationship:​​​​​ 5) P1, P2, P3, P4 are the coordinates of the target vehicle in the real world in the vehicle image, P i (i = 1, 2, 3, 4) coordinates are represented by Z i , and Z1= Z2= Z3= Z4; then according to the classification of different vehicle types, the corresponding actual height H is set, and the distance Z i is solved by combining the following formula as the relative distance between the target vehicle and the task vehicle; S302: converting relative distance of target vehicle and task vehicle into moving distance of target vehicle, and judging whether target vehicle is in parking state based on moving distance of target vehicle combined with parking judgment threshold value; S4: judging whether target vehicle is located in no-parking area according to real-time position of task vehicle, and taking target vehicle in parking state and located in no-parking area as illegal parking vehicle; S5: performing license plate recognition on illegal parking vehicle to realize vehicle illegal parking detection. 2.The urban high complexity detection scene based vehicle illegal parking detection method according to claim 1, wherein, In step S2, target vehicle is tracked by the following steps: S201: performing vehicle target detection on each frame of video frame of vehicle-mounted video to generate detection box of target vehicle; S202: predicting vehicle track of target vehicle, and associating detection box of target vehicle to corresponding vehicle track to realize track tracking of target vehicle; S203: repeating step S202 until video frame of vehicle-mounted video ends, and outputting vehicle image of target vehicle tracked under continuous video frame.

3. The method for vehicle illegal parking detection based on urban high complexity detection scene according to claim 1, characterized in that, In step S202, track tracking of target vehicle is realized by the following steps: S2021: creating vehicle track for target vehicle detected in initial video frame of vehicle-mounted video; S2022: predicting state vector of all vehicle tracks in next video frame by Kalman filtering and obtaining prediction box; S2023: reading current video frame, matching detection box of target vehicle in current video frame with prediction box of vehicle track in last video frame based on motion feature and appearance feature of target vehicle by Hungarian algorithm, associating target vehicle to corresponding vehicle track, and updating matching result by Kalman filtering; S2024: repeating steps S2022 to S2023 to realize track tracking of target vehicle.

4. The vehicle illegal parking detection method based on urban high complexity detection scene according to claim 3, characterized in that: In step S2022, vehicle track is divided into determined state and uncertain state, vehicle track of initial video frame and newly appeared vehicle track of subsequent video frame are in uncertain state; vehicle track in uncertain state becomes in determined state after its prediction box continuously matches with detection box of target vehicle for a certain number of times; vehicle track in determined state is deleted after its prediction box continuously mismatches for a certain number of times.

5. The vehicle illegal parking detection method based on urban high complexity detection scene according to claim 4, characterized in that: For the determined state vehicle track, the prediction box of the last video frame and the detection box of the target vehicle in the current video frame are cascaded and matched to obtain three results: an unmatched vehicle track, an unmatched detection box, and a matched vehicle track; For the uncertain state vehicle track, it is first combined with the unmatched vehicle track, and then matched with the unmatched detection box through IOU, and finally the obtained IOU result is taken as the basis for the next round of track tracking. 6.The vehicle illegal parking detection method based on urban high complexity detection scene according to claim 4, characterized in that: In step S2023, the matching degree between the prediction box and the detection box is evaluated by calculating the cost matrix between the prediction box and the detection box of the target vehicle based on the motion feature and the appearance feature of the target vehicle.

7. The vehicle illegal parking detection method based on urban high complexity detection scene according to claim 6, characterized in that: The cost matrix between the prediction box and the detection box is calculated by the following steps: 1) Calculate the motion feature between the prediction box and the detection box; Formula described as: d (1) (i,j) = (d j -y i ) T S i -1 (d j -y i ); wherein: d (1) (i,j) denotes the Mahalanobis distance between the ith prediction box and the jth detection box, i.e. the motion feature; d j denotes the state vector of the jth detection box; y i denotes the state vector of the ith prediction box; S i denotes the average covariance matrix between the detection boxes and the prediction boxes; 2) Calculate the appearance feature between the prediction box and the detection box; The formula is described as: wherein: d (2) (i,j) represents the minimum cosine distance between the i-th prediction box and the j-th detection box, i.e., the appearance feature; r j represents the appearance feature vector extracted by the detection frame j; r k (i) represents the appearance feature vector of the last k successful associations of the i-th vehicle trajectory; r j T r k (i) denotes the cosine similarity; 3) Calculate the cost matrix between the prediction box and the detection box; The formula is described as: C i,j -λd (1) (i,j) + (1 - λ)d (2) (i,j); wherein: C i,j represents the cost matrix between the ith prediction box and the jth detection box; d (1) (i,j) represents the Mahalanobis distance; d (2) (i,j) represents the minimum cosine distance; and λ represents a set hyperparameter. 8.The urban high complexity detection scene based vehicle illegal parking detection method of claim 1, wherein: In step S302, first, the relative distances D1 and D2 of the target vehicle to the task vehicle in two vehicle-mounted video frames are obtained by dynamic visual ranging; then, the moving distances S of the task vehicle in the two vehicle-mounted video frames are obtained by combining the GPS data of the task vehicle with the geodetic distance formula cam , and the motion of the target vehicle relative to the task vehicle is converted into its linear moving distance S in the real world by the following formula car ; finally, the stopping judgment is realized by setting a parking judgment threshold s, that is, when the moving distance S of the target vehicle is less than the threshold s, it is judged that the target vehicle is in a parking state car . S car = S cam + D2 - D1.

Images