A method, device and medium for detecting illegally parked vehicles in intelligent inspection scenarios
By constructing a lightweight vehicle license plate detection model and a license plate recognition model, and combining multi-source sensors for multi-modal fusion, the real-time and accuracy problems of vehicle illegal parking detection in intelligent inspection scenarios are solved, and rapid and accurate vehicle illegal parking perception is achieved in complex dynamic scenarios.
Patent Information
- Application Number
- CN202410762133.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-13
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-06-13
AI Technical Summary
In intelligent inspection scenarios, existing target detection algorithms cannot achieve real-time detection while meeting the required detection accuracy, and license plate recognition algorithms perform poorly in dynamic and complex scenarios, making it difficult to accurately determine whether a vehicle is in an illegally parked area.
A lightweight vehicle license plate detection model and a license plate recognition model are constructed. Multimodal fusion is performed by combining multiple source sensors, and real-time vehicle illegal parking perception is achieved through edge computing.
It enables fast and accurate recognition of vehicles and license plates in complex and dynamic scenarios, supports image input of any size, meets real-time detection requirements, and can quickly and accurately determine whether a vehicle is illegally parked.
Smart Images

Figure CN118587695B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle management technology, and in particular to a method, device and medium for detecting illegal parking of vehicles in intelligent inspection scenarios. Background Technology
[0002] In communities and schools, illegal parking in no-parking zones is a common and prominent problem, causing considerable inconvenience to residents. Therefore, accurate detection and timely handling of illegal parking incidents are crucial. Currently, intelligent patrol vehicles have become a valuable tool for intelligent management of communities and schools. Equipped with various sensors and target detection algorithms, these vehicles can monitor vehicle parking in real time, effectively improving the management of illegal parking. However, intelligent patrol vehicles have limited power consumption, storage space, and computing power, making it difficult to handle the real-time processing of large amounts of data and the operation of complex algorithms. Existing high-precision target detection algorithms have a large number of model parameters, resulting in slow inference speeds and difficulty meeting real-time detection requirements, while lightweight models often have low detection accuracy. Furthermore, commonly used license plate recognition algorithms are cumbersome and affected by factors such as camera angle and lighting, leading to poor recognition performance in dynamic and complex scenarios.
[0003] In recent years, many scholars have conducted research on target detection algorithms for illegally parked vehicles. They have used various feature extraction methods to detect vehicles and developed corresponding strategies for identifying illegal parking incidents. However, these research methods still face some challenges in applying them to intelligent inspection scenarios. Most existing studies rely on images captured by fixed-view cameras for vehicle violation detection, which struggles to accurately detect vehicles in dynamic or blurry scenes. High-precision target detection algorithms based on convolutional neural networks can accurately detect vehicles, but these models are often highly complex, making real-time detection difficult to achieve on Jetson devices. Furthermore, many scholars rely on the boundary relationship between the vehicle and the detection box in a two-dimensional image to determine illegal parking. However, since vehicles are three-dimensional objects, the detection box deviates significantly from the vehicle's actual position, making it difficult to accurately determine whether a vehicle is within an illegal parking area. Summary of the Invention
[0004] The purpose of this invention is to address the problem that existing target detection algorithms cannot perform real-time detection while meeting the required detection accuracy in complex and ever-changing intelligent inspection scenarios. This invention provides a method, device, and medium for detecting illegally parked vehicles in intelligent inspection scenarios.
[0005] The above-mentioned objective of this application is achieved through the following technical solution:
[0006] S1: Construct and label datasets for various types of vehicles and license plates;
[0007] S2: Based on the YOLOv8 framework, design the CEN module and the depthwise separable convolution module to build a lightweight vehicle license plate detection model;
[0008] S3: Based on the LPRNet model, design a lightweight license plate recognition model; nest the lightweight license plate recognition model with the lightweight vehicle license plate detection model to construct a target recognition model;
[0009] S4: Train the target recognition model using the labeled dataset;
[0010] S5: Acquire the image to be detected; use the trained target recognition model to detect the image to be detected and obtain the vehicle information detection result;
[0011] S6: Acquire spatial information; Based on vehicle information detection results and spatial information, complete the detection of illegally parked vehicles in the inspection scenario.
[0012] Optionally, the lightweight vehicle license plate detection model includes: a Backbone module, a Neck module, and a Head module;
[0013] The input module is used to preprocess the input images of the dataset;
[0014] The Backbone module is used to extract deep features from the input image;
[0015] The Neck module is used to improve feature extraction capabilities;
[0016] The Head module is used to output the vehicle information detection results of the lightweight vehicle license plate detection model;
[0017] The construction process of the new Backbone module is as follows: the C2f module and some convolutional modules in the original Backbone module are replaced with the CEN module and depthwise separable convolutional modules;
[0018] The construction process of the Neck module is as follows: replace the original C2f module in the Neck module with the CEN module, and introduce the BiFuse module to build a new Neck module.
[0019] Optionally, the network hierarchy of the Backbone module includes:
[0020] First layer: Conv module, 64 convolutional kernels of size 3×3, stride of 2, resulting in data with features of 320×320×64;
[0021] The second layer: Conv module, with 128 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 160×160×128;
[0022] The third layer: CEN module, which uses shortcut connection to obtain data with a feature of 160×160×128;
[0023] The fourth layer: Conv module, with 256 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 80×80×256;
[0024] Fifth layer: CEN module, using shortcut connection, to obtain data with features of 80×80×256;
[0025] The sixth layer: Conv module, with 512 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 40×40×512;
[0026] The seventh layer: CEN module, using shortcut connections, yields data with a feature size of 40×40×512;
[0027] The eighth layer: Conv module, with 512 convolutional kernels of size 1×1 and a stride of 1, yields data with features of 20×20×512;
[0028] Ninth layer: CEN module, using shortcut connection, to obtain data with a feature of 20×20×512;
[0029] Layer 10: SPPF module, 3×3 pooling window, stride of 1, yields data with features of 20×20×512.
[0030] Optionally, the network hierarchy of the new Neck module includes:
[0031] First layer: Conv module, 512 convolutional kernels of size 3×3, stride of 1, resulting in data with features of 40×40×512;
[0032] The second layer: the BiFuse module connects the second layer of the new Neck module with the outputs of the fifth and seventh layers of the new Backbone module to obtain data with a feature of 40×40×512;
[0033] The third layer: CEN module, which outputs data with a feature of 40×40×512;
[0034] The fourth layer: Conv module, with 256 3×3 convolutional kernels and a stride of 2, yields data with features of 40×40×256;
[0035] Fifth layer: BiFuse module, which connects the fourth layer of the new Neck module's network hierarchy with the outputs of the third and fifth layers of the new Backbone module's network hierarchy to obtain data with a feature of 80×80×768;
[0036] Sixth layer: CEN module, which obtains data with features of 80×80×256;
[0037] The seventh layer: Conv module, with 256 3×3 convolutional kernels and a stride of 2, yields data with features of 40×40×256;
[0038] Layer 8: Connect the output of the sixth layer of the new Neck module's network hierarchy to the output of the fifth layer of the new Backbone module's network hierarchy, and output data with a feature of 40×40×768;
[0039] Ninth layer: CEN module, which yields data with a feature size of 40×40×512;
[0040] The tenth layer: Conv module, with 512 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 20×20×512;
[0041] Eleventh layer: Connects the output of the twelfth layer and the first layer of the new network hierarchy of the Neck module, and outputs data with a feature of 20×20×512;
[0042] The twelfth layer: CEN module, which obtains data with a feature of 20×20×512;
[0043] The thirteenth layer: the Detect module, which obtains data with features of 80×80, 40×40 and 20×20 respectively, for detecting targets of different sizes.
[0044] Optionally, step S2 includes: the lightweight vehicle license plate detection model adopts the YOLO-MTC model.
[0045] Optionally, step S3 includes:
[0046] The lightweight license plate recognition model mainly includes: a backbone structure, a context embedding recognition module, and an output module;
[0047] Replace the convolutions in the Basic block module of the LPRNet model with depthwise separable convolutions; replace the Dropout operation with the BatchNorm operation.
[0048] Optionally, the Backbone structure includes:
[0049] First layer: Conv module, 64 convolutional kernels of size 3×3, stride 1;
[0050] The second layer: the MaxPooling module, with 64 pooling layers of size 3×3 and a stride of 2;
[0051] The third layer: Basic block modules, 128 of size 3×3, with a step size of 1;
[0052] Fourth layer: MaxPooling module, 64 pooling layers of size 3×3, with a stride of 2;
[0053] Fifth layer: Basic block modules, 256 of each size (3×3) with a step size of 1;
[0054] Sixth layer: Basic block modules, 256 of each size (3×3) with a step size of 1;
[0055] The seventh layer: the MaxPooling module, consisting of 64 pooling layers of size 3×3 with a stride of 2;
[0056] Eighth layer: BatchNorm layer performs normalization;
[0057] Ninth layer: Conv module, 256 convolutional kernels of size 4×1, stride of 1;
[0058] The tenth layer: The BatchNorm layer performs normalization;
[0059] The eleventh layer: Conv module, with 256 convolutional kernels of size 1×13 and a stride of 1.
[0060] Optionally, step S6 includes:
[0061] The spatial information includes: geographic information data acquired by the antenna, point cloud data acquired by the lidar, and attitude information acquired by the inertial measurement unit;
[0062] Spatial information is fused through time synchronization and spatial registration to obtain the vehicle's real-time positioning coordinates;
[0063] Obtain data on the extent of illegally parked areas;
[0064] Based on the data of the illegal parking area and the location coordinates, it is determined whether a vehicle is illegally parked in a no-parking zone. Specifically, this includes:
[0065] If it is determined that the vehicle is illegally parked in a no-parking zone, the vehicle information detection results and location coordinates will be reported to the backend in real time.
[0066] If it is determined that the parking violation did not occur within the no-parking zone, no action will be taken.
[0067] A device for detecting illegally parked vehicles includes a Jetson edge computer, multiple sensors, and a memory. The memory stores instructions, and the Jetson edge computer executes the instructions stored in the memory to enable the device to perform a method for detecting illegally parked vehicles in intelligent inspection scenarios.
[0068] A computer-readable storage medium storing instructions that, when executed, perform a method for detecting illegally parked vehicles in an intelligent inspection scenario.
[0069] The beneficial effects of the technical solution provided in this application are:
[0070] 1. A target recognition model for vehicle and license plate detection proposes a basic feature extraction module (CEN module) to reduce model complexity and improve feature extraction capabilities. A BiFPAN Neck module is also proposed, which uses information from three different levels of feature maps as input for feature fusion, improving feature extraction capabilities to increase model accuracy while reducing computational load, meeting real-time requirements, and is suitable for deployment in intelligent inspection vehicles for real-time detection. Compared with existing methods, the target recognition model is lighter and has higher recognition accuracy for different types of vehicles, supports image input of arbitrary size, and can accurately identify license plate information in complex dynamic scenes. By nesting the above two models in an edge computer and combining them with multi-source sensors for multimodal fusion, the geographical coordinates of the detected vehicle can be calculated quickly and accurately, enabling accurate judgment and response to vehicle illegal parking events.
[0071] 2. In license plate recognition, by concatenating a lightweight vehicle license plate detection model with a lightweight license plate recognition model, the cumbersome preprocessing steps such as image enhancement and license plate localization in traditional license plate recognition algorithms are omitted, achieving integrated license plate detection and recognition, thus significantly simplifying the license plate recognition algorithm process. For license plate character recognition, to be applied in complex inspection scenarios, this paper proposes a further lightweight LPR-Nano algorithm based on the LPRNet model. This algorithm supports images of arbitrary sizes at the input end, eliminating the limitation of fixed input in the LPRNet model. In terms of network architecture, the feature extraction module and backbone convolution and pooling operations of the LPRNet model are optimized, and the STN localization module is removed, thereby reducing model complexity. Furthermore, the accuracy of license plate recognition is improved for various datasets. Therefore, this algorithm can achieve accurate license plate recognition in various complex scenarios.
[0072] 3. This innovative approach completes data classification and collection at the equipment level, model deployment and application at the edge, and data storage and complex analysis at the cloud center, achieving cloud-edge-device collaboration. It solves the problems of computational difficulties and untimely responses in traditional computing frameworks for urban management services. The cloud-edge-device collaborative computing framework can achieve the goals of optimizing resource allocation and reducing latency. This invention demonstrates that intelligent vehicles can quickly and accurately detect illegal parking incidents in community and campus patrol scenarios, and shows promising application prospects in urban grid-based integrated management services. Attached Figure Description
[0073] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:
[0074] Figure 1 This is a flowchart illustrating the steps of a vehicle illegal parking detection method for intelligent inspection scenarios in this application embodiment;
[0075] Figure 2 This is a structural diagram of a lightweight vehicle license plate detection model for a vehicle illegal parking perception method for intelligent inspection scenarios in this application embodiment;
[0076] Figure 3 This is a structural diagram of a lightweight license plate recognition model for a vehicle illegal parking perception method for intelligent inspection scenarios in this application embodiment;
[0077] Figure 4 This is a flowchart of a vehicle illegal parking detection method for intelligent inspection scenarios in this application embodiment;
[0078] Figure 5 This is a schematic diagram of the device structure for a vehicle illegal parking detection method for intelligent inspection scenarios in this application embodiment. Detailed Implementation
[0079] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0080] The embodiments of this application provide a method for detecting illegally parked vehicles in intelligent inspection scenarios.
[0081] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a vehicle illegal parking detection method for intelligent inspection scenarios in this application embodiment, including:
[0082] S1: Construct and label datasets for various types of vehicles and license plates;
[0083] Specifically, the sample dataset is labeled using an image annotation tool to generate corresponding annotation files. These files include a file format and file content. The file format can be XML, and the file content contains the target's category and location information. The target's location information includes the coordinates of the top-left and bottom-right corners of the real target. The labeled sample dataset is then divided into training, validation, and test sets according to a preset ratio. The annotation tool can be, but is not limited to, an image annotation tool, such as Labelimg software. In this embodiment, images of various vehicles and license plates are labeled as bicycles, motorcycles, tricycles, cars, and license plates, respectively. Labelimg software is used to annotate the positions and corresponding categories of vehicles and license plates in the images. The vehicle and license plate images and their corresponding annotation files are then divided into training, validation, and test sets according to an 8:1:1 ratio.
[0084] S2: Based on the YOLOv8 framework, design the CEN (Conv-ELAN-NeXt) module and the depthwise separable convolution module to build a lightweight vehicle license plate detection model;
[0085] S3: Based on the LPRNet model, design a lightweight license plate recognition model; nest the lightweight license plate recognition model with the lightweight vehicle license plate detection model to construct a target recognition model (LPRNet-Nano);
[0086] S4: Train the target recognition model using the labeled dataset;
[0087] Specifically, the number of training rounds is set in advance, and the target recognition model is trained based on the training set. Generally, 100 rounds are set to achieve convergence. It can be understood that after each round of training and validation, the accuracy of the current target recognition model is obtained, allowing the model to calculate its final accuracy after training ends. The final accuracy represents the best accuracy of the lightweight improved license plate recognition model on the validation set.
[0088] Specifically, the model is tested. The validated target recognition model is used to predict on the test set to be recognized, obtaining the prediction results on the test set. An accuracy threshold is pre-set; if the accuracy is less than the preset threshold, training stops and the target recognition model is output.
[0089] Specifically, Figure 4 This is a flowchart of a vehicle illegal parking detection method for intelligent inspection scenarios.
[0090] S5: Acquire the image to be detected; use the trained target recognition model to detect the image to be detected and obtain the vehicle information detection result;
[0091] S6: Acquire spatial information; Based on vehicle information detection results and spatial information, complete the detection of illegally parked vehicles in the inspection scenario.
[0092] Specifically, a cloud-edge-device collaborative processing model is adopted. The trained target detection model and license plate recognition model are deployed to the Jetson terminal of the intelligent inspection vehicle. The Jetson calculator is connected to the sensors. It receives and analyzes spatial information such as geographic information data acquired by the GNSS antenna, point cloud data acquired by the lidar, and attitude information acquired by the IMU inertial measurement unit, and uploads the above spatial information to the database. On the server side, the spatial information stored in the database is read, and multimodal data fusion is completed through time synchronization and spatial registration to obtain the vehicle's real-time accurate positioning coordinates. Based on the real-time coordinate changes of the vehicle during the inspection period and the range of illegal parking areas, it is determined whether the vehicle is illegally parked in a no-parking zone, achieving rapid and accurate detection of illegal parking.
[0093] Specifically, the intelligent inspection vehicle is driven into the inspection area to acquire images; real-time detection is performed using a target recognition model; if a motor vehicle is detected, the vehicle's movement and position are determined based on the multi-source sensor information of the intelligent inspection vehicle, and then, according to the established vehicle illegal parking rule algorithm, it is determined whether there is a vehicle illegally parked in the current inspection area. If so, the relevant information of the illegally parked vehicle is reported in real time; otherwise, no action is taken. The specific steps are as follows: By acquiring and calculating the real-time, accurate geographical coordinates of the detected vehicle, combined with the coordinate range of the illegal parking area, it is determined whether the vehicle is in the illegal parking area and has not moved during the inspection period. The vehicle information detection results are obtained based on the target recognition model. For illegally parked vehicles, their coordinate information, license plate number, vehicle and license plate color are recorded, uploaded to the backend server, and displayed on the real-view map.
[0094] The lightweight vehicle license plate detection model includes: a Backbone module, a Neck module, and a Head module;
[0095] The input module is used to preprocess the input images of the dataset;
[0096] The Backbone module is used to extract deep features from the input image;
[0097] The Neck module is used to improve feature extraction capabilities;
[0098] The Head module is used to output the vehicle information detection results of the lightweight vehicle license plate detection model;
[0099] Specifically, the Head module is used to measure the quality of the target recognition model's predictions. This module includes a classification loss function, a localization loss function, and a confidence loss function. Here, the classification loss function calculates the classification loss between the anchor box and the target box; the localization loss function calculates the error between the predicted box and the target box, which can determine the mean average precision (mAP) of the target recognition model; and the confidence loss function calculates the confidence score of the target recognition model. The target recognition model is then trained, validated, and tested using training, validation, and test sets.
[0100] The construction process of the new Backbone module is as follows: the C2f module and some convolutional modules in the original Backbone module are replaced with the CEN module and depthwise separable convolutional modules;
[0101] For details, please refer to Figure 2 , Figure 2 This is a structural diagram of the lightweight vehicle license plate detection model in this invention. The CEN module combines the ideas of CSPNet, ELAN, and NeXt structures to construct an aggregation structure with an effective gradient propagation path, avoiding network convergence degradation during model scaling. A stacked structure is used to optimize the gradient length of the entire network. Specifically, n CSPNet blocks are stacked, but each result is output, and finally concatenated. The C2f module is replaced by the CEN module, making it more lightweight and enabling more efficient extraction of license plate and vehicle features.
[0102] The construction process of the Neck module is as follows: replace the original C2f module in the Neck module with the CEN module, and introduce the BiFuse module to build a new Neck module.
[0103] Specifically, the BiFuse (Bic-Fusion) module fuses three feature maps from the backbone to obtain deeply fused image features. Lightweight vehicle license plate detection model (YOLO-MTC).
[0104] The network hierarchy of the Backbone module includes:
[0105] First layer: Conv module, 64 convolutional kernels of size 3×3, stride of 2, resulting in data with features of 320×320×64;
[0106] The second layer: Conv module, with 128 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 160×160×128;
[0107] The third layer: CEN module, which uses shortcut connection to obtain data with a feature of 160×160×128;
[0108] The fourth layer: Conv module, with 256 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 80×80×256;
[0109] Fifth layer: CEN module, using shortcut connection, to obtain data with features of 80×80×256;
[0110] The sixth layer: Conv module, with 512 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 40×40×512;
[0111] The seventh layer: CEN module, using shortcut connections, yields data with a feature size of 40×40×512;
[0112] The eighth layer: Conv module, with 512 convolutional kernels of size 1×1 and a stride of 1, yields data with features of 20×20×512;
[0113] Ninth layer: CEN module, using shortcut connection, to obtain data with a feature of 20×20×512;
[0114] Layer 10: SPPF module, 3×3 pooling window, stride of 1, yields data with features of 20×20×512.
[0115] Specifically, the Conv module is a convolutional module. The SPPF module is a spatial pyramid pooling module, which solves the problem of inconsistent input image sizes. It's understandable that in the lightweight vehicle license plate detection model, the Backbone module is used for feature extraction. Traditional lightweight vehicle and license plate detection models consist of convolutional Conv and C2f modules; the improved lightweight vehicle license plate detection model is more lightweight and has higher accuracy.
[0116] The new Neck module's network hierarchy includes:
[0117] First layer: Conv module, 512 convolutional kernels of size 3×3, stride of 1, resulting in data with features of 40×40×512;
[0118] The second layer: the BiFuse module connects the second layer of the new Neck module with the outputs of the fifth and seventh layers of the new Backbone module to obtain data with a feature of 40×40×512;
[0119] The third layer: CEN module, which outputs data with a feature of 40×40×512;
[0120] The fourth layer: Conv module, with 256 3×3 convolutional kernels and a stride of 2, yields data with features of 40×40×256;
[0121] Fifth layer: BiFuse module, which connects the fourth layer of the new Neck module's network hierarchy with the outputs of the third and fifth layers of the new Backbone module's network hierarchy to obtain data with a feature of 80×80×768;
[0122] Sixth layer: CEN module, which obtains data with features of 80×80×256;
[0123] The seventh layer: Conv module, with 256 3×3 convolutional kernels and a stride of 2, yields data with features of 40×40×256;
[0124] Layer 8: Connect the output of the sixth layer of the new Neck module's network hierarchy to the output of the fifth layer of the new Backbone module's network hierarchy, and output data with a feature of 40×40×768;
[0125] Ninth layer: CEN module, which yields data with a feature size of 40×40×512;
[0126] The tenth layer: Conv module, with 512 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 20×20×512;
[0127] Eleventh layer: Connects the output of the twelfth layer and the first layer of the new network hierarchy of the Neck module, and outputs data with a feature of 20×20×512;
[0128] The twelfth layer: CEN module, which obtains data with a feature of 20×20×512;
[0129] The thirteenth layer: the Detect module, which obtains data with features of 80×80, 40×40 and 20×20 respectively, for detecting targets of different sizes.
[0130] Specifically, the Neck module is composed of a novel BiFPAN structure proposed in this invention. BiFPAN is an FPN+PAN structure that combines a BiFuse module. The BiFuse module is a multi-directional joint structure with strong representational capabilities. To achieve a better tradeoff between accuracy and latency, this module uses feature inputs at three scales for fusion, introducing a bottom-up information flow into the top-down transmission path, enabling shallow features to participate in multi-scale feature fusion more efficiently and further enhancing the expressive power of the fused features. This module helps retain more accurate localization signals, which is of great significance for object localization. BiFPAN adds an extra edge between the original input and output nodes to fuse more functions without increasing costs. Improved multi-scale information fusion also improves contextual understanding of objects, helping to reduce false positives or false negatives. Compared to the YOLOv8 Neck structure, BiFPAN encapsulates the downsampling and upsampling processes within the BiFuse structure during the top-down transmission process and fuses three scales instead of two. Shallow information is downsampled, and deep information is upsampled to make the feature scales of the three different scales the same. These are then concatted and fused before output. Then, during the PAN (Packet Animation) process (i.e., bottom-up transmission), fusion is performed step-by-step, ultimately outputting feature maps of the three scales. Here, the Neck module immediately follows the Backbone module; that is, the final output of the Backbone module serves as the input to the Neck module.
[0131] Step S2 includes: The lightweight vehicle license plate detection model adopts the YOLO-MTC (Multi-CEN) model.
[0132] Step S3 includes:
[0133] The lightweight license plate recognition model mainly includes: a backbone structure, a context embedding recognition module, and an output module;
[0134] Replace the convolutions in the Basic block module of the LPRNet model with depthwise separable convolutions; replace the Dropout operation with the BatchNorm operation.
[0135] Specifically, the Dropout operation is replaced with the BatchNorm operation to enhance feature extraction.
[0136] Specifically, a lightweight vehicle license plate detection model and a lightweight license plate recognition model are nested together to construct a target recognition model. This model utilizes license plate corner points for tilted license plates, eliminating the need for preprocessing steps such as denoising, image enhancement, and license plate localization found in traditional license plate recognition algorithms. The lightweight vehicle license plate detection model accurately detects vehicles, license plates, and license plate corner points. By using these corner points and the license plate detection bounding box, distorted license plate images can be corrected. The license plate image is then directly input into the lightweight license plate recognition model for character recognition.
[0137] The Backbone structure includes:
[0138] First layer: Conv module, 64 convolutional kernels of size 3×3, stride 1;
[0139] The second layer: the MaxPooling module, with 64 pooling layers of size 3×3 and a stride of 2;
[0140] The third layer: Basic block modules, 128 of size 3×3, with a step size of 1;
[0141] Fourth layer: MaxPooling module, 64 pooling layers of size 3×3, with a stride of 2;
[0142] Fifth layer: Basic block modules, 256 of each size (3×3) with a step size of 1;
[0143] Sixth layer: Basic block modules, 256 of each size (3×3) with a step size of 1;
[0144] The seventh layer: the MaxPooling module, consisting of 64 pooling layers of size 3×3 with a stride of 2;
[0145] Eighth layer: BatchNorm layer performs normalization;
[0146] Ninth layer: Conv module, 256 convolutional kernels of size 4×1, stride of 1;
[0147] The tenth layer: The BatchNorm layer performs normalization;
[0148] The eleventh layer: Conv module, with 256 convolutional kernels of size 1×13 and a stride of 1.
[0149] Specifically, Figure 3This is a structural diagram of the lightweight license plate recognition model. The backbone structure of the lightweight license plate recognition model mainly consists of convolutional layers and pooling layers connected to the basic feature extraction module, Basicblock. Basicblock itself only uses a few convolutional layers for feature extraction, making the network very lightweight. The Basicblock module is the core feature extraction module in the backbone, mainly composed of Conv and depthwise separable convolutions, which improves feature extraction capabilities while reducing the number of model parameters. The context embedding module uses average pooling layers and also performs fusion at four scales to extract contextual information.
[0150] Step S6 includes:
[0151] The spatial information includes: geographic information data acquired by the antenna, point cloud data acquired by the lidar, and attitude information acquired by the inertial measurement unit;
[0152] Spatial information is fused through time synchronization and spatial registration to obtain the vehicle's real-time positioning coordinates;
[0153] Obtain data on the extent of illegally parked areas;
[0154] Based on the data of the illegal parking area and the location coordinates, it is determined whether a vehicle is illegally parked in a no-parking zone. Specifically, this includes:
[0155] If it is determined that the vehicle is illegally parked in a no-parking zone, the vehicle information detection results and location coordinates will be reported to the backend in real time.
[0156] If it is determined that the parking violation did not occur within the no-parking zone, no action will be taken.
[0157] Specifically, in this implementation example, the user plans a community as the area to be detected, and the intelligent inspection vehicle patrols within the community. The model analyzes the video stream in real time and detects each frame in real time to obtain the detection boxes and license plate numbers of various vehicles and license plates.
[0158] If a motor vehicle is detected, the system will determine whether the vehicle is illegally parked according to the established vehicle illegal parking judgment strategy. If so, the relevant information of the illegally parked vehicle will be reported in real time; otherwise, no action will be taken.
[0159] The steps include establishing rules for identifying illegally parked vehicles. When a motor vehicle is detected, the rules determine whether it is illegally parked. If so, the rule is reported to the back-end management center, which then notifies relevant staff to handle the situation promptly.
[0160] In this implementation case, the judgment of illegal parking events is a dynamic process within a time period. Actual testing revealed that the process from when a vehicle is detected by the camera to when it disappears from the frame takes approximately one minute. Therefore, one minute was chosen as the judgment time period for illegal parking. A Jetson sensor is installed on the intelligent vehicle, and through communication with the chassis control unit, it can acquire information collected by multiple sensors. During the vehicle's inspection process, the YOLO-MTC target detection algorithm is used for real-time detection. When a vehicle is detected, the system saves the corresponding laser point cloud file acquired by LiDAR; the latitude and longitude information of the inspection vehicle acquired by the GNSS antenna; and the attitude information of the inspection vehicle acquired by the IMU inertial measurement unit. It also records the corresponding time, detection frame information, and detection label, and then encodes and uploads these files and information to the database in real time. If the detected target is a motor vehicle, the LPR-Nano license plate recognition algorithm is further used to identify the vehicle's license plate, and the license plate information is also recorded and uploaded to the database. On the cloud server, the data rows corresponding to the vehicle information are read from the database table. Multimodal information (attitude information) is analyzed, and spatial registration and coordinate transformation are performed to complete the fusion of multi-source information. Through calculation, the latitude and longitude information of the detected target vehicle is obtained. Based on the coordinate range of no-parking zones defined on local maps of the community and campus, it is determined whether the detected vehicle is within an illegal parking area. If the vehicle's position does not change within the judgment period and it is within the illegal parking area, it is judged as illegally parked, and the event is sent back to the backend for processing by the property management department within the community or campus. Vehicles leaving the illegal parking area during this period are considered to be temporarily parked and are not judged as illegally parked.
[0161] like Figure 5 As shown, this embodiment of the invention also provides a terminal, which includes a Jetson edge computer 301 and a memory 302 for computer programs; wherein, when the Jetson edge computer 301 is used to run computer programs, it implements the vehicle and license plate detection model and the license plate recognition model described in any embodiment of the invention, and the memory 302 stores instructions and data.
[0162] In some embodiments of the present invention, the memory 302 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 302 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0163] The Jetson Edge Computer 301 features a compact size and high performance. It comes pre-installed with an ARM-based Ubuntu system and provides a GPU environment, significantly improving the training speed of reinforcement learning algorithms. Jetson Xavier NX supports developers in building and deploying high-quality, software-defined functionalities on embedded and edge devices. Software modules can reside in readily available storage media such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, and registers. This storage medium is located in memory 302. The Jetson Edge Computer 301 reads information from memory 302 and, in conjunction with its hardware, completes the steps described above.
[0164] In some embodiments, the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.
[0165] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.
[0166] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.
[0167] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.
Claims
1. A method for detecting illegally parked vehicles in intelligent inspection scenarios, characterized in that, The method includes the following steps: S1: Construct and label datasets for various types of vehicles and license plates; S2: Based on the YOLOv8 framework, design the CEN module and the depthwise separable convolution module to build a lightweight vehicle license plate detection model; The lightweight vehicle license plate detection model includes: a Backbone module, a Neck module, and a Head module; The input module is used to preprocess the input images of the dataset; The Backbone module is used to extract deep features from the input image; The Neck module is used to improve feature extraction capabilities; The Head module is used to output the vehicle information detection results of the lightweight vehicle license plate detection model; The construction process of the new Backbone module is as follows: the C2f module and some convolutional modules in the original Backbone module are replaced with the CEN module and depthwise separable convolutional modules; The construction process of the Neck module is as follows: replace the original C2f module in the Neck module with the CEN module, and introduce the BiFuse module to build a new Neck module; The new Neck module's network hierarchy includes: First layer: Conv module, 512 convolutional kernels of size 3×3, stride of 1, resulting in data with features of 40×40×512; The second layer: the BiFuse module connects the second layer of the new Neck module with the outputs of the fifth and seventh layers of the new Backbone module to obtain data with a feature of 40×40×512; The third layer: CEN module, which outputs data with a feature of 40×40×512; The fourth layer: Conv module, with 256 3×3 convolutional kernels and a stride of 2, yields data with features of 40×40×256; Fifth layer: BiFuse module, which connects the fourth layer of the new Neck module's network hierarchy with the outputs of the third and fifth layers of the new Backbone module's network hierarchy to obtain data with a feature of 80×80×768; Sixth layer: CEN module, which obtains data with features of 80×80×256; The seventh layer: Conv module, with 256 3×3 convolutional kernels and a stride of 2, yields data with features of 40×40×256; Layer 8: Connect the output of the sixth layer of the new Neck module's network hierarchy to the output of the fifth layer of the new Backbone module's network hierarchy, and output data with a feature of 40×40×768; Ninth layer: CEN module, which yields data with a feature size of 40×40×512; The tenth layer: Conv module, with 512 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 20×20×512; Eleventh layer: Connects the output of the twelfth layer and the first layer of the new network hierarchy of the Neck module, and outputs data with a feature of 20×20×512; The twelfth layer: CEN module, which obtains data with a feature of 20×20×512; The thirteenth layer: the Detect module, which obtains data with features of 80×80, 40×40 and 20×20 respectively, for detecting targets of different sizes; S3: Based on the LPRNet model, design a lightweight license plate recognition model; nest the lightweight license plate recognition model with the lightweight vehicle license plate detection model to construct a target recognition model; Step S3 includes: The lightweight license plate recognition model mainly includes: a backbone structure, a context embedding recognition module, and an output module; Replace the convolutions in the Basic block module of the LPRNet model with depthwise separable convolutions; replace the Dropout operation with the BatchNorm operation; S4: Train the target recognition model using the labeled dataset; S5: Acquire the image to be detected; use the trained target recognition model to detect the image to be detected and obtain the vehicle information detection result; S6: Acquire spatial information; Based on vehicle information detection results and spatial information, complete the detection of illegally parked vehicles in the inspection scenario.
2. The vehicle illegal parking detection method for intelligent inspection scenarios as described in claim 1, characterized in that, The network hierarchy of the Backbone module includes: First layer: Conv module, 64 convolutional kernels of size 3×3, stride of 2, resulting in data with features of 320×320×64; The second layer: Conv module, with 128 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 160×160×128; The third layer: CEN module, which uses shortcut connection to obtain data with a feature of 160×160×128; The fourth layer: Conv module, with 256 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 80×80×256; Fifth layer: CEN module, using shortcut connection, to obtain data with features of 80×80×256; The sixth layer: Conv module, with 512 convolutional kernels of size 3×3 and a stride of 2, yields data with features of 40×40×512; The seventh layer: CEN module, using shortcut connections, yields data with a feature size of 40×40×512; The eighth layer: Conv module, with 512 convolutional kernels of size 1×1 and a stride of 1, yields data with features of 20×20×512; Ninth layer: CEN module, using shortcut connection, to obtain data with a feature of 20×20×512; Layer 10: SPPF module, 3×3 pooling window, stride of 1, yields data with features of 20×20×512.
3. The vehicle illegal parking detection method for intelligent inspection scenarios as described in claim 1, characterized in that, The Backbone structure includes: First layer: Conv module, 64 convolutional kernels of size 3×3, stride 1; The second layer: the MaxPooling module, with 64 pooling layers of size 3×3 and a stride of 2; The third layer: Basic block modules, 128 of size 3×3, with a step size of 1; Fourth layer: MaxPooling module, 64 pooling layers of size 3×3, with a stride of 2; Fifth layer: Basic block modules, 256 of each size (3×3) with a step size of 1; Sixth layer: Basic block modules, 256 of each size (3×3) with a step size of 1; The seventh layer: the MaxPooling module, consisting of 64 pooling layers of size 3×3 with a stride of 2; Eighth layer: BatchNorm layer performs normalization; Ninth layer: Conv module, 256 convolutional kernels of size 4×1, stride of 1; The tenth layer: The BatchNorm layer performs normalization; The eleventh layer: Conv module, with 256 convolutional kernels of size 1×13 and a stride of 1.
4. The vehicle illegal parking detection method for intelligent inspection scenarios as described in claim 1, characterized in that, Step S6 includes: The spatial information includes: geographic information data acquired by the antenna, point cloud data acquired by the lidar, and attitude information acquired by the inertial measurement unit; Spatial information is fused through time synchronization and spatial registration to obtain the vehicle's real-time positioning coordinates; Obtain data on the extent of illegally parked areas; Based on the data of the illegal parking area and the location coordinates, it is determined whether a vehicle is illegally parked in a no-parking zone. Specifically, this includes: If it is determined that the vehicle is illegally parked in a no-parking zone, the vehicle information detection results and location coordinates will be reported to the backend in real time. If it is determined that the parking violation did not occur within the no-parking zone, no action will be taken.
5. A device for detecting illegally parked vehicles, characterized in that, It includes a Jetson edge computer, multiple sensors, and a memory; the Jetson edge computer loads and executes instructions and data in the memory to implement the vehicle illegal parking detection method for intelligent inspection scenarios as described in any one of claims 1 to 4.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform a vehicle illegal parking detection method for intelligent inspection scenarios as described in any one of claims 1-4.
Citation Information
Patent Citations
Smart community night illegal parking vehicle detection method and system based on Yolo feature enhancement
CN118172733A
Method and system for collecting and monitoring vehicle status information
US20230267779A1
