Illegal parking detection method based on visual identification of unmanned aerial vehicle

Abstract

The invention discloses an illegal parking detection method based on visual identification of an unmanned aerial vehicle, and belongs to the technical field of illegal parking detection. The unmanned aerial vehicle cruises according to preset parameters, and detects static vehicles in an illegal area in real time by using an airborne vehicle detection model; when a target vehicle is detected, the unmanned aerial vehicle adjusts the attitude and shoots the license plate of the vehicle from two different angles to obtain multiple groups of license plate images; performing feature matching and image alignment on the multiple groups of license plate images, and fusing to generate a complete license plate image; and based on the fused complete license plate image, license plate feature information is extracted for violation determination. The method can effectively improve the precision and real-time performance of vehicle detection, the integrity of a license plate image and the accuracy of subsequent character recognition.

Description

Technical Field

[0001] This invention belongs to the field of illegal parking detection technology, specifically relating to an illegal parking detection method based on UAV visual recognition. Background Technology

[0002] As cars become a means of transportation, their ownership in cities is increasing. As a result, available parking spaces in cities are becoming increasingly scarce, leading to numerous cases of illegal parking.

[0003] Currently, the detection of illegal parking is mainly done manually. This method requires users to manually monitor all prohibited parking areas for extended periods, and staff also need to issue tickets to vehicles. Due to weather conditions, users cannot continuously patrol all prohibited parking areas 24 / 7. Furthermore, illegally parked areas are often chaotic, and existing visual recognition algorithms suffer from issues such as missed detections and false detections in traffic scenarios. Especially in complex environments, their detection accuracy for small or obscured vehicles is insufficient, making it difficult to meet the demands for high-precision, real-time detection. Therefore, both traditional methods and existing algorithms have significant limitations and fail to meet the needs of personnel.

[0004] Mainstream visual recognition algorithms such as YOLOv12 are prone to character omissions and misidentifications in scenarios involving small target vehicles or tilted / obscured license plates, leading to failure in tracing illegal vehicles. When vehicles are densely packed, the license plates of target vehicles are easily obscured by vehicles in front and behind, roadside obstacles, or the characters on the license plates may be incomplete due to excessive drone shooting angles (such as side shots), failing to meet subsequent recognition requirements. Summary of the Invention

[0005] In view of the above-mentioned problems in the existing technology, the present invention proposes a method for detecting illegal parking based on UAV visual recognition. The method is reasonably designed, solves the shortcomings of the existing technology, and has good results.

[0006] To achieve the above objectives, the following technical solution is adopted: A method for detecting illegal parking based on drone visual recognition includes the following steps: Step 1: The drone cruises according to preset parameters and uses the onboard vehicle detection model to detect stationary vehicles in the violation area in real time; Step 2: When the target vehicle is detected, the drone adjusts its posture and takes pictures of the vehicle's license plate from two different angles to obtain multiple sets of license plate images; Step 3: Perform feature matching and image alignment on the multiple sets of license plate images, and fuse them to generate a complete license plate image; Step 4: Based on the fused complete license plate image, extract license plate feature information for violation determination.

[0007] Furthermore, in step 1, the vehicle detection model combines optical flow to determine whether the vehicle is stationary. The vehicle detection model is the YOLOv12-PCW model, and the backbone network of YOLOv12 is replaced with the PP-LCNet network. This network consists of five PP-LCNet Block layers stacked together. Each PP-LCNet Block uses a depthwise separable convolutional module to extract image features at different levels. YOLOv12's neck-embedded F-CBAM attention mechanism module, together with the neck's A2C2f and C3k2 modules, performs dual channel and spatial weighting on the fused multi-scale features. The F-CBAM attention mechanism module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to filter the channel information that is key to violation detection in the neck fused features, and the spatial attention submodule is used to focus on the spatial region where the target is located in the neck feature map. In the channel attention submodule, the input feature map x is first subjected to adaptive average pooling and adaptive max pooling respectively to obtain two 1×1 feature maps. and ;Will and First, channel compression is performed on each channel through the first 1x1 convolutional layer to obtain... and Then, a nonlinear transformation is performed using the SiLU activation function to obtain... and Finally, channel recovery is performed through a second 1x1 convolutional layer to obtain... and ;Will and Element-wise addition is performed, and the final channel attention weight matrix is ​​generated using the Sigmoid function. The generated channel attention weight matrix Multiply the original input feature map x element-wise to obtain the channel attention-enhanced feature map. , shape is , For height, For width, Number of channels; In the spatial attention submodule, the output feature map of the channel attention module Spatial features with different geometric meanings are extracted from four independent paths; path one extracts horizontal features by first performing adaptive average pooling in the horizontal direction. Then to Dimension expansion and mean processing are performed to obtain Path two involves extracting vertical features, first performing adaptive average pooling in the vertical direction to obtain... Then to Dimension expansion and mean processing are performed to obtain Path three involves extracting the maximum channel feature by taking the maximum value across the channel dimensions. Path four involves extracting the channel average feature by averaging across the channel dimensions. The features extracted from the four paths are concatenated along the channel dimension to obtain... Then, perform depthwise convolution and pointwise convolution sequentially to obtain... ; Will Batch normalization is performed, and the final spatial attention weight matrix is ​​generated using the Sigmoid function; the generated spatial attention weight matrix is ​​then compared with the feature map after channel attention enhancement. Multiply each element to get the final output. .

[0008] Furthermore, the vehicle detection model is trained using the Wk-IoU loss function. The expression is: ; in, The expression for the basic intersection-union ratio is: ; in, This is the intersection area of ​​the predicted bounding box and the ground truth bounding box in object detection, i.e., the area of ​​the overlapping portion of the two boxes. The height of the predicted bounding box, The width of the prediction box. The width of the actual bounding box. The actual height of the bounding box. It is a numerically stable term; The dynamic scaling factor is expressed as: ; in, For hyperparameters, for The moving average, expressed as: ; in, The momentum coefficient, Let IoU be the moving average at iteration step t. Calculated as an average; The overall weight is expressed as follows: ; in, Weighting factors for small objectives: ; Weighting factors for occlusion scenes: ; Aspect ratio weighting factor: .

[0009] Furthermore, step 3 includes the following sub-steps: Step 3.1: Preprocess multiple sets of license plate images, including denoising, grayscale conversion, and binarization; Step 3.2: Based on the preprocessed images, use the DeepLab V3+ semantic segmentation model to segment the license plate region images in each image; Step 3.3: For the segmented multiple license plate area images, use the SuperGlue model to match feature points from multiple images and establish relationships between images; Step 3.4: Based on the matched feature points, use the RANSAC algorithm to calculate the homography matrix and align the images of each license plate region to the same coordinate space; Step 3.5: Alpha-weighted fusion of the aligned license plate region images is used to generate a complete license plate image.

[0010] Furthermore, in step 3.2, the preprocessed vehicle images taken from different angles are input into the DeepLab V3+ semantic segmentation model to obtain license plate images from different angles. The DeepLab V3+ semantic segmentation model includes an encoder module and a decoder module. The encoder module captures multi-scale contextual information through dilated convolution, and the decoder is used to recover spatial details, thereby achieving pixel-level segmentation of the license plate area. The expression for dilated convolution is: ; in, To output the feature map at the th The value at each position, For expansion rate, The sampling location for the input feature. For the convolution kernel in the th The weight of each position.

[0011] Furthermore, in step 3.3, the SuperGlue model first constructs two graph structures from the feature points of the two license plate area images, with each feature point... As a graph node Then, the matching relationship of feature points is iteratively optimized through a multi-round iterative message passing mechanism; the message passing formula is: ; in, Indicates the first Nodes during round iteration eigenvectors, Represents a node and nodes Edge properties between The update function representing edge features. It is a multilayer perceptron. For nodes The set of neighboring nodes; The updated node features are obtained by calculating similarity scores between nodes. The similarity score matrix is ​​obtained. , The expression is: ; in, This represents the vector dot product operation. It is the total number of rounds of message passing; Finally, matching optimization is performed, using the Sinkhorn algorithm to optimize the similarity score matrix. Convert to a bidirectional normalized probability matrix , representing the final matching probability, probability matrix elements in Representing feature points and The matching probability; According to the matching probability matrix If and only if At the same time, it is the first The maximum value of the row and the first row Feature point pair is determined only when the column reaches its maximum value. To effectively match point pairs, thus forming a set of matched point pairs.

[0012] Furthermore, step 3.4 includes the following steps: Step 3.4.1: First, randomly select four non-collinear matching point pairs from the matching point sets of the two images, and calculate the candidate homography matrix. ; ; in, Matching points for the first image. For the corresponding points in the second image, the formula is expanded into a system of linear equations. Substituting the four matching point pairs, a 3×3 candidate homography matrix is ​​obtained. ; Step 3.4.2: Match all points in the first image use Mapping onto the second image, calculate the reprojection error for each matching point pair: ; Among them, feature points and These are the matching points and corresponding points in the two images, respectively. If the value is less than the threshold t, the matching point pair is identified as an interior point; otherwise, it is identified as an exterior point. Record the total number of interior points in the current iteration. Step 3.4.3: Determine if the proportion of interior points exceeds the preset value d. If it does not exceed d, repeat steps 3.4.1 and 3.4.2. If it exceeds d, terminate the iteration and output the current optimal homography matrix. ; Step 3.4.4: According to Transform the first image to the coordinate system of the second image.

[0013] Further, in step 3.5, the pixel values ​​of the two images are weighted and summed using weighting coefficients to obtain the fused license plate area image, expressed as: ; in, The merged image refers to the coordinates Pixel value at that location, and These refer to the coordinates of the two input images respectively. Pixel value at that location, These are weighting coefficients; The merged license plate area image is corrected to a standard rectangular plane by perspective transformation.

[0014] Furthermore, in step 4, the corrected license plate area image is preprocessed, including grayscale conversion, binarization, and noise reduction. The characters in the preprocessed image are segmented one by one. The cv2.bound-ingRect() function is used to obtain the information within the smallest rectangular bounding box of the image. The license plate information is sorted and finally the license plate character segmentation image is displayed. To address the characteristics of license plate characters—the first character being a Chinese abbreviation of the province, the second a letter, and the remaining characters a combination of letters and numbers—a template matching strategy is adopted: the first character is compared using a Chinese character template library, the second character uses an English character template, and the remaining characters use a mixed English and number template library. In the specific matching process, the template path and the image to be recognized are first read, and the templates are formatted and binarized. Then, the size of the image to be inspected is extracted, and the templates are scaled to the same size. Finally, the license plate number is recognized using a template matching algorithm.

[0015] The beneficial technical effects of this invention are as follows: 1. This invention constructs a vehicle detection model based on an improved YOLOv12-PCW, combining a lightweight backbone network PP-LCNet, an F-CBAM attention mechanism, and a Wk-IoU loss function in a collaborative architecture. This effectively solves the problems of missed and false detections in the detection of small target vehicles and occluded vehicles by existing visual algorithms, as well as the high computational requirements of the model, which are difficult to adapt to edge computing of UAVs. At the same time, the lightweight and precise design effectively improves the accuracy and real-time performance of vehicle detection.

[0016] 2. By controlling a drone to adjust its attitude, the system captures license plates of target vehicles from two different angles. DeepLab V3+ semantic segmentation is used to extract clean license plate regions, and SuperGlue feature matching and RANSAC homography matrix are used to align and fuse the images from these two angles. This solves the problem of incomplete information caused by occlusion or tilt in traditional single-view photography. Through complementary extraction and accurate reconstruction of license plate regions from two angles, the integrity of the license plate image and the accuracy of subsequent character recognition are significantly improved.

[0017] 3. By employing a pre-set drone flight path and an automatic triggering process that uses optical flow to determine the stationary state of vehicles, combined with an all-weather image preprocessing unit, the system overcomes the limitations of manual detection due to weather, time constraints, and the difficulty in covering blind spots. Drone patrols focus on covering wide areas and blind spots, effectively achieving all-weather, no-blind-spot coverage for illegal parking detection.

[0018] 4. By designing a fully automated violation detection process and deploying a lightweight detection model at the edge, the high labor costs and low response efficiency of traditional manual inspection methods are solved. The synergy between fully automated operation and the lightweight model significantly reduces the labor and time costs of detection, while shortening the detection response time from hours to minutes, thus improving the overall efficiency of illegal parking detection. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for detecting illegal parking based on UAV visual recognition in this invention.

[0020] Figure 2 This is a schematic diagram of the YOLOv12-PCW model structure in this invention.

[0021] Figure 3 This is the mAP training graph of the YOLOv12-PCW model in this invention.

[0022] Figure 4 This is a schematic diagram of the DeepLab V3+ model structure in this invention.

[0023] Figure 5 This is a schematic diagram of the input image for the DeepLab V3+ model. (a) is the first input image, and (b) is the second input image.

[0024] Figure 6 This is a schematic diagram of the input image for the DeepLab V3+ model. (a) is the first output image, and (b) is the second output image.

[0025] Figure 7 This is a schematic diagram of the matching process through matching points.

[0026] Figure 8 This is a schematic diagram of the merged license plate image.

[0027] Figure 9 This is a schematic diagram of the corrected license plate image.

[0028] Figure 10 This is a segmentation diagram of license plate characters. Detailed Implementation

[0029] The specific embodiments of the present invention will be further described below with reference to specific examples: A method for detecting illegal parking based on drone vision recognition, such as Figure 1 As shown, it includes the following steps: Step 1: The drone cruises according to preset parameters (flight route, altitude) and uses the onboard vehicle detection model to detect stationary vehicles in the violation area in real time; When the vehicle detection model detects a vehicle target within a violation area in a single-frame cruise image, it controls the drone gimbal to track the target vehicle and acquires multiple consecutive frames containing the vehicle. The optical flow method is used to determine whether the target vehicle is stationary. The vehicle detection model is the YOLOv12-PCW model, which replaces the YOLOv12 backbone network with the PP-LCNet network. This network consists of five stacked PP-LCNet Block layers, and each PP-LCNet Block uses a depthwise separable convolutional module. Figure 2Blocks 1-5 are the hierarchical feature extraction stages of the PPLCNet backbone network—stacked sequentially from 1 to 5, representing feature extraction from shallow to deep. Each block is responsible for extracting image features at different levels, and finally, multi-scale features are output through inter-layer fusion to adapt to subsequent vehicle detection and license plate segmentation tasks. Block 1 is used for the shallowest basic feature capture, mainly low-level visual features such as edges, textures, and colors. It includes 6 consecutive depthwise separable convolutional modules, each using a 3×3 convolutional kernel, with 32 output channels. Block 2 is used for the initial integration of shallow features, mainly simple shape features. It includes 2 consecutive depthwise separable convolutional modules, each using a 3×3 convolutional kernel. Block 3 is used for mid-level semantic feature enhancement, mainly for key target features. It includes two consecutive depthwise separable convolutional modules, each using a 3×3 convolutional kernel, with 128 output channels. Block 4 is used for deep high-level feature abstraction, mainly for complex semantic features. It includes six consecutive depthwise separable convolutional modules, each using a 5×5 convolutional kernel, with 256 output channels. Block 5 is used for the deepest level global feature fusion, mainly for global context features. It includes two consecutive depthwise separable convolutional modules, each using a 5×5 convolutional kernel, with 512 output channels. An SE attention submodule is embedded after each depthwise separable convolutional module. Inside the DepthSepConv module, channel-wise convolution (D-Conv), batch normalization (BN), H-Swish activation function, attention mechanism (SE), and pointwise convolution (P-Conv) are arranged in sequence. By leveraging the depth-separable mode, image features are efficiently captured, thereby achieving improved model performance while achieving lightweight design.

[0030] YOLOv12's neck-embedded F-CBAM attention mechanism module, in conjunction with the neck's A2C2f and C3k2 modules, performs dual channel and spatial weighting on the fused multi-scale features. The F-CBAM attention mechanism module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to filter the channel information that is key to violation detection in the neck fusion features and suppress redundant background channel interference. The spatial attention submodule is used to focus on the spatial region where the target is located in the neck feature map, solving the problem of target localization ambiguity in dense vehicle and occluded scenes. In conjunction with the neck's upsampling and feature stitching operations, it allows multi-scale features to more accurately align target information during the fusion process, providing high-quality feature input for head detection.

[0031] CBAM (Convolutional Block Attention Module) is a lightweight attention mechanism module for convolutional neural networks. It simulates the attention focusing characteristics of human vision by weighting features from the channel dimension and spatial dimension, highlighting key information and suppressing irrelevant noise, thereby improving the performance of the model in vision tasks.

[0032] The F-CBAM attention mechanism module is an improved CBAM attention mechanism, suitable for small target detection scenarios. The F-CBAM attention mechanism module includes a channel attention submodule and a spatial attention submodule. In the channel attention submodule, the channel dimension of the feature map is modeled to capture the importance correlation between channels; firstly, adaptive average pooling and adaptive max pooling are performed on the input feature map x to obtain two 1×1 feature maps. and : ; ; Will and First, channel compression is performed on each channel through the first 1x1 convolutional layer to obtain... and : ; ; Then, a nonlinear transformation is performed using the SiLU activation function to obtain... and : ; ; Finally, channel recovery is performed using a second 1x1 convolutional layer to obtain... and : ; ; Will and Adding elements together yields The final channel attention weight matrix is ​​generated using the Sigmoid function. : ; ; The generated channel attention weight matrix Multiply the original input feature map x element-wise to obtain the channel attention-enhanced feature map. , shape is , For height, For width, Number of channels; .

[0033] In the spatial attention submodule, the spatial dimension of the feature map is modeled, and lightweighting is achieved through 4-path fusion and depthwise separable convolution, combined with geometric perception. First, the output feature map of the channel attention module is processed. Spatial features with different geometric meanings are extracted from four independent paths respectively; Path 1 involves extracting horizontal features, first performing adaptive average pooling in the horizontal direction. Then to Dimension expansion and mean processing are performed to obtain : ; ; Path two involves extracting vertical features, first through adaptive average pooling. Then to Dimension expansion and mean processing are performed to obtain : ; ; Path three involves extracting the maximum channel feature by taking the maximum value across the channel dimensions. : ; Path four involves extracting the channel average feature by averaging across the channel dimensions. : ; The features extracted from the four paths are concatenated along the channel dimension to obtain... : ; Perform depthwise convolution sequentially to obtain Then, perform point-by-point convolution to obtain... : ; ; Will Batch normalization is performed, and the final spatial attention weight matrix is ​​generated using the Sigmoid function. : ; ; The generated spatial attention weight matrix is ​​compared with the feature map enhanced by channel attention. Multiply each element to get the final output. : .

[0034] Compared to CBAM, the F-CBAM attention mechanism adds dual-channel aggregation. The original CBAM only uses a single pooling method, while the improved version uses both average pooling and max pooling, then processes them separately through a shared MLP before fusing them, thus capturing channel information more comprehensively. It also adds directional spatial modeling, with adaptive pooling in both horizontal and vertical directions, specifically designed for the geometry in road scenes. The horizontal direction focuses on road layout, while the vertical direction focuses on target height. It performs four-way spatial feature fusion, combining global average, global maximum, horizontal, and vertical features to form a richer spatial representation. Finally, it implements a lightweight convolution design, using depthwise separable convolution instead of standard convolution in the spatial attention part. The combination of grouping operations and point convolutions ensures feature extraction capabilities while reducing the number of parameters.

[0035] Dual-channel aggregation, through a combination of average pooling and max pooling, simultaneously focuses on the overall channel distribution and peak response, which is useful for identifying different types of targets in roads. The shared MLP design ensures that while improving expressive power, it does not introduce too much additional overhead. Directional spatial modeling is the core innovation of this module, because road scenes have very obvious geometric characteristics: the horizontal direction mainly represents the extension of the road, and the vertical direction mainly represents the distribution of targets. Processing these two directions separately allows the model to better understand the scene structure. Four-channel spatial feature fusion combines global and directional information, preserving the perception of the overall scene while enhancing the grasp of local geometry. The application of depthwise separable convolution makes the spatial attention module very lightweight, finding a good balance between complexity and performance. The entire module achieves a good trade-off between the diversity of road scenes and computational efficiency requirements.

[0036] like Figure 2As shown, the input size of the YOLOv12-PCW model is 640×640. The input passes through PPLCNetBlock1, PPLCNet Block2, PPLCNet Block3, PPLCNet Block4, and PPLCNet Block5 in sequence, and then through the A2C2f module. The outputs of PPLCNet Block3, PPLCNet Block4, and A2C2f are respectively fed into the Neck module. After receiving the output of A2C2f from the backbone network, the neck performs upsampling and concatenates it with the output of PPLCNet Block4. The output obtained by the A2C2f module is denoted as feature map A. After upsampling, feature map A is concatenated with the output of PPLCNet Block3 and then concatenated with the A2C2f module and convolution (Conv) operation. After concatenation with feature map A, it is then concatenated with the A2C2f module to generate feature map B. Feature map B is then concatenated with the F-CBAM module to generate feature map C. Feature map C is then concatenated with the output of the A2C2f module in the backbone network and concatenated with the C3k2 module to generate feature map D. In the head, the feature maps B, C, and D output from the neck are respectively fed into the detection component to complete the detection task.

[0037] The vehicle detection model is trained using the Wk-IoU loss function. Wk-IoU is a precise enhancement and scene adaptation optimization of the traditional IoU loss. It focuses on "dynamic gradient allocation + special scene adaptation" to overcome the shortcomings of traditional IoU, which relies solely on the intersection-over-union ratio (IoU) and is insufficient in adapting to anchor box quality and scene differences. A dynamic non-monotonic focusing mechanism is introduced, moving away from simply using IoU values ​​to determine anchor box priority. Instead, it dynamically adjusts gradient weights based on the moving average of IoU, weakening over-optimization of high-quality anchor boxes and noise interference from low-quality anchor boxes, focusing on ordinary quality anchor boxes. Furthermore, scene-based comprehensive weights are added to address situations where traditional IoU handles small targets, occluded scenes, and large aspect ratio differences poorly, through three sub-weights (…). Precise empowerment makes the loss function more sensitive to special scenarios.

[0038] Wk-IoU loss function The expression is: ; in, The expression for the basic intersection-union ratio is: ; in, This is the intersection area of ​​the predicted bounding box and the ground truth bounding box in object detection, i.e., the area of ​​the overlapping portion of the two boxes. The height of the predicted bounding box, The width of the prediction box. The width of the actual bounding box. The actual height of the bounding box. It is a numerically stable term; The dynamic scaling factor is expressed as: ; in, For hyperparameters, =3, =1.9, for The moving average, expressed as: ; in, The momentum coefficient, For the t-th iteration Moving average For averaging, it is typically calculated across all samples in the current training batch. The average value; The overall weight is expressed as follows: ; in, Weighting factors for small objectives: ; Weighting factors for occlusion scenes: ; Aspect ratio weighting factor: ; Compared to WIoU, the Wk-IoU loss function adds a small target perception mechanism. Wk-IoU dynamically adjusts weights based on the bounding box area, using an inverse proportional function to give more attention to small targets. It also enhances the handling of difficult occlusion samples by adding occlusion judgment logic based on an IoU threshold. When the IoU falls below a certain value, the weights are automatically increased, so the model pays more attention to those difficult-to-detect occlusion situations. Furthermore, it adds the calculation of aspect ratio differences, guiding the model to learn more accurate target shapes by comparing the shape differences between the predicted and ground truth boxes. Finally, it adopts a hierarchical weighting strategy using multiplicative fusion, allowing weights of different dimensions to work together to form a more refined adjustment strategy.

[0039] The small target perception mechanism addresses the issue of the original version treating targets of all sizes equally. By using inverse area-proportional weights, it guides the model to focus more on small targets that are easily missed during training. The occlusion-heavy sample augmentation feature is specifically designed to handle frequent target occlusion in road scenes. When the overlap between the detection box and the ground truth box is low, it automatically increases the training weights, enabling the model to learn to accurately locate targets even under occlusion. The addition of shape constraints ensures that the model not only focuses on positional accuracy but also on shape resemblance, which is helpful for detecting road targets with diverse shapes. The multiplicative fusion strategy of multidimensional weights allows these improvements to work synergistically, ensuring the model receives appropriate training signals even with small targets, occlusion, and complex shapes, thus improving the overall detection capability of the model in complex road scenes.

[0040] Step 2: When the target vehicle is detected, the drone adjusts its attitude and takes pictures of the vehicle's license plate from at least two different angles to obtain multiple sets of license plate images; When a drone photographs a vehicle's license plate, if the target vehicle is close to other vehicles in front and behind, or if the license plate is obscured by objects in front or behind, it is difficult to obtain complete license plate information by shooting from a single direction. To address these special cases, the drone needs to take two images of the incomplete license plate from two different directions. These two images are first semantically segmented, then merged into a single, fully recognizable image. Finally, the complete image is then recognized to obtain the target license plate number.

[0041] Step 3: Perform feature matching and image alignment on the multiple sets of license plate images, and fuse them to generate a complete license plate image; Step 3 includes the following sub-steps: Step 3.1: Preprocess multiple sets of license plate images. Preprocessing includes denoising (Gaussian filtering), grayscale conversion (converting to single channel), and binarization (separating characters from background). Step 3.2: Based on the preprocessed images, use the DeepLab V3+ semantic segmentation model to segment the license plate region images in each image (excluding background interference). After preprocessing multiple sets of license plate images (denoising, grayscale conversion, binarization), although initial separation of characters from the background has been achieved, the directly extracted license plate regions may still contain redundant information due to interference from complex backgrounds such as vehicle bodies, surrounding obstructions, and road textures in the shooting environment, affecting the accuracy of subsequent feature matching and fusion. Therefore, semantic segmentation technology needs to be introduced—using a deep learning model to perform pixel-level classification of the images, accurately locating and extracting clean license plate regions, providing high-quality foundational data for multi-image alignment and fusion. The following will detail the specific implementation process of semantic segmentation based on DeepLab V3+, including model structure, feature extraction logic, and license plate region purification methods.

[0042] Given that the input consists of two license plate images taken from different angles, and we need to focus on the license plate region, we first perform license plate detection and segmentation operations on the input images to extract the specific region where the license plate is located. This task is accomplished using the DeepLab V3+ deep learning model, an advanced semantic segmentation model that captures multi-scale contextual information through dilated convolutions and combines an encoder-decoder structure to achieve high-precision segmentation results.

[0043] In practical application, after the two input images are processed by this model, a semantic segmentation mask is output, which is used to accurately locate the license plate region and extract the rectangular image containing the license plate. This process can effectively filter background noise and provide high-quality license plate region data for subsequent feature extraction and fusion tasks, laying a solid foundation for the entire process.

[0044] like Figure 4 As shown, the DeepLab V3+ semantic segmentation model includes an encoder module and a decoder module. The encoder module captures multi-scale contextual information through dilated convolutions, and it includes a backbone deep convolutional neural network (DCNN) and parallel dilated convolution modules. The backbone deep convolutional neural network uses continuous dilated convolutions. (Convolution), where "continuous" means that dilated convolutions are stacked layer by layer on the network; the features extracted by the backbone deep convolutional neural network are input to the parallel dilated convolution module, and the shallow features are transmitted to the decoder module. The dilated convolution module processes the input features in parallel using convolutions with different dilation rates, and then merges the outputs of each path and integrates and reduces dimensionality through a 1×1 convolution; the decoder module is used to restore spatial details, thereby achieving pixel-level segmentation of the license plate area. It receives two inputs: one is the high-level semantic features output by the dilated convolution module, which is first passed through a 1×1 convolution layer and then upsampled; the other is the shallow spatial detail features extracted from the backbone deep convolutional neural network. The decoder concatenates the upsampled high-level features with the shallow features, and then, after passing through a 3×3 convolution, it is upsampled to the original image size through bilinear interpolation to output the final segmentation mask; the scaling factor for the above upsampling operations is 4.

[0045] Among them, dilated convolution, by setting a dilation interval (i.e., the rate parameter), can effectively increase the receptive field when performing convolution operations on the input feature map, without increasing the computational burden or sacrificing spatial resolution. The expression is: ; in, To output the feature map at the th The value at each position, For expansion rate, The sampling location for the input feature. For the convolution kernel in the th Weight of each position In standard convolution operations, the convolution kernel slides continuously across the input feature map, performing point-by-point computation. Dilated convolution, however, inserts "gaps" between elements of the kernel by setting a dilation rate *r*, effectively increasing the spacing between sampling points by a factor of *r*. This makes the sampling locations of the kernel sparser, effectively expanding its receptive field. When *r* = 1, dilated convolution is equivalent to standard convolution, with the kernel sliding continuously. When *r* > 1, the spacing between sampling points increases, and the receptive field covered by the kernel expands accordingly. In this way, the dilation rate helps the model acquire broader contextual information without significantly increasing computational complexity.

[0046] Step 3.3: For the segmented license plate area images, use the SuperGlue model to match feature points from multiple images and establish relationships between them; SuperGlue is a feature matching framework based on graph neural networks, designed specifically for tasks such as image registration and multi-view reconstruction. Traditional feature matching algorithms (such as SIFT and ORB-based feature matching) typically rely on simple distance metrics (such as nearest neighbor matching) without considering the geometric relationships between feature points, making them susceptible to noise and mismatches. SuperGlue, building upon SuperPoint's feature point and descriptor extraction, uses a graph neural network (GNN) for global optimization of feature point matching and combines it with the Sinkhorn algorithm to achieve bidirectional consistency matching, significantly improving the accuracy and robustness of feature matching.

[0047] The SuperGlue model first constructs two graph structures from the feature points of the two license plate area images, with each feature point... As a graph node Then, the matching relationship of feature points is iteratively optimized through a multi-round iterative message passing mechanism; the message passing formula is: ; in, Indicates the first Nodes during round iteration eigenvectors, Represents a node and nodes Edge properties between The update function representing edge features. It is a multilayer perceptron. For nodes The set of neighboring nodes; The updated node features are obtained by calculating similarity scores between nodes. The similarity score matrix is ​​obtained. , The expression is: ; in, This represents the vector dot product operation. It is the total number of rounds of message passing; Finally, matching optimization is performed, using the Sinkhorn algorithm to optimize the similarity score matrix. Convert to a bidirectional normalized probability matrix , representing the final matching probability, and the elements in the probability matrix P. Representing feature points and The matching probability; According to the matching probability matrix If and only if At the same time, it is the first The maximum value of the row and the first row Feature point pair is determined only when the column reaches its maximum value. To effectively match point pairs, thus forming a set of matched point pairs.

[0048] Step 3.4: Based on the matched feature points, the homography matrix is ​​calculated using the RANSAC algorithm (Random Sample Consensus Algorithm) to align the multiple license plate area images to the same coordinate space; Step 3.4 includes the following steps: Step 3.4.1: First, randomly select 4 pairs of points from the matching point set of the two images, and calculate the candidate homography matrix. : ; in, Matching points for the first image. For the corresponding points in the second image, the formula is expanded into a system of linear equations. Substituting the four matching point pairs, a 3×3 candidate homography matrix is ​​obtained. ; Step 3.4.2: Match all points in the first image use Mapping onto the second image, calculate the reprojection error for each matching point pair: ; Among them, feature points and These are the matching points and corresponding points in the two images, respectively. If the value is less than the threshold t, the matching point pair is identified as an interior point; otherwise, it is identified as an exterior point. Record the total number of interior points in the current iteration. Step 3.4.3: Determine if the proportion of interior points exceeds the preset value d. If it does not exceed d, repeat steps 3.4.1 and 3.4.2. If it exceeds d, terminate the iteration and output the current optimal homography matrix. It can accurately describe the geometric transformation relationship between two images, while effectively eliminating the interference of noise points, thus providing a reliable foundation for subsequent image alignment and fusion.

[0049] Step 3.4.4: According to Transform the first image to the coordinate system of the second image.

[0050] A homography matrix is ​​a geometric transformation matrix used to describe the perspective transformation relationship between two images. When the relationship between two images can be represented by a planar projection model, the homography matrix can map points in one image to corresponding positions in the other image.

[0051] For input point and output point The relationship between the two can be determined by homography. Represented as: ; in, It is a 3x3 matrix.

[0052] Step 3.5: Alpha-weighted fusion (weighting based on pixel integrity) is applied to the aligned images of multiple license plate regions to generate a complete license plate image.

[0053] Alpha-weighted fusion achieves a smooth boundary transition by weighting pixel values ​​in overlapping regions, reducing discontinuities and visual artifacts.

[0054] Its core idea is to calculate the weighted sum of pixel values ​​from two images based on the position of a point in the image, using weighting coefficients, to obtain the fused license plate area image. The expression is: ; in, The merged image refers to the coordinates Pixel value at that location, and These refer to the coordinates of the two input images respectively. Pixel value at that location, These are the weighting coefficients.

[0055] The merged license plate area image is transformed and corrected to a standard rectangular plane using perspective transformation.

[0056] Step 4: Based on the fused complete license plate image, extract license plate feature information for violation determination.

[0057] The corrected license plate area image is preprocessed, including grayscale conversion, binarization, and denoising, to optimize the subsequent character feature extraction effect. A grayscale image is a special type of color image where each pixel has identical R, G, and B channel values. In computer representation, grayscale images have 256 pixel values ​​ranging from 0 to 255: 0 represents pure black, 255 represents pure white, and intermediate values ​​correspond to different grayscale levels. Converting an RGB color image to a grayscale image effectively simplifies image information and lays the foundation for subsequent processing.

[0058] The formula for converting RGB to grayscale is: ; The binarized pixel set usually refers only to the area with pixel values ​​between 0 and 255, without considering multiple pixel values, thus simplifying the processing. At the same time, binarized images also have the advantages of small computer memory capacity and fast processing speed, thus significantly improving the processing effect.

[0059] The binarization formula is: ; The characters in the preprocessed image are segmented one by one. The cv2.bound-ingRect() function is used to obtain the information within the smallest rectangular bounding box of the image. The license plate information is sorted and finally the license plate character segmentation image is displayed. To address the characteristics of license plate characters—the first character being a Chinese abbreviation of the province, the second a letter, and the remaining characters a combination of letters and numbers—a template matching strategy is adopted: the first character is compared using a Chinese character template library, the second character uses an English character template, and the remaining characters use a mixed English and number template library. In the specific matching process, the template path and the image to be recognized are first read, and the templates are formatted and binarized. Then, the size of the image to be inspected is extracted, and the templates are scaled to the same size. Finally, the license plate number is recognized using a template matching algorithm.

[0060] The experimental design for the YOLOv12-PCW model is as follows: Experimental configuration and parameter settings are as follows: The experiment was conducted on a Windows 11 system with a 13th-generation Intel Core i9-14900H CPU and a single NVIDIA 4070 graphics card. The algorithm was implemented using PyTorch 11.3 and accelerated with CUDA 11.8. Each network was trained for 100 epochs on a single card. The model was initialized with pre-trained weights, the training image size was 640×640 pixels, the batch size was 32, and the initial learning rate was 0.01. Training, validation, and testing were all completed under the same hardware and software environment.

[0061] The dataset and evaluation metrics are as follows: The open-source KITTI dataset was used, and the vehicle dataset was filtered. The dataset contains 7481 images and is divided into training, validation and test datasets, which are randomly allocated in a ratio of 8:1:1.

[0062] Precision (P), recall (R), mAP, and GFLOPs are commonly used as evaluation metrics in object detection experiments. Precision reflects the proportion of actual positives among the model's predicted positives; recall measures the proportion of correctly detected positives out of all actual positives; mAP is the core metric for evaluating the overall performance of an object detection model, used to assess its overall performance in "classification accuracy" and "localization accuracy"; and GFLOPs, or gigabit floating-point operations per second, measures the computational complexity of the model (the total number of floating-point operations performed during inference or training).

[0063] The comparative experiments are shown in Table 1. It can be seen that YOLOv12-PCW has the best overall detection performance, with an mAP of 88.58%, a 4.29% improvement over the benchmark model YOLOv12 (84.29%), and significantly exceeding mainstream advanced models such as YOLOv9t (83.94%) and YOLO11 (84.08%), ranking first among all compared models. Precision is significantly improved, while recall remains stable. Precision (P) reaches 89.51%, a 6.15% improvement over YOLOv12 (83.36%), indicating a significant improvement in the model's accuracy in predicting positive examples. Recall (R) is 77.81%, basically on par with YOLOv12 (76.34%), achieving a balance between "high precision + high recall". With a GFLOPs of only 5.21, lower than YOLOv12 (5.98) and most comparable models (such as YOLO11 at 6.05 and YOLOv8 at 6.95), YOLOv12-PCW improves performance while reducing computational complexity. Compared to traditional models such as RetinaNet and SSD, YOLOv12-PCW demonstrates a significant advantage in both performance and efficiency.

[0064] Table 1. Comparative experiments; ; The ablation experiments are shown in Table 2. It can be seen that the YOLOv12-PCW model achieved a mAP of 88.58%, an improvement of 4.29 percentage points compared to the baseline, and 2 percentage points higher than the highest improvement achieved by a single module (pplcnet's 86.59%). Figure 3As shown, the YOLOv12-PCW model achieves an accuracy of 89.51% and a recall of 77.81%, resolving the contradiction of "improving accuracy but reducing recall" or "improving recall but reducing accuracy" in a single module, and achieving an efficient balance between the two. The combined model's GFLOPs remain at 5.21, lower than the baseline (5.98), achieving a significant performance improvement without sacrificing computational efficiency, thus balancing high performance and real-time requirements.

[0065] Table 2 Ablation experiments; ; The experimental design for step 3 is as follows: Firstly, as Figure 5 Images (a) and (b) show two input images. The DeepLab V3+ model can be used to extract the license plate image, as shown below. Figure 6 As shown in (a) and (b), SuperGlue is used for feature point extraction, and then matching is performed based on the extracted feature points, as follows. Figure 7 The image shows the specific details of the matching points. The license plate area images are aligned to the same coordinate space and then fused using alpha blending, as shown below. Figure 8 As shown. Finally, perspective transformation is used to correct the license plate with an angled tilt, as shown. Figure 9 The image shown is the corrected license plate. The character segmentation image of the license plate was obtained using the `cv2.bound-ingRect()` function, as shown below. Figure 10 As shown.

[0066] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for detecting illegal parking based on UAV visual recognition, characterized in that, Includes the following steps: Step 1: The drone cruises according to preset parameters and uses the onboard vehicle detection model to detect stationary vehicles in the violation area in real time; Step 2: When the target vehicle is detected, the drone adjusts its posture and takes pictures of the vehicle's license plate from two different angles to obtain multiple sets of license plate images; Step 3: Perform feature matching and image alignment on the multiple sets of license plate images, and fuse them to generate a complete license plate image; Step 4: Based on the fused complete license plate image, extract license plate feature information for violation determination.

2. The method for detecting illegal parking based on UAV visual recognition according to claim 1, characterized in that, In step 1, the vehicle detection model combines optical flow to determine whether the vehicle is stationary. The vehicle detection model is the YOLOv12-PCW model. The backbone network of YOLOv12 is replaced with the PP-LCNet network. This network consists of five PP-LCNet Block layers stacked together. Each PP-LCNet Block uses a depth-separable convolutional module to extract image features at different levels. YOLOv12's neck-embedded F-CBAM attention mechanism module, together with the neck's A2C2f and C3k2 modules, performs dual channel and spatial weighting on the fused multi-scale features. The F-CBAM attention mechanism module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to filter the channel information that is key to violation detection in the neck fused features, and the spatial attention submodule is used to focus on the spatial region where the target is located in the neck feature map. In the channel attention submodule, the input feature map x is first subjected to adaptive average pooling and adaptive max pooling respectively to obtain two 1×1 feature maps. and ;Will and First, channel compression is performed on each channel through the first 1x1 convolutional layer to obtain... and Then, a nonlinear transformation is performed using the SiLU activation function to obtain... and Finally, channel recovery is performed through a second 1x1 convolutional layer to obtain... and ;Will and Element-wise addition is performed, and the final channel attention weight matrix is ​​generated using the Sigmoid function. The generated channel attention weight matrix Multiply the original input feature map x element-wise to obtain the channel attention-enhanced feature map. , shape is , For height, For width, Number of channels; In the spatial attention submodule, the output feature map of the channel attention module Spatial features with different geometric meanings are extracted from four independent paths respectively; Path 1 involves extracting horizontal features, first performing adaptive average pooling in the horizontal direction. Then to Dimension expansion and mean processing are performed to obtain Path two involves extracting vertical features, first performing adaptive average pooling in the vertical direction to obtain... Then to Dimension expansion and mean processing are performed to obtain Path three involves extracting the maximum channel feature by taking the maximum value across the channel dimensions. Path four involves extracting the channel average feature by averaging across the channel dimensions. The features extracted from the four paths are concatenated along the channel dimension to obtain... Then, perform depthwise convolution and pointwise convolution sequentially to obtain... ; Will Batch normalization is performed, and the final spatial attention weight matrix is ​​generated using the Sigmoid function; the generated spatial attention weight matrix is ​​then compared with the feature map after channel attention enhancement. Multiply each element to get the final output. .

3. The method for detecting illegal parking based on UAV visual recognition according to claim 1, characterized in that, The vehicle detection model is trained using the Wk-IoU loss function. The expression is: ； in, The expression for the basic intersection-union ratio is: ； in, This is the intersection area of ​​the predicted bounding box and the ground truth bounding box in object detection, i.e., the area of ​​the overlapping portion of the two boxes. The height of the predicted bounding box, The width of the prediction box. The width of the actual bounding box. The actual height of the bounding box. It is a numerically stable term; The dynamic scaling factor is expressed as: ； in, For hyperparameters, for The moving average, expressed as: ； in, The momentum coefficient, For the t-th iteration Moving average Calculated as an average; The overall weight is expressed as follows: ； in, Weighting factors for small objectives: ; Weighting factors for occlusion scenes: ; Aspect ratio weighting factor: .

4. The method for detecting illegal parking based on UAV visual recognition according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: Preprocess multiple sets of license plate images, including denoising, grayscale conversion, and binarization; Step 3.2: Based on the preprocessed images, use the DeepLab V3+ semantic segmentation model to segment the license plate region images in each image; Step 3.3: For the segmented multiple license plate area images, use the SuperGlue model to match feature points from multiple images and establish relationships between images; Step 3.4: Based on the matched feature points, use the RANSAC algorithm to calculate the homography matrix and align the images of each license plate region to the same coordinate space; Step 3.5: Alpha-weighted fusion of the aligned license plate region images is used to generate a complete license plate image.

5. The method for detecting illegal parking based on UAV visual recognition according to claim 4, characterized in that, In step 3.2, preprocessed vehicle images taken from different angles are input into the DeepLab V3+ semantic segmentation model to obtain license plate images from different angles. The DeepLab V3+ semantic segmentation model includes an encoder module and a decoder module. The encoder module captures multi-scale contextual information through dilated convolution, and the decoder is used to recover spatial details, thereby achieving pixel-level segmentation of the license plate area. The expression for dilated convolution is: ； in, To output the feature map at the th The value at each position, For expansion rate, The sampling location for the input feature. For the convolution kernel in the th The weight of each position.

6. The method for detecting illegal parking based on UAV visual recognition according to claim 4, characterized in that, In step 3.3, the SuperGlue model first constructs two graph structures from the feature points of the two license plate area images, with each feature point... As a graph node Then, the matching relationship of feature points is iteratively optimized through a multi-round iterative message passing mechanism; the message passing formula is: ； in, Indicates the first Nodes during round iteration eigenvectors, Represents a node and nodes Edge properties between The update function representing edge features. It is a multilayer perceptron. For nodes The set of neighboring nodes; The updated node features are obtained by calculating similarity scores between nodes. The similarity score matrix is ​​obtained. , The expression is: ； in, This represents the vector dot product operation. It is the total number of rounds of message passing; Finally, matching optimization is performed, using the Sinkhorn algorithm to optimize the similarity score matrix. Convert to a bidirectional normalized probability matrix , representing the final matching probability, probability matrix elements in Representing feature points and The matching probability; According to the matching probability matrix If and only if At the same time, it is the first The maximum value of the row and the first row Feature point pair is determined only when the column reaches its maximum value. To effectively match point pairs, thus forming a set of matched point pairs.

7. The method for detecting illegal parking based on UAV visual recognition according to claim 4, characterized in that, Step 3.4 includes the following steps: Step 3.4.1: First, randomly select four non-collinear matching point pairs from the matching point sets of the two images, and calculate the candidate homography matrix. : ； in, Matching points for the first image. For the corresponding points in the second image, the formula is expanded into a system of linear equations. Substituting the four matching point pairs, a 3×3 candidate homography matrix is ​​obtained. ; Step 3.4.2: Match all points in the first image use Mapping onto the second image, calculate the reprojection error for each matching point pair: ； Among them, feature points and These are the matching points and corresponding points in the two images, respectively. If the value is less than the threshold t, the matching point pair is identified as an interior point; otherwise, it is identified as an exterior point. Record the total number of interior points in the current iteration. Step 3.4.3: Determine if the proportion of interior points exceeds the preset value d. If it does not exceed d, repeat steps 3.4.1 and 3.4.

2. If it exceeds d, terminate the iteration and output the current optimal homography matrix. ; Step 3.4.4: According to Transform the first image to the coordinate system of the second image.

8. The method for detecting illegal parking based on UAV visual recognition according to claim 4, characterized in that, In step 3.5, the pixel values ​​of the two images are weighted and summed using weighting coefficients to obtain the fused license plate area image, expressed as: ； in, The merged image refers to the coordinates Pixel value at that location, and These refer to the coordinates of the two input images respectively. Pixel value at that location, These are weighting coefficients; The merged license plate area image is corrected to a standard rectangular plane by perspective transformation.

9. The method for detecting illegal parking based on UAV visual recognition according to claim 4, characterized in that, In step 4, the corrected license plate area image is preprocessed, including grayscale conversion, binarization, and noise reduction. The characters in the preprocessed image are segmented one by one. The cv2.bound-ingRect() function is used to obtain the information within the smallest rectangular bounding box of the image. The license plate information is sorted and finally the license plate character segmentation image is displayed. To address the characteristics of license plate characters—the first character being a Chinese abbreviation of the province, the second a letter, and the remaining characters a combination of letters and numbers—a template matching strategy is adopted: the first character is compared using a Chinese character template library, the second character uses an English character template, and the remaining characters use a mixed English and number template library. In the specific matching process, the template path and the image to be recognized are first read, and the templates are formatted and binarized. Then, the size of the image to be inspected is extracted, and the templates are scaled to the same size. Finally, the license plate number is recognized using a template matching algorithm.

Images