Unmanned aerial vehicle based non-motorized lane parking detection method and device

Abstract

The application discloses a non-motor vehicle lane parking detection method and device based on a UAV, and belongs to the field of image data analysis.The method comprises the following steps: acquiring image data of the UAV, identifying the image data, and determining a non-motor vehicle lane; acquiring a first feature map, a second feature map, a third feature map, a fourth feature map and a fifth feature map from the image data; determining first feature information according to the fifth feature map; determining a first intermediate feature, a second intermediate feature, a third intermediate feature and a fourth intermediate feature; determining a first fusion feature and a second fusion feature; determining an aggregated feature according to the first intermediate feature, the first fusion feature, the second fusion feature and the fourth intermediate feature, and performing analysis according to the aggregated feature to determine the position of a vehicle on the non-motor vehicle lane and a parking spot and to perform illegal analysis; and the scheme can identify vehicle illegal behaviors by using UAV images, so that the road traffic can be better managed.

Description

Technical Field

[0001] This invention relates to the field of image data analysis technology, and in particular to a method and device for detecting parking on non-motorized vehicle lanes based on unmanned aerial vehicles (UAVs). Background Technology

[0002] In traffic management scenarios, drones serve as a valuable supplement to ground-based monitoring facilities. They can be used for traffic flow monitoring, traffic accident evidence collection, traffic violation detection, and traffic guidance and control, which can alleviate the problems of limited coverage and blind spots of fixed monitoring equipment to a certain extent.

[0003] Regarding the detection of motor vehicles occupying non-motorized vehicle lanes, existing methods mainly include fixed road monitoring equipment and manual patrols. While fixed monitoring equipment can achieve video surveillance, its limited deployment location and number can easily create blind spots, making it difficult to achieve full road coverage. Manual patrols, on the other hand, suffer from high labor costs, low work efficiency, and untimely evidence collection. When dealing with sudden violations such as lane occupation and temporary parking, these methods often suffer from problems such as untimely response and incomplete evidence collection, affecting the effectiveness of road traffic management. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention aims to provide a method and device for detecting parking in non-motorized vehicle lanes based on unmanned aerial vehicles (UAVs). This method utilizes UAVs to collect road images and employs an improved target detection model to more effectively integrate multi-scale feature maps, thereby more accurately analyzing vehicles and parking spaces in non-motorized vehicle lanes and identifying vehicle violations for better road traffic management.

[0005] In a first aspect, this application provides a method for detecting parking in non-motorized vehicle lanes based on unmanned aerial vehicles (UAVs). The method includes: acquiring image data from the UAV; identifying the image data to determine the non-motorized vehicle lane; acquiring multi-scale feature representations from the image data to form a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map in ascending order of hierarchy; determining first feature information based on the fifth feature map; determining first intermediate features, second intermediate features, third intermediate features, and fourth intermediate features based on the second feature map, third feature map, fourth feature map, and first feature information; determining a first fusion feature based on the first intermediate feature and second intermediate feature; determining a second fusion feature based on the third intermediate feature and fourth intermediate feature; determining an aggregate feature based on the first intermediate feature, first fusion feature, second fusion feature, and fourth intermediate feature; analyzing based on the aggregate feature to determine the vehicle position and parking spot on the non-motorized vehicle lane; performing vehicle violation analysis based on the vehicle position and parking spot; and acquiring vehicle information of the target vehicle through a UAV camera when the target vehicle violates a violation.

[0006] Optionally, the image data processing is performed by an object detection model. The first and second fusion features are determined by the fusion module in the object detection model. The fusion module is used to: perform 1×1 convolution processing on the input features to form spliced ​​features. The input features include the following combinations: a combination of the first and second intermediate features, and a combination of the third and fourth intermediate features; perform spatial information fusion processing on the spliced ​​features using a first branch module to determine the first branch feature; process the spliced ​​features using a second branch module based on spatial attention and channel attention mechanisms to determine the second branch feature; and fuse the first and second branch features to determine the fused features, which include either the first or second fusion feature.

[0007] Optionally, the step of using the first branch module to perform spatial information fusion processing on the spliced ​​features to determine the first branch features includes: sequentially performing 3×3 convolution processing, normalization processing, activation processing, max pooling processing, 3×3 convolution processing, and upsampling processing on the spliced ​​features to capture spatial context information and determine the first branch features.

[0008] Optionally, determining the aggregated feature based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature includes: performing feature fusion based on the first fusion feature and the second fusion feature to determine the fifth intermediate feature; and performing feature fusion based on the first intermediate feature, the fourth intermediate feature, and the fifth intermediate feature to determine the aggregated feature.

[0009] Optionally, the step of analyzing based on aggregated features to determine the vehicle positions and parking spots on the non-motorized vehicle lane includes: performing uncertainty measurement based on aggregated features to filter out candidate regions for each object; querying the candidate regions for target objects to determine target feature data, where the target objects include vehicles and parking spots on the non-motorized vehicle lane; and decoding the target feature data to determine the vehicle positions and parking spots on the non-motorized vehicle lane.

[0010] Optionally, the analysis of vehicle violations based on vehicle location and parking spot includes: determining the parking area on the non-motorized vehicle lane based on the parking spot; determining the first vehicle outside the parking area based on the parking area and vehicle location; tracking the first vehicle on the non-motorized vehicle lane; and identifying target vehicles whose parking status and driving time are less than a preset time threshold as violating vehicles.

[0011] Optionally, the step of determining whether the target vehicle is in a parked or driving state includes: obtaining the rotation box parameters corresponding to the target vehicle in multiple consecutive frames of image data, and performing exponential smoothing on the rotation box parameters to determine the smoothed center information and scale information, wherein the scale information is determined based on the width or height of the rotation box parameters; analyzing the background covered by other vehicles outside the target vehicle based on the multiple consecutive frames of image data and generating replacement information to form the background data corresponding to the target vehicle; extracting corner features based on the background data of the image data; analyzing the corner features using optical flow to determine the homography matrix, wherein the homography matrix is ​​used to determine the transformation of background data between adjacent frames; and determining whether the vehicle's state in the multiple consecutive frames of image data is a parked or driving state based on the homography matrix, the smoothed center information, and the scale information.

[0012] Optionally, determining the parking area on the non-motorized vehicle lane based on parking spots includes: determining the main direction corresponding to multiple parking spots and determining the spot array corresponding to the main direction; projecting the parking spots in the spot array onto the main direction and sorting them; traversing the sorted parking spots and calculating the difference to determine the average distance of the parking spots; filling in missing spots based on the average distance, parking spots, and the start and end points of the main direction to form corrected parking spots, and mapping the parking area to form a parking area; the step of filling in missing spots further includes: acquiring images of adjacent frames of image data and determining the target spots corresponding to the missing spots to fill in the missing spots corresponding to the image data.

[0013] Optionally, the step of identifying the image data to determine the non-motorized vehicle lane includes: identifying roads, flower bed railings, and lane lines in the image data to form an extraction result; and analyzing the extraction result and logical judgment rules to determine the non-motorized vehicle lane.

[0014] Secondly, this application provides a non-motorized vehicle lane parking detection device based on unmanned aerial vehicles (UAVs). The device includes: an image data acquisition module for acquiring image data from the UAV, recognizing the image data, and determining the non-motorized vehicle lane; a feature map acquisition module for acquiring multi-scale feature representations from the image data to form a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map in a hierarchy from low to high; and determining first feature information based on the fifth feature map; an intermediate feature acquisition module for determining a first intermediate feature, a second intermediate feature, a third intermediate feature, and a fourth intermediate feature based on the second feature map, the third feature map, the fourth feature map, and the first feature information; determining a first fusion feature based on the first intermediate feature and the second intermediate feature; and determining a second fusion feature based on the third intermediate feature and the fourth intermediate feature; an aggregation feature acquisition module for determining an aggregation feature based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature, and analyzing the aggregation feature to determine the vehicle position and parking spot on the non-motorized vehicle lane; and a vehicle violation analysis module for analyzing vehicle violations based on the vehicle position and parking spot, and acquiring vehicle information of the target vehicle through the UAV camera when the target vehicle violates a violation.

[0015] Thirdly, this application provides an electronic device, comprising: a memory and at least one processor; the memory being used to store computer execution instructions; the at least one processor being used to execute the computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in the first aspect.

[0016] Fourthly, this application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0017] This application can be applied to road traffic management scenarios based on unmanned aerial vehicles (UAVs). It utilizes UAVs to collect road images and employs an improved target detection model to more effectively fuse multi-scale feature maps, thereby more accurately analyzing vehicles and parking spaces in non-motorized vehicle lanes and identifying vehicle violations. Specifically, this solution improves the target detection model, enabling it to fuse features at more scales and integrate features from adjacent layers. This allows for better fusion of features across different levels, resulting in more accurate identification of vehicle positions and parking locations. Furthermore, this solution can analyze the transformation information of vehicles and the background in consecutive frames of image data to identify whether a vehicle is stationary or in motion. Vehicles that are parked or have a travel time shorter than a preset threshold can be identified as violating traffic rules, and this vehicle information is saved using the UAV's camera. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the architecture of an object detection model according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the steps of a non-motorized vehicle lane parking detection method based on an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a non-motorized vehicle lane parking detection device based on an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms, while “a plurality” refers to two or more, and other quantifiers are similarly understood. It should be further understood that the word “comprising” as used in this application’s specification means the presence of the stated feature, integer, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The word “and / or” as used herein describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0021] This application can be applied to road traffic management scenarios based on unmanned aerial vehicles (UAVs). It utilizes UAVs to collect road images and employs an improved target detection model to more effectively fuse multi-scale feature maps, thereby more accurately analyzing vehicles and parking spaces in non-motorized vehicle lanes and identifying vehicle violations. Specifically, this solution can acquire UAV image data and identify the non-motorized vehicle lanes. It can also extract multi-scale feature representations from the image data, forming a hierarchical structure of first, second, third, fourth, and fifth feature maps. Based on the fifth feature map, it determines the first feature information at a higher level. Feature extraction and fusion processing are then performed based on the second, third, and fourth feature maps and the first feature information to determine the first, second, third, and fourth intermediate features. After determining these four layers of intermediate features, the solution can fuse the lower two layers and the upper two layers of intermediate features to determine the first and second fused features. Subsequently, based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature, aggregate features are determined. Based on the aggregate features, analysis is performed to determine the vehicle positions and parking spots on the non-motorized vehicle lane. Based on the vehicle positions and parking spots, vehicle violation analysis is performed, and when a target vehicle violates a violation, vehicle information of the target vehicle is obtained through a drone camera.

[0022] The method and apparatus provided by this invention mainly utilize a pre-set flight path or manual flight of a drone to fly the drone above urban roads, detect vehicles illegally driving or parking in non-motorized vehicle lanes in drone inspection images, and take corresponding actions. Specifically, this invention adopts the following technical solution: Image acquisition step: Control the drone to acquire a sequence of aerial images of the target road segment according to a pre-set flight path or manual flight, and simultaneously acquire the drone's pose information. Road element segmentation step: Use a semantic segmentation model to segment the road area, flower bed railings, and solid white lines in the aerial images to obtain corresponding segmentation masks. Online non-motorized vehicle lane delineation step: Based on the road area segmentation mask, flower bed railing segmentation mask, and solid white line segmentation mask, delineate the effective area of ​​the non-motorized vehicle lane in the current frame image through pre-set topological rules and geometric operations; the topological rules at least include: using solid white lines as boundary constraints between the motorized vehicle lane and adjacent areas, and combining flower bed railings or curb structures to determine the range of the non-motorized vehicle lane.

[0023] Vehicle detection and tracking steps: An improved target detection model (such as the RT-DETR model) is used to detect vehicles in the image, and a target tracking algorithm (such as the BoT-SORT algorithm) is used to track the detected vehicles across frames to obtain the tracking trajectory and unique identifier of each vehicle. Parking status determination steps: The homography matrix + optical flow analysis method is used to determine whether a vehicle is in a parked state. Online parking space reconstruction steps: The improved RT-DETR + BOT-SORT method model is used to detect visible parking space feature points in the image and track them; the parking space features detected in several consecutive frames are correlated. For parking spaces that are not detected in the current frame due to occlusion or other reasons, the occluded or missing parking spaces are restored by correlating consecutive previous and subsequent frames and combining with the parking space spatial restoration module. Illegal parking determination steps: For vehicles in the non-motorized vehicle lane, it is determined whether they are in a parked state and whether they are located in a parking space, thereby determining whether they are illegally parked. The steps for identifying license plates of illegally parked vehicles are as follows: For illegally parked vehicles, the drone's camera is magnified 7 times to capture the license plate, the license plate is identified, and the identification results are reported.

[0024] The specific steps of this solution include: Road segmentation of UAV imagery to obtain non-motorized vehicle lane segmentation results. This process involves training a real-time target detection model (such as the YOLOv11 model) using pre-labeled roads, flower bed railings, and lane lines. The trained model is then used to segment and extract roads, flower bed railings, and lane lines from UAV urban road imagery. Based on a segmentation mask, solid white lines are used as the boundary constraints between the motorized vehicle lane and adjacent areas. Combined with flower bed railings and curb structures, and through preset topological rules and geometric operations, the effective area of ​​the non-motorized vehicle lane in the current image is delineated online. An improved RT-DETR+BoT-SORT algorithm is used to detect and track vehicles in the non-motorized vehicle lane. A homography matrix + optical flow analysis method is used to determine whether a vehicle in the non-motorized vehicle lane is in a moving or parked state.

[0025] Some existing technical solutions for determining whether a vehicle is stopped or moving have several shortcomings. For example, platform jitter interference: even slight camera or drone shaking can cause a stationary vehicle to appear with noticeable pixel displacement in the image, easily misjudged as moving; global motion interference: when the platform translates / rotates, stationary vehicles move synchronously with the background; the displacement of moving vehicles may be partially canceled out, easily misjudged as stationary; detection box jitter: the center of the box jitters frame by frame, making it difficult for the single-frame displacement threshold to simultaneously account for minor movements and true stops; inconsistent resolution: pixels appear larger when closer and smaller when farther away, meaning the same absolute pixel threshold has different implications for vehicles at different distances. This solution can robustly determine whether a vehicle is stopped or stationary relative to the ground while the drone is moving. Specifically, it detects and tracks the target vehicle in consecutive video frames, obtaining the OBB parameters for each frame. The OBB parameters are then exponentially smoothed to obtain the smoothed center. and scale The angle is smoothed against sinθ and cosθ respectively before being restored to its original state to avoid abrupt angle changes. Here, w and h represent the width and height, and θ is the orientation angle. This step helps suppress the impact of detection box jitter on the center point.

[0026] Obtain or estimate the road region mask and remove all vehicles within that region. This step constructs the road background feature region excluding vehicles. Extract corner features from the previous frame's road background ROI. Use optical flow to match the current frame and perform forward and backward consistency checks, discarding unreliable points; if there are enough matching points, estimate the homography matrix using the Random Sample Consensus (RANSAC) algorithm; perform validity checks on H; if invalid, backtrack to the most recent valid H, or do not update the motion state in this frame. H represents the geometric transformation of the road background from the previous frame to the current frame.

[0027] Calculate the homography prediction residuals. Let the smoothing center of the previous frame be... The smoothing center of the current frame is Then the residuals can be calculated; where, when the residuals When the value is close to 0, the vehicle is stationary; when... A significant increase indicates that the vehicle's actual position deviates from the background prediction, meaning the vehicle is in motion. A multi-frame state machine is used to confirm whether the vehicle is stopped or parked. A sliding window of the most recent N candidate states is maintained. In the most recent N frames, if the number of frames in motion is greater than 6, the vehicle is classified as moving; if the number of frames in stationary state is greater than 7, the vehicle is classified as parked.

[0028] An improved RT-DETR+BoT-SORT method is used for target detection and line tracking at parking locations. A parking space reconstruction module, combining consecutive frames, reconstructs the detected parking locations into actual parking spaces. If no parking space is detected in the non-motorized vehicle lane, the camera is adjusted to 7x focal length to identify vehicle license plates. Once a license plate is identified, the result is reported.

[0029] If a parking space is detected in the non-motorized vehicle lane, the vehicle in the parking space is removed, and then it is determined whether the vehicle is in a moving or parked state. If it is parked, the camera is adjusted to 7x zoom to identify the vehicle's license plate, and the license plate recognition result is reported. If it is in a moving state, to avoid classifying a vehicle that has just left the parking space as illegally occupying it, it is determined whether the moving vehicle has been in motion for more than 5 seconds. If it has been in motion for more than 5 seconds, the camera is adjusted to 7x zoom to identify the vehicle's license plate, and the license plate recognition result is reported. Otherwise, no action is taken.

[0030] The drone integrates different task detection or segmentation models, including road segmentation detection, flower bed and railing segmentation detection, lane line segmentation detection, vehicle tracking detection, parking space detection, and license plate recognition detection. Specifically, a logic judgment processing module is added to the road detection, flower bed and railing detection, and lane line detection models to delineate non-motorized vehicle lanes.

[0031] This solution employs an improved RT-DETR framework to simultaneously and accurately detect both vehicles and parking spaces. The corresponding improvements are detailed below. Figure 1 As shown in the figure: S1 represents the first feature module, S2 represents the second feature module, S3 represents the third feature module, S4 represents the fourth feature module, S5 represents the fifth feature module, Fusion represents feature fusion, DBFF represents dual-branch feature fusion, AIFI represents internal scale feature interaction module, Uncertainty-vminimal Query Selection represents uncertainty query module, Decoder & Head represents decoder and prediction head, C1 represents 1X1 convolution channel transformation module, C3 represents 3X3 convolution spatial fusion module, CBRM represents the first branch module, CBAM represents the second branch module, C represents feature concatenation, BN represents batch normalization, Silu represents linear unit activation function, ReLU represents modified linear unit activation function, Maxpool represents max pooling, Upsample represents upsampling, Convlxl represents 1x1 convolution, and Conv3x3 represents 3x3 convolution. Figure 1 In the original RT-DETR, the first feature map S1, the second feature map S2, the third feature map S3, the fourth feature map S4, and the fifth feature map S5 can be extracted. The original RT-DETR's cross-scale contextual feature fusion (CCFF) uses S3 as the shallowest layer for fusion, and uses S3 and S4 for feature fusion module (FUSION). The improved structure moves the FUSION starting point forward to S2, and uses S2, S3, and S4 for FUSION fusion, so that the cross-scale fusion covers a feature layer "closer to the input". S2 has a higher spatial resolution than S3 (smaller stride, larger feature map), and retains more edge, texture, and local structural information. Incorporating it into FUSION enhances the ability to detect small targets at a distance in dense scenes. In addition, the spatial location information is finer, and the boundaries and contours are clearer, which helps to reduce localization errors, especially for small targets, narrow targets, and partially occluded targets.

[0032] This scheme also includes a fusion module (DBFF). DBFF aligns and fuses features from two different sources, then extracts "contextual extension features" and "attention-selected features" through two complementary branches, respectively. Finally, the outputs are merged to obtain fused features that preserve spatial details while possessing semantic discriminative power and significant region focusing capabilities. The fusion module comprises a first branch module (CBRM) and a second branch module (CBAM). CBAM applies channel attention and spatial attention sequentially to adaptively recalibrate features: strengthening target-related expressions and suppressing redundant channels and background regions. Its channel attention determines across the entire image which feature dimensions (channels) are more discriminative for detection, such as target boundaries, structural lines, surface textures, and repetitive patterns. CBAM's spatial attention mechanism further determines which spatial locations should be strengthened based on the features already selected in each channel. In conjunction with channel attention, it first selects "what features are important," then selects "where these features are important." CBAM takes the 1×1 convolutional feature map from DBFF as input and outputs attention-weighted features of the same size as the input. CBRM expands the effective receptive field without changing the output resolution by first downsampling and then upsampling, injecting a larger range of spatial context into each location. CBRM input is the same as CBAM, and the output is a context-enhanced feature of the same size. The corresponding physical logic of fusion is as follows: along two paths, top-down and bottom-up, features of different scales are aligned by upsampling or downsampling, concatenated along the channel dimension, and then integrated through 1×1 convolution (channel transformation) and 3×3 convolution (spatial fusion) to achieve cross-scale feature interaction.

[0033] The RT-DETR model is a deep learning (Transformer) architecture for real-time object detection. It employs an end-to-end process: backbone network extracts features, encoder performs global modeling, dynamically selects high-quality features as queries, and decoder interacts to output predictions. The overall process includes: Backbone—Encoder—QuerySelection—Decoder. The functions of each module are as follows: The backbone network is responsible for extracting multi-scale feature representations from the input image. The network outputs three layers of feature maps, denoted as S3, S4, and S5, where S3 has higher spatial resolution and weaker semantic information, while S5 has lower spatial resolution and stronger semantic information. This module provides a hierarchical feature foundation for subsequent encoding and decoding.

[0034] The encoder consists of two sub-modules: an attention-based internal scale feature interaction module (AIFI) and a CCFF. AIFI processes only the highest-level feature, S5. It captures long-range dependencies within the S5 feature map through a self-attention mechanism, enhancing the contextual modeling capability of high-level semantic features. By limiting attention computation to a single scale, it avoids redundant computation caused by cross-scale global attention, thus achieving a balance between accuracy and speed. The CCFF takes into account features S3, S4, and the AIFI-processed S5. A feature pyramid-like structure is adopted, using multiple fusion blocks to achieve top-down and bottom-up cross-scale information interaction and fusion. The fused multi-scale features are then concatenated and fed into subsequent modules.

[0035] After feature fusion, the Uncertainty-minimal Query Selection module selects the most informative candidate regions from the fused features output by the encoder, based on uncertainty metric criteria, and initializes a set of object queues. The decoder and prediction head, based on a Transformer decoder structure, perform self-attention and cross-attention interactions between the selected object queues and the encoder features, refining the target's location and category predictions layer by layer. Finally, the bounding box coordinates and category labels of the targets are generated, completing end-to-end object detection.

[0036] This solution improves the object detection model (RT-DETR model). The improved RT-DETR model uses ResNet-101 as the backbone network, extracts three levels of features (S2-S4), and uses a lightweight Fusion module to complete the initial multi-scale fusion. The AIFI module functions the same as the original RT-DETR, performing intra-scale self-attention interaction on high-level features (S5) to enhance semantic context modeling capabilities. The CCFF module is also improved. C1, C3, the Fusion Block, and the core improved module DBFF are introduced to form a hierarchical cross-scale fusion structure.

[0037] The C1 module structure includes: 1×1 convolution, normalization, and activation (i.e., Conv1×1+BN+SiLU). It performs linear transformation of the channel dimension and feature dimensionality reduction / upgrading for feature alignment and information compression. The C3 module structure includes: 3×3 convolution, normalization, and activation (i.e., Conv3×3+BN+SiLU). It extracts local spatial features through 3×3 convolution, enhancing the detail representation within the receptive field. DBFF employs a dual-branch parallel fusion strategy, concatenating the two input feature maps, reducing the channel dimension through 1×1 convolution, and then performing a dual-branch parallel fusion strategy. Branch one is CBRM; branch two is CBAM.

[0038] The CBRM module sequentially performs 3×3 convolution, BN, ReLU activation, and max pooling operations, followed by another 3×3 convolution, and finally upsampling to restore resolution. This branch captures a wider range of spatial context through a downsampling-upsampling mechanism, introducing broader spatial context information without significantly increasing parameters, thus enhancing multi-scale spatial awareness. The CBAM module is a plug-and-play module that applies channel attention and spatial attention separately, adaptively enhancing salient features and suppressing redundant information.

[0039] Uncertainty-minimal Query Selection remains consistent with the original RT-DETR Decoder&Head, outputting the final detection result. The core of the original RT-DETR lies in the collaborative design of the AIFI+CCFF efficient hybrid encoder and the uncertainty-minimizing query selection; the improved RT-DETR, while retaining the above framework, upgrades the basic FusionBlock in CCFF to a DBFF dual-branch fusion module. Through the parallel complementarity of CBRM and CBAM, the cross-scale feature fusion capability of the Neck stage is significantly enhanced.

[0040] In addition, it is necessary to determine whether the vehicle is parked in a parking space to avoid misjudgment. Existing parking space databases are usually established on a unified coordinate system. However, during actual inspections using preset flight paths or hand-operated drones, the flight altitude, flight attitude, heading angle, and observation position are all in a dynamic state. The image position, scale, and perspective of the same parking space are different at different inspection times. Therefore, it is impossible to directly use preset parking space coordinates to complete the determination of illegal parking.

[0041] Therefore, this invention also proposes a parking space recovery module, which tracks parking locations detected by an improved RT-DETR model using the BOT-SORT algorithm. It automatically infers and recovers complete parking space areas. This module effectively addresses situations where parking space corners are occluded, thereby significantly improving the overall detection rate.

[0042] Because drones fly continuously, the same parking space will appear repeatedly in multiple video frames from different observation angles. This solution establishes a spatiotemporal buffer queue for parking areas, associating the features of parking spaces detected within several consecutive frames. For parking spaces that are not detected in the current frame due to vehicle occlusion, tree occlusion, changes in lighting, etc., when they are successfully detected in previous or subsequent video frames, the parking space spatial recovery module is used to recover the missing parking spaces. Furthermore, this solution establishes a dynamic credibility model for each parking space. Whenever a parking space is detected again in consecutive videos, the credibility of that parking space is cumulatively updated; if a parking space is not detected for several consecutive frames, but its historical trajectory and spatial location satisfy the parking area continuity constraint, the valid status of that parking space is maintained; only when a parking space cannot be recovered in several consecutive inspection frames, and the road segmentation results indicate that the parking area has undergone structural changes, is the parking space determined to be an invalid parking space. This avoids the problem of parking spaces being mistakenly deleted due to occlusion in a single frame.

[0043] Based on the above embodiments, this application also provides a method for detecting parking in non-motorized vehicle lanes based on unmanned aerial vehicles (UAVs), such as... Figure 2 As shown, the method includes: Step 102: Acquire image data from the UAV, identify the image data, and determine the non-motorized vehicle lane. Step 104: Obtain multi-scale feature representations from the image data to form a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map in ascending order of hierarchy; and determine the first feature information based on the fifth feature map. Step 106: Determine the first intermediate feature, the second intermediate feature, the third intermediate feature, and the fourth intermediate feature based on the second feature map, the third feature map, the fourth feature map, and the first feature information; determine the first fusion feature based on the first intermediate feature and the second intermediate feature; and determine the second fusion feature based on the third intermediate feature and the fourth intermediate feature. Step 108: Determine the aggregated feature based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature; and analyze based on the aggregated feature to determine the vehicle position and parking spot on the non-motorized vehicle lane. Step 110: Perform vehicle violation analysis based on the vehicle position and parking spot, and acquire the vehicle information of the target vehicle through the UAV camera when the target vehicle violates the rules.

[0044] This solution improves the target detection model. The improved model can fuse features at more scales and can fuse features from adjacent layers, resulting in better integration of features across different levels and more accurate identification of vehicle positions and parking spots. Specifically, as an optional embodiment, the image data processing is completed through the target detection model. The first and second fusion features are determined by the fusion module in the target detection model. The fusion module is used to: perform 1×1 convolution processing on the input features to form spliced ​​features, where the input features include the following combinations: a combination of the first and second intermediate features, and a combination of the third and fourth intermediate features; perform spatial information fusion processing on the spliced ​​features using a first branch module to determine the first branch feature; process the spliced ​​features using a second branch module based on spatial attention and channel attention mechanisms to determine the second branch feature; and fuse the first and second branch features to determine the fused feature, which includes either the first or second fusion feature. Specifically, as an optional embodiment, the step of using the first branch module to perform spatial information fusion processing on the spliced ​​features to determine the first branch features includes: sequentially performing 3×3 convolution processing, normalization processing, activation processing, max pooling processing, 3×3 convolution processing, and upsampling processing on the spliced ​​features to capture spatial context information and determine the first branch features.

[0045] The fusion module can utilize two branches to extract spatial information and channel information separately, and then fuse the spatial information. Furthermore, the input features are adjacent layers of features, resulting in stronger feature correlation and more accurate feature extraction and fusion. The fused features are then fused with the top and bottom intermediate features of the original intermediate features to obtain richer aggregated features. Specifically, as an optional embodiment, determining the aggregated features based on the first intermediate feature, the first fused feature, the second fused feature, and the fourth intermediate feature includes: fusing features based on the first and second fused features to determine the fifth intermediate feature; and fusing features based on the first, fourth, and fifth intermediate features to determine the aggregated features.

[0046] After determining the aggregation features, uncertainty measurement can be performed to filter candidate regions. After continuous refinement, vehicle locations and parking spots are determined. Specifically, as an optional embodiment, the analysis based on aggregation features to determine vehicle locations and parking spots on the non-motorized vehicle lane includes: performing uncertainty measurement based on aggregation features to filter candidate regions for each object; querying target objects in the candidate regions to determine target feature data, where target objects include vehicles and parking spaces on the non-motorized vehicle lane; and decoding the target feature data to determine vehicle locations and parking spots on the non-motorized vehicle lane.

[0047] Parking locations can be used to determine parking areas (or parking spaces), and vehicle positions can be used to determine whether a vehicle is within a parking area and whether the vehicle is stationary or in motion. Specifically, as an optional embodiment, the vehicle violation analysis based on vehicle position and parking location includes: determining a parking area on the non-motorized vehicle lane based on the parking location; determining the first vehicle outside the parking area based on the parking area and vehicle position; tracking the first vehicle on the non-motorized vehicle lane; and identifying vehicles whose parking status and travel time are less than a preset time threshold as violating vehicles. Specifically, as an optional embodiment, the step of determining whether a target vehicle is in a parked or driving state includes: obtaining the bounding box parameters corresponding to the target vehicle in multiple consecutive frames of image data, and performing exponential smoothing on the bounding box parameters to determine the smoothed center information and scale information, wherein the scale information is determined based on the width or height of the bounding box parameters; analyzing the background covered by other vehicles outside the target vehicle based on the multiple consecutive frames of image data and generating replacement information to form the background data corresponding to the target vehicle; extracting corner features based on the background data of the image data; analyzing the corner features using optical flow to determine the homography matrix, wherein the homography matrix is ​​used to determine the transformation of background data between adjacent frames; and determining whether the vehicle's state in the multiple consecutive frames of image data is parked or driving based on the homography matrix, the smoothed center information, and the scale information. This solution can smooth the bounding box parameters in multiple consecutive frames of images and analyze background changes, thereby reducing false detections caused by jitter and other reasons. Furthermore, it can analyze changes in the vehicle's position within multiple consecutive frames (e.g., 7 frames) to determine the vehicle's state.

[0048] After determining parking locations, missing locations can be identified and supplemented by performing main direction analysis and distance calculation on multiple locations to more accurately determine the parking area. Alternatively, missing locations can be supplemented based on parking locations in multiple frames of images. Specifically, as an optional embodiment, determining the parking area on the non-motorized vehicle lane based on parking locations includes: determining the main direction corresponding to multiple parking locations and determining the location array corresponding to the main direction; projecting the parking locations in the location array onto the main direction and sorting them; traversing the sorted parking locations and calculating the difference to determine the average distance of the parking locations; supplementing the missing locations based on the average distance, parking locations, and the start and end points of the main direction to form corrected parking locations, and mapping the parking area to form the parking area; the step of supplementing missing locations also includes: acquiring images of adjacent frames of image data and determining the target locations corresponding to the missing locations to supplement the missing locations corresponding to the image data.

[0049] This solution can use an instance segmentation model to identify fixed objects in the road and use preset logical judgment rules to analyze non-motorized vehicle lanes. Specifically, as an optional embodiment, the step of identifying image data and determining non-motorized vehicle lanes includes: identifying roads, flower bed railings, and lane lines in the image data to form extraction results; and analyzing the extraction results and logical judgment rules to determine non-motorized vehicle lanes.

[0050] Based on the above embodiments, this application also provides a non-motorized vehicle lane parking detection device based on an unmanned aerial vehicle (UAV), such as... Figure 3 As shown, the device includes: an image data acquisition module 202, used to acquire image data from a UAV, identify the image data, and determine the non-motorized vehicle lane; a feature map acquisition module 204, used to acquire multi-scale feature representations from the image data, forming a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map in a hierarchy from low to high; and to determine first feature information based on the fifth feature map; an intermediate feature acquisition module 206, used to determine a first intermediate feature, a second intermediate feature, a third intermediate feature, and a fourth intermediate feature based on the second feature map, the third feature map, the fourth feature map, and the first feature information; to determine a first fusion feature based on the first intermediate feature and the second intermediate feature; and to determine a second fusion feature based on the third intermediate feature and the fourth intermediate feature; and an aggregated feature acquisition module 208, used to determine aggregated features based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature, and to analyze the aggregated features to determine the vehicle positions and parking spots on the non-motorized vehicle lane. The vehicle violation analysis module 210 is used to analyze vehicle violations based on vehicle location and parking spot, and to obtain vehicle information of the target vehicle through a drone camera when the target vehicle violates the rules.

[0051] The data processing flow in this embodiment is similar to that in the above method embodiment. For specific implementation details, please refer to the specific implementation details in the above method embodiment. These details will not be repeated here.

[0052] This application can be applied to road traffic management scenarios based on unmanned aerial vehicles (UAVs). It utilizes UAVs to collect road images and employs an improved target detection model to more effectively fuse multi-scale feature maps, thereby more accurately analyzing vehicles and parking spaces in non-motorized vehicle lanes and identifying vehicle violations. Specifically, this solution improves the target detection model, enabling it to fuse features at more scales and integrate features from adjacent layers. This allows for better fusion of features across different levels, resulting in more accurate identification of vehicle positions and parking locations. Furthermore, this solution can analyze the transformation information of vehicles and the background in consecutive frames of image data to identify whether a vehicle is stationary or in motion. Vehicles that are parked or have a travel time shorter than a preset threshold can be identified as violating traffic rules, and this vehicle information is saved using the UAV's camera.

[0053] Based on the above embodiments, this application also provides an electronic device, including: a memory and at least one processor; the memory is used to store computer execution instructions; the at least one processor is used to execute the computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in the above embodiments.

[0054] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0060] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0061] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0062] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0063] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for detecting parking in non-motorized vehicle lanes based on unmanned aerial vehicles (UAVs), characterized in that, The method includes: Acquire image data from the drone, identify the image data, and determine the non-motorized vehicle lane; Multi-scale feature representations are obtained from image data to form a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map with a hierarchy from low to high; and the first feature information is determined based on the fifth feature map. Based on the second feature map, the third feature map, the fourth feature map, and the first feature information, determine the first intermediate feature, the second intermediate feature, the third intermediate feature, and the fourth intermediate feature; based on the first intermediate feature and the second intermediate feature, determine the first fusion feature; based on the third intermediate feature and the fourth intermediate feature, determine the second fusion feature. Based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature, the aggregation feature is determined. Based on the aggregation feature, the vehicle position and parking spot on the non-motorized vehicle lane are determined. Based on vehicle location and parking spot, traffic violations are analyzed, and when a target vehicle violates a traffic rule, vehicle information is obtained through drone cameras.

2. The method according to claim 1, characterized in that, The image data processing is accomplished through a target detection model. The first fusion feature and the second fusion feature are determined based on the fusion module in the target detection model. The fusion module is used for: The input features are processed by 1×1 convolution to form concatenated features. The input features include the following combinations: the combination of the first and second intermediate features, and the combination of the third and fourth intermediate features. The first branch module is used to perform spatial information fusion processing on the splicing features to determine the first branch features; The second branch module uses spatial attention and channel attention mechanisms to process the splicing features and determine the second branch features; The first branch feature and the second branch feature are fused together to determine the fused feature, which includes either the first fused feature or the second fused feature.

3. The method according to claim 2, characterized in that, The step of using the first branch module to perform spatial information fusion processing on the splicing features to determine the first branch features includes: The spliced ​​features are sequentially processed by 3×3 convolution, normalization, activation, max pooling, 3×3 convolution, and upsampling to capture spatial context information and determine the first branch features.

4. The method according to claim 3, characterized in that, The determination of aggregation features based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature includes: Based on the first and second fusion features, feature fusion is performed to determine the fifth intermediate feature; Feature fusion is performed based on the first, fourth, and fifth intermediate features to determine the aggregated features.

5. The method according to claim 4, characterized in that, The analysis based on aggregation features determines the vehicle positions and parking spots on the non-motorized vehicle lane, including: Based on the aggregation features, uncertainty is measured to filter out candidate regions for each object; The candidate region is queried to determine the target feature data. The target objects include vehicles and parking spaces on non-motorized vehicle lanes. The target feature data is decoded to determine the vehicle positions and parking spots on the non-motorized vehicle lane.

6. The method according to claim 1, characterized in that, The analysis of vehicle violations based on vehicle location and parking spot includes: The parking area on the non-motorized vehicle lane is determined based on the parking location, and the first vehicle outside the parking area is determined based on the parking area and the vehicle's position. The system tracks the first vehicle in the non-motorized vehicle lane and identifies vehicles whose parking status and travel time are less than a preset time threshold as violating traffic rules.

7. The method according to claim 6, characterized in that, The steps to determine whether the target vehicle is parked or in motion include: Obtain the rotation box parameters corresponding to the target vehicle in multiple consecutive frames of image data, and perform exponential smoothing on the rotation box parameters to determine the smoothed center information and scale information. The scale information is determined based on the width or height of the rotation box parameters. Based on multiple consecutive frames of image data, the background covered by other vehicles outside the target vehicle is analyzed and replacement information is generated to form the background data corresponding to the target vehicle; corner features are extracted based on the background data of the image data; the corner features are analyzed according to the optical flow method to determine the homography matrix, which is used to determine the transformation of background data between adjacent frames; Based on the homography matrix, smoothed center information, and scale information corresponding to multiple consecutive frames of image data, the state of the vehicle within the multiple consecutive frames of image data is determined to be either a stopped state or a moving state.

8. The method according to claim 1, characterized in that, The determination of parking areas on non-motorized vehicle lanes based on parking locations includes: Determine the main direction corresponding to multiple parking spots, and determine the spot array corresponding to the main direction; Project the parking spots in the spot array onto the main direction and sort them; Traverse the sorted parking spots and calculate the difference to determine the average distance between the parking spots; Based on the average distance, parking spots, and the start and end points of the main direction, missing spots are filled in to form corrected parking spots, and parking areas are mapped to form parking areas. The steps for filling in missing sites also include: acquiring images of adjacent frames of image data and determining the target sites corresponding to the missing sites in order to fill in the missing sites in the image data.

9. The method according to claim 1, characterized in that, The process of identifying image data to determine the non-motorized vehicle lane includes: Identify roads, flower bed railings, and lane lines in image data to generate extraction results; Based on the extraction results and logical judgment rules, the non-motorized vehicle lanes are determined.

10. A non-motorized vehicle lane parking detection device based on unmanned aerial vehicles (UAVs), characterized in that, The device includes: The image data acquisition module is used to acquire image data from the drone, identify the image data, and determine the non-motorized vehicle lane; The feature map acquisition module is used to acquire multi-scale feature representations from image data, forming a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map with a hierarchy from low to high; and to determine the first feature information based on the fifth feature map; The intermediate feature acquisition module is used to determine a first intermediate feature, a second intermediate feature, a third intermediate feature, and a fourth intermediate feature based on the second feature map, the third feature map, the fourth feature map, and the first feature information; to determine a first fusion feature based on the first intermediate feature and the second intermediate feature; and to determine a second fusion feature based on the third intermediate feature and the fourth intermediate feature. The aggregation feature acquisition module is used to determine the aggregation feature based on the first intermediate feature, the first fusion feature, the second fusion feature, and the fourth intermediate feature, and to analyze the aggregation feature to determine the vehicle position and parking spot on the non-motorized vehicle lane. The vehicle violation analysis module is used to analyze vehicle violations based on vehicle location and parking spot, and to obtain vehicle information of the target vehicle through drone camera when the target vehicle violates the rules.

Images