A static traffic routine inspection system and method based on unmanned airport networking
By combining unmanned airport networking and cloud server processing platform, all-weather unmanned management and illegal parking detection of urban road parking spaces have been realized, solving the problems of high management cost and low efficiency in existing technologies, and realizing static traffic management at all times and in all areas.
Patent Information
- Application Number
- CN202311543682.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-23
- Filing Date
- 2023-11-20
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-11-20
AI Technical Summary
Existing technologies cannot effectively manage urban road parking spaces, leading to unfair pricing, increased illegal parking, and high construction and operation and maintenance costs, low inspection efficiency, and inability to achieve static traffic management at all times and in all areas.
The static traffic routine inspection system adopts unmanned airport networking. The unmanned airport provides automatic charging and data transmission for the inspection drones. Combined with the cloud server processing platform, the system processes data, identifies vehicle information and illegal parking, and achieves all-weather unmanned operation and grid management.
It enables all-day, all-area parking space management and illegal parking detection, reduces construction and operation and maintenance costs, improves inspection efficiency, meets the needs of multiple departments, and achieves efficient static traffic management.
Smart Images

Figure CN117558158B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of urban road parking management technology, specifically relating to a static traffic routine inspection system and method based on unmanned airport networking. Background Technology
[0002] With urban development and the continuous increase in the number of vehicles, parking difficulties have become a significant urban problem. Against this backdrop, setting up parking spaces on non-main roads has become an important solution. However, the inherent openness of on-street parking, the difficulty in accurately tracking vehicle entry and exit times, and the inaccurate charging methods can lead to the following problems:
[0003] 1. Unfair pricing and difficulties in managing vehicle entry and exit: The inability to accurately track vehicle entry and exit times leads to inaccurate user payments, resulting in loopholes in pricing or excessive charges, thus creating unfair parking fees; the lack of real-time monitoring also causes considerable trouble when it comes to managing parking spaces.
[0004] 2. Increase in illegal parking: Due to the inability to monitor and charge fees in real time, vehicles may be illegally parked or left behind, causing traffic congestion, affecting normal travel, and creating difficulties for urban traffic management.
[0005] Currently, the main methods for handling on-street parking are divided into manual management and intelligent parking management.
[0006] Manual management involves inspectors using handheld PDAs to record information about vehicles entering parking spaces within their assigned area and charging them upon departure. However, manual management suffers from several drawbacks, including limited coverage areas for toll collectors, difficulty in monitoring toll collection, inability to recover unpaid fees, and extremely high labor costs for operation and management.
[0007] Traditional intelligent parking management detects vehicles by installing parking space locks or geomagnetic devices on the ground of the parking space and records vehicle information through high-mounted license plate recognition devices.
[0008] One known parking management method, such as patent publication number CN202310329871.7, uses a monitoring unit to collect parking status data, calculate fees in real time, and locks the car with a parking lock, unlocking it after receiving payment information. Another known parking management method, such as patent publication number CN202223095144.9, uses two geomagnetic sensors and a photosensitive sensor installed on the roadside parking space for parking space management.
[0009] Although intelligent parking management does not require manual monitoring, each parking space needs to be equipped with devices such as ground magnetic sensors or parking locks, which greatly increases construction and maintenance costs.
[0010] In recent years, the use of user-operated video patrol vehicles for urban road parking inspections has emerged. These devices typically use small electric vehicles as carriers, with cameras mounted on the vehicles to capture images, and users can pay via their mobile phones to promote unmanned parking management. However, these patrol vehicles need to travel on urban roads, and their efficiency is affected by current traffic conditions. Patrol vehicles generally travel at slow speeds, which can lead to traffic congestion during peak hours and even affect the normal passage of other vehicles, thus compromising inspection efficiency.
[0011] Currently, the monitoring methods for illegal parking on roads are: firstly, through patrols by law enforcement officers, which requires a large amount of manpower and resources, is inefficient, and prone to missed detections; and secondly, through monitoring by fixed cameras (such as electronic eyes) deployed on major traffic routes, but these methods generally have blind spots and limited monitoring range, and the cost of deploying and maintaining city-level monitoring equipment is also very high.
[0012] A known method for detecting illegal parking on roads, such as patent publication number CN202310338608.4, involves collecting video data through patrol vehicles; processing the data and determining whether the target vehicle is located in a no-parking zone based on the real-time location of the patrol vehicle; identifying target vehicles that are parked and located in no-parking zones as illegally parked vehicles; and performing license plate recognition on illegally parked vehicles.
[0013] However, patrol vehicles need to travel on city roads, and their efficiency is affected by current road traffic conditions. During peak hours, they are affected by traffic congestion, which makes it impossible to guarantee the efficiency of inspections.
[0014] Therefore, current technology cannot provide a parking management and fee collection method or a method for detecting illegally parked vehicles that has low management labor costs, requires minimal equipment installation, and has low construction, operation, and maintenance costs. Furthermore, there is no method that integrates parking management and illegal parking detection. This means that more innovation and development are needed to find a more effective static traffic management system. Summary of the Invention
[0015] To address the shortcomings of existing technologies, this invention provides a static traffic routine inspection system and method based on unmanned airport networking, which can effectively solve the above-mentioned problems.
[0016] The technical solution adopted in this invention is as follows:
[0017] This invention provides a static traffic routine inspection system based on unmanned airport networking, including a front-end data acquisition system, a cloud server processing platform, and a user terminal system;
[0018] The front-end acquisition system includes multiple front-end acquisition nodes; each front-end acquisition node includes an unmanned aerial vehicle (UAV) farm and multiple inspection UAVs. The UAV farm is used for automatic charging, maintenance, and data transmission of each inspection UAV. The inspection UAVs are used for periodic inspections according to planned routes to acquire inspection images.
[0019] The cloud server processing platform includes a drone network management module and a data processing module. The drone network management module is used to deploy various drone airports to form a drone airport network. The data processing module is used to process the inspection images collected by the inspection drones, identify parking information and illegal parking information at different inspection locations, and send them to the user terminal system.
[0020] The user terminal system includes a traffic management department terminal system and a parking management department terminal system, which are used to receive the parking information and illegal parking information sent by the cloud server processing platform.
[0021] Preferably, the unmanned airport includes a fast charging module, a monitoring camera module, a weather station module, a network module, an omnidirectional antenna module, and an RTK module;
[0022] The fast charging module is used to quickly charge the inspection drone;
[0023] The monitoring camera module is used to monitor the environment around the unmanned airport;
[0024] The weather station module is used to monitor wind speed, rainfall, and temperature information around the unmanned airport;
[0025] The network module can be connected to the network via a network cable or by installing a 4G / 5G module.
[0026] The omnidirectional antenna module is used to communicate with the inspection drone and receive or send signals;
[0027] The RTK module is used for high-precision navigation and positioning of the inspection drone;
[0028] Using unmanned parking lots as forward nodes in the inspection system, these unmanned parking lots are deployed in different areas. By networking these unmanned parking lots, parking spaces can be managed in a grid-like manner, thereby achieving full coverage.
[0029] Preferably, the inspection drone is equipped with a gimbal camera and a high-precision positioning module.
[0030] This invention also provides a method for a static traffic routine inspection system based on unmanned airport networking, comprising the following steps:
[0031] Step 1: The inspection drone performs autonomous inspections at pre-planned inspection paths at regular intervals. During the inspection process, the data processing module of the cloud server processing platform receives the drone status information and collected inspection images of the inspection drone in real time. The drone status information includes the inspection drone's GPS information and drone flight parameters.
[0032] Step 2: The data processing module pre-divides the inspection area map into three types of areas: non-identification area, illegal parking area, and parking area; wherein, the non-identification area is an area where vehicle detection is not required; the illegal parking area is a no-parking area; and the parking area includes normal parking spaces and yellow-lined no-parking spaces.
[0033] The data processing module determines the current location of the inspection drone in real time based on its GPS information and the inspection area map. If it is an unrecognized area, the received inspection image is not processed. If it is an illegal parking area, the license plate detection module is activated to identify the vehicle's license plate number from the inspection image and mark it as illegally parked. If it is a parked area, both the license plate detection module and the illegal parking detection module are activated simultaneously. The license plate detection module identifies the vehicle's license plate number from the inspection image, and the illegal parking detection module further identifies whether the vehicle is in a normal parking space or a yellow-line no-parking space. If it is in a normal parking space, it is marked as normally parked; if it is in a yellow-line no-parking space, it is marked as illegally parked.
[0034] Therefore, the data processing module analyzes the vehicle's license plate number and parking status information in real time. The parking status information includes normal parking status and illegal parking status. Then, it integrates the data collection time and GPS information of the inspection drone to obtain the vehicle inspection results.
[0035] Step 3: The data processing module sends the vehicle inspection results to the user terminal system in real time.
[0036] Preferably, the license plate detection module is specifically used for:
[0037] Step A1: Construct a cascaded license plate image detection network and a license plate character detection network;
[0038] Step A2: Use the license plate image training sample set to train the license plate image detection network to obtain the trained license plate image detection network;
[0039] Step A3: Use the license plate character training sample set to train the license plate character detection network to obtain the trained license plate character detection network;
[0040] Step A4: Input the inspection image to be detected into the trained license plate image detection network. The trained license plate image detection network identifies and marks the license plate detection box in the inspection image, and then transmits the inspection image with the marked license plate detection box to the trained license plate character detection network.
[0041] Step A5: The trained license plate character detection network performs license plate character recognition within the license plate detection frame to obtain the license plate number information.
[0042] Preferably, step A2 specifically includes:
[0043] Step A2.1: Obtain vehicle images taken from multiple drone perspectives and open-source vehicle images;
[0044] Step A2.2: Establish an image quality evaluation function and use the image quality evaluation function to evaluate the quality of the vehicle images. If the image is of low quality, it is discarded; if the image is of high quality, it is retained, thereby obtaining multiple high-quality vehicle images.
[0045] Step A2.3: In each high-quality vehicle image, mark the smallest bounding rectangle of the license plate as the license plate annotation box;
[0046] Step A2.4: Preprocess each vehicle image with a license plate marking frame, adjust it to a uniform size, and then perform gridding to form multiple cells, thus obtaining the processed vehicle image.
[0047] Step A2.5: Input the processed vehicle image into the license plate image detection network. The license plate image detection network predicts the probability value of each cell belonging to a license plate, denoted as Pr. Then, filter out cells with probability values Pr greater than a set threshold to obtain a set of filtered cells.
[0048] Step A2.6: Identify the overall boundary of the area covered by the filtered cell set, which is the license plate prediction box; then, calculate the overlap between the license plate prediction box and the license plate annotation box, denoted as the overlap ratio IOU; based on the overlap ratio IOU, calculate the loss value using a loss function. If the loss value is less than a set value, the training of the license plate image detection network ends, and a trained license plate image detection network is obtained; otherwise, adjust the parameters of the license plate image detection network according to the loss value to obtain an updated license plate image detection network, and then continue to train the updated license plate image detection network using the remaining license plate image training samples, iterating continuously until a trained license plate image detection network is obtained.
[0049] Preferably, step A3 specifically includes:
[0050] Step A3.1: Construct a license plate character training sample set; the license plate character training sample set includes multiple license plate character training samples, each license plate character training sample is a license plate character image with the smallest rectangular frame of the license plate character, and the license plate character content is labeled;
[0051] Step A3.2: Input the license plate character training samples into the license plate character detection network. The license plate character detection network performs feature extraction and text transcription on the license plate character training samples, specifically as follows:
[0052] Step A3.2.1, the license plate character detection network converts the license plate character training samples into a license plate character image T1 with height H * width W * number of channels 3;
[0053] Step A3.2.2: The license plate character detection network uses a patch partitioning method to divide the license plate character image T1 into n small image blocks T2 with a height of H / 4 and a width of W / 4.
[0054] Step A3.2.3: The license plate character detection network extracts image features for each image patch T2 through convolutional layers; wherein, the image features include texture, shape, and edge features; then, a batch normalization processing module is used to normalize the image features of the image patch T2 to obtain normalized image features; then, an activation function is used to perform a non-linear mapping on the normalized image features to obtain the image patch feature vector;
[0055] Step A3.2.4, the license plate character detection network adopts a three-stage feature module to extract local and global features of the feature vectors of each small image block, and then extracts the local features of the license plate characters and the dependencies between license plate characters to obtain the license plate character feature C;
[0056] Specifically, positional information is embedded into the feature vector of each image patch, where the positional information refers to the relative position of the image patch feature vector within the license plate character image T1. Then, the first-stage feature module perceives the dependencies between all characters through the global fusion submodule and extracts global features; it perceives the correlations between all characters within a predefined receptive field through the local fusion submodule, extracts local features, encodes character morphological features, and establishes associations between characters, thereby simulating stroke features crucial for character recognition. The first-stage feature module extracts the first-stage character features. Then, the second-stage feature module further extracts character features from the first-stage features to obtain the second-stage features. Finally, the third-stage feature module further extracts character features from the second-stage features to obtain the third-stage features. Through repeated application of the three-stage feature module, the license plate character feature C is finally extracted.
[0057] Step A3.2.5: The license plate character detection network classifies the license plate character features C using a parallel linear classifier to identify the license plate characters. Each license plate character has a confidence level.
[0058] Based on the confidence level, a loss function is used to calculate the loss value. If the loss value is less than a set value, the training of the license plate character detection network ends, and a trained license plate character detection network is obtained. Otherwise, the parameters of the license plate character detection network are adjusted according to the loss value to obtain an updated license plate character detection network. For the updated license plate character detection network, the remaining license plate character sample images are used for further training. This process is iterated until a trained license plate character detection network is obtained.
[0059] Preferably, the illegal parking detection module includes two stages: vehicle detection and yellow line detection, specifically used for:
[0060] Step B1: Using a vehicle detection network, identify vehicle identification boxes from the inspection image. The height of the vehicle identification box is h.
[0061] With the top left corner of the inspection image as the origin O, the horizontal rightward direction as the positive x-axis, and the vertical downward direction as the positive y-axis, an image coordinate system is established to obtain the coordinates (x0, y0) of the center point P0 of the vehicle recognition frame.
[0062] Step B2: The intersection of the horizontal line passing through the center point P0 and the left boundary of the vehicle recognition frame is point P1(x1, y1). Let the length of P0P1 be d0. Shift point P1 to the left by the length d to obtain point P2. The value of d is as follows.
[0063]
[0064] Set the height with P0 and P2 as the axis of symmetry The rectangular ROI region;
[0065] Step B3: Perform HSV color gamut conversion and edge extraction on the rectangular ROI region to extract a yellow line segment with a length of L. If the length L of the yellow line segment is greater than the threshold, it is determined to be a no-parking space with a yellow line, and the vehicle is illegally parked; if it is less than the threshold, it is determined to be a normal parking space, and the vehicle is normally parked.
[0066] The static traffic routine inspection system and method based on unmanned airport networking provided by this invention has the following advantages:
[0067] Unmanned aerial vehicle (UAV) airports, acting as forward nodes, are deployed across various urban areas to automatically charge, maintain, and transmit data for inspection drones, enabling 24 / 7 unmanned operation. Through a network of airport nodes, grid-based management of urban parking spaces is achieved, providing full-time, all-area coverage. A cloud server processing platform, leveraging its powerful computing and storage capabilities, enables efficient static traffic management. A single drone patrol can meet the needs of multiple departments, maximizing drone utilization. Attached Figure Description
[0068] Figure 1 The structural framework diagram of the static traffic routine inspection system based on unmanned airport networking provided by the present invention;
[0069] Figure 2 A road area division map for a static traffic routine inspection system based on unmanned airport networking provided by the present invention;
[0070] Figure 3 A flowchart of a static traffic routine inspection system based on unmanned airport networking provided by the present invention;
[0071] Figure 4 An algorithm framework diagram of a static traffic routine inspection system based on unmanned airport networking provided by the present invention;
[0072] Figure 5 The confusion matrix diagram of a static traffic routine inspection system based on unmanned airport networking provided by the present invention;
[0073] Figure 6 A license plate recognition network structure diagram for a static traffic routine inspection system based on unmanned airport networking provided by the present invention;
[0074] Figure 7 This is a schematic diagram illustrating the vehicle detection and yellow line detection principles provided by the present invention. Detailed Implementation
[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0076] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.
[0077] This invention provides a static traffic routine inspection system and method based on unmanned aerial vehicle (UAV) airport networking. By setting up multiple UAV airports in various urban road sections, the system uses UAVs for regular inspections, while simultaneously charging the UAVs and communicating with a cloud server processing platform. The system also utilizes the data processing module of the cloud server processing platform to process data, thereby solving the problem that existing technologies cannot effectively manage urban parking spaces.
[0078] This invention discloses a static traffic routine inspection system based on unmanned aerial vehicle (UAV) airport networking, comprising a front-end data acquisition system, a cloud server processing platform, and a user terminal system. The front-end data acquisition system consists of multiple inspection UAVs and unmanned aerial vehicle (UAV) airports. The cloud server processing platform includes a UAV network management module and a data processing module. The inspection UAVs of this invention conduct scheduled inspections along planned routes, uploading the collected data to the cloud server processing platform. After processing by the cloud server, the data is sent to the user terminal. Unmanned aerial vehicle (UAV) airports, acting as front-end nodes, are deployed across various urban road sections to automatically charge, maintain, and transmit data for the inspection UAVs, achieving 24 / 7 unmanned operation. Through airport node networking, grid-based management of urban parking spaces is achieved, providing full-time and all-area coverage. The cloud server processing platform, relying on powerful computing and storage capabilities, enables efficient static traffic management. A single UAV patrol can meet the needs of multiple departments, achieving high utilization of the UAVs.
[0079] Specifically, this invention provides a static traffic routine inspection system based on unmanned airport networking, combined with... Figures 1-6 It includes a front-end data acquisition system, a cloud server processing platform, and a user terminal system.
[0080] The following is a detailed introduction to the front-end data acquisition system, cloud server processing platform, and user terminal system:
[0081] (I) Front-end data acquisition system
[0082] The front-end acquisition system includes multiple front-end acquisition nodes; each front-end acquisition node includes an unmanned aerial vehicle (UAV) farm and multiple inspection UAVs. The UAV farm is used for automatic charging, maintenance, and data transmission of each inspection UAV. The inspection UAVs are used for periodic inspections according to planned routes to acquire inspection images.
[0083] In this invention, the inspection drone can be a lightweight multi-rotor drone equipped with a gimbal camera, a high-precision positioning module, obstacle avoidance sensors, etc.; the gimbal camera can achieve wide-angle, zoom, and infrared shooting functions. The inspection drone collects video or image data of vehicles on the roadside according to a fixed safe route planned by the drone networking management module of the cloud server processing platform. After each patrol is completed or when the energy is low, it automatically returns to the drone airport for charging.
[0084] The inspection drone is equipped with a gimbal camera located at the head of the drone to photograph the parking space; the positioning module on the inspection drone is used to locate the drone and determine the specific parking location.
[0085] The unmanned airport includes a fast charging module, a monitoring camera module, a weather station module, a network module, an omnidirectional antenna module, and an RTK module;
[0086] The fast charging module is a device specifically designed for the rapid charging of inspection drones. Utilizing advanced charging technology and equipment, this module enables the drone to charge quickly and efficiently. By connecting the drone and the charging module, the drone can obtain sufficient energy in a short time, allowing it to operate continuously and significantly improving overall work efficiency.
[0087] The surveillance camera module is used to monitor the environment around the airport.
[0088] The weather station module is used to monitor wind speed, rainfall, and temperature information around the airport.
[0089] The network module can be connected to the network via a network cable or by installing a 4G / 5G module.
[0090] The omnidirectional antenna is used to communicate with the inspection drone and to receive or send signals;
[0091] The RTK module is used for high-precision navigation and positioning.
[0092] In this invention, unmanned airports serve as the forward nodes of this inspection system. The unmanned airports are deployed in urban areas, and urban parking spaces are managed in a grid manner through airport node networking, thereby achieving full coverage of the city. The inspection routes of the inspection drones in the designated areas are planned through a cloud platform.
[0093] The drone transmits captured images to the cloud server, or the drone transmits video information to the cloud server in real time, and then the cloud server's data processing module processes the data.
[0094] (II) Cloud Server Processing Platform
[0095] The cloud server processing platform includes a drone network management module and a data processing module. The drone network management module is used to deploy various drone airports to form a drone airport network. The data processing module is used to process the inspection images collected by the inspection drones, identify parking information and illegal parking information at different inspection locations, and send them to the user terminal system.
[0096] Specifically, the drone network management module is used to rationally arrange the drone airport network nodes to ensure the rationality of the planned area. At the same time, it plans the inspection path of the inspection drones according to the arrangement of each drone airport network node, which can comprehensively coordinate the entire network, realize grid management of some areas of the city, plan the working time of the drones to ensure all-time inspection, monitor the status of the drones, and receive the status information of the drones in real time.
[0097] The data processing module is used to divide the inspection route into unidentified areas, illegal parking areas, and parking areas based on road conditions. The parking areas include both regular parking spaces and yellow no-parking zones. Figure 2 As shown;
[0098] The data processing module is mainly used to process data, including images or real-time video data transmitted by inspection drones and drone airports, as well as data from each network node.
[0099] (III) User Terminal System
[0100] The user terminal system includes a traffic management department terminal system and a parking management department terminal system, which are used to receive the parking information and illegal parking information sent by the cloud server processing platform.
[0101] Specifically, the user terminal system receives road traffic information from the cloud platform, and relevant departments process it promptly to resolve corresponding issues immediately. Specifically, the parking management department collects parking fees based on the parking space availability, parking time, license plate information, and parking location information received by the user terminal system from the cloud server processing platform's data processing module. The traffic management department processes and resolves issues based on the license plate information, parking location, and images of illegally parked vehicles received by the terminal system from the cloud server processing platform's data processing module.
[0102] This invention also provides a method for a static traffic routine inspection system based on unmanned airport networking, comprising the following steps:
[0103] Step 1: The inspection drone performs autonomous inspections at pre-planned inspection paths at regular intervals. During the inspection process, the data processing module of the cloud server processing platform receives the drone status information and collected inspection images of the inspection drone in real time. The drone status information includes the inspection drone's GPS information and drone flight parameters.
[0104] Step 2, the data processing module pre-divides the inspection area map into three area types, referring to... Figure 2The three zones are: a non-identification zone, an illegal parking zone, and a parking zone; wherein, the non-identification zone is an area where vehicle detection is not required; the illegal parking zone is a no-parking zone; and the parking zone includes regular parking spaces and yellow-lined no-parking spaces.
[0105] The data processing module determines the current location of the inspection drone in real time based on its GPS information and the inspection area map. Based on the drone's location, the data processing module activates different algorithms to process the image. The algorithm modules used in different task segments are shown in Table 1. Specifically, if the area is not recognized, the received inspection image is not processed. If it is an illegal parking area, the license plate detection module is activated to identify the vehicle's license plate number from the inspection image and mark it as illegally parked. If it is a parking area, both the license plate detection module and the illegal parking detection module are activated simultaneously. The license plate detection module identifies the vehicle's license plate number from the inspection image, and the illegal parking detection module further identifies whether the vehicle is in a normal parking space or a yellow-line no-parking space. If it is in a normal parking space, it is marked as normally parked; if it is in a yellow-line no-parking space, it is marked as illegally parked.
[0106] Therefore, the data processing module analyzes and obtains the vehicle's license plate number information and vehicle parking status information in real time. The vehicle parking status information includes normal parking status and illegal parking status. Then, it integrates the collection time and GPS information of the inspection drone to obtain the vehicle inspection results. The vehicle inspection results are sent to the user terminal system in real time, and each user terminal system processes the information in a timely manner.
[0107] The data processing module uses two algorithms: one for license plate detection and the other for illegal parking detection. The main approach is as follows: First, the license plate detection module divides the image of parked vehicles into license plate regions, determines the location of the license plate, and identifies and extracts it. Then, the extracted license plate is segmented and matched to obtain license plate number information labeled with confidence levels. Finally, the identified license plate number information, GPS information returned by the drone, time information, and inspection area information are fused together, annotated on the image, and a formatted file is output.
[0108] In this invention, the license plate detection module is specifically used for:
[0109] The license plate detection module performs license plate detection in two stages: license plate area capture and license plate number recognition. Specifically, it includes the following steps:
[0110] Step A1: Construct a cascaded license plate image detection network and a license plate character detection network;
[0111] Step A2: The license plate image detection network is trained using a training sample set of license plate images to obtain a trained license plate image detection network; wherein, the license plate image detection network can adopt a feature pyramid structure deep learning network with residual attention mechanism.
[0112] Step A2 is as follows:
[0113] Step A2.1: Obtain vehicle images taken from multiple drone perspectives and open-source vehicle images;
[0114] Step A2.2: Establish an image quality evaluation function and use the image quality evaluation function to evaluate the quality of the vehicle images. If the image is of low quality, it is discarded; if the image is of high quality, it is retained, thereby obtaining multiple high-quality vehicle images.
[0115] In this invention, the image quality evaluation function adopts a combination of subjective and objective evaluation methods, and the formula is as follows:
[0116]
[0117] In the formula, δ1, δ2, and δ3 represent the proportions of various evaluation factors, δ1 < δ2 < δ3 and δ1 + δ2 + δ3 = 1, F i Z is the score given to the image in the i-th subjective evaluation. i Let χ be the score given to the image in the i-th objective evaluation, where χ represents the peak signal-to-noise ratio of the image, N is the number of subjective evaluations, and M is the number of objective evaluations.
[0118] Step A2.3: In each high-quality vehicle image, mark the smallest bounding rectangle of the license plate as the license plate annotation box;
[0119] Step A2.4: Preprocess each vehicle image with a license plate marking frame, adjust it to a uniform size, for example, 448 pixels * 448 pixels, and then perform gridding to form multiple cells, for example, S*S cells, thus obtaining the processed vehicle image.
[0120] Step A2.5: Input the processed vehicle image into the license plate image detection network. The license plate image detection network predicts the probability value of each cell belonging to a license plate, denoted as Pr. Then, filter out cells with probability values Pr greater than a set threshold to obtain a set of filtered cells.
[0121] Wherein, the probability value Pr can be calculated using accuracy, precision, and recall, all of which are derived from the confusion matrix. Figure 5 As shown.
[0122] Step A2.6: Identify the overall boundary of the area covered by the filtered cell set, which is the license plate prediction box; then, calculate the overlap between the license plate prediction box and the license plate annotation box, denoted as the overlap ratio IOU; based on the overlap ratio IOU, calculate the loss value using a loss function. If the loss value is less than a set value, the training of the license plate image detection network ends, and a trained license plate image detection network is obtained; otherwise, adjust the parameters of the license plate image detection network according to the loss value to obtain an updated license plate image detection network, and then continue to train the updated license plate image detection network using the remaining license plate image training samples, iterating continuously until a trained license plate image detection network is obtained.
[0123] Step A3: Use the license plate character training sample set to train the license plate character detection network to obtain the trained license plate character detection network;
[0124] Step A3 is as follows:
[0125] Step A3.1: Construct a license plate character training sample set; the license plate character training sample set includes multiple license plate character training samples, each license plate character training sample is a license plate character image with the smallest rectangular frame of the license plate character, and the license plate character content is labeled;
[0126] Step A3.2: Input the license plate character training samples into the license plate character detection network. The license plate character detection network is a convolutional neural network, with the structure as follows: Figure 6 As shown. The license plate character detection network performs feature extraction and text transcription on the license plate character training samples, specifically as follows:
[0127] Step A3.2.1, the license plate character detection network converts the license plate character training samples into a license plate character image T1 with height H * width W * number of channels 3;
[0128] Step A3.2.2: The license plate character detection network uses a patch embedding method to segment the license plate character image T1 into n small image blocks T2 with a height of H / 4 and a width of W / 4.
[0129] Step A3.2.3: The license plate character detection network extracts image features for each image patch T2 through convolutional layers; wherein, the image features include texture, shape, and edge features; then, a batch normalization (BN) module is used to normalize the image features of the image patch T2, making the network training more stable, accelerating the network convergence process, and enhancing the network's robustness and generalization performance, resulting in normalized image features; then, an activation function is used to perform a non-linear mapping on the normalized image features, enabling the network to learn more complex and abstract features, thereby improving the network's ability to understand and recognize text in images, resulting in image patch feature vectors;
[0130] Step A3.2.4: The license plate character detection network employs a three-stage feature module to extract local and global features from the feature vectors of each image patch. This allows for the extraction of local features of the license plate characters and the dependencies between them, resulting in license plate character feature C. Each stage in the three-stage feature module consists of a series of mixing blocks and merging or combinating modules. Local and global mixing blocks are used to extract local text features and dependencies between elements. The feature vectors of each image patch are processed by the three-stage feature module to extract character element features and dependencies at different distances and scales. After the third stage, the feature size is... Represented by the symbol C, specifically:
[0131] Positional information, referring to the relative position of the image patch feature vector within the license plate character image T1, is embedded in the feature vector of each image patch. Then, in the first-stage feature module, the global mixing submodule perceives the dependencies between all characters and extracts global features; the local mixing submodule perceives the correlations between all characters within a predefined receptive field and extracts local features, encoding character morphological features and establishing associations between characters, thereby simulating stroke features crucial for character recognition. Local and global feature extraction are implemented using LM and MLP networks. The first-stage feature module extracts the first-stage character features; the second stage follows the same process. These two stages are repeatedly applied for comprehensive feature extraction. Specifically, the second-stage feature module further extracts character features from the first-stage features to obtain the second-stage features; then, the third-stage feature module further extracts character features from the second-stage features to obtain the third-stage features. Through repeated application of the three-stage feature module, the license plate character feature C is finally extracted.
[0132] If the three stages maintain a constant spatial resolution This would lead to high computational costs and redundancy in feature representation. This invention, through high-downsampling convolution, not only reduces computational costs but also constructs a hierarchical structure adapted to text images. It increases the number of channels (d). i =2*d i-1 To compensate for the information loss caused by height downsampling, in the third stage, the height pooling is set to 1, followed by non-linear activation and dropout units. Thus, the character components are further compressed into a feature sequence, with sequence elements represented by features of dimension D3.
[0133] Step A3.2.5: The license plate character detection network classifies the license plate character features C using a parallel linear classifier to identify the license plate characters. Each license plate character has a confidence level.
[0134] Based on the confidence level, a loss function is used to calculate the loss value. If the loss value is less than a set value, the training of the license plate character detection network ends, and a trained license plate character detection network is obtained. Otherwise, the parameters of the license plate character detection network are adjusted according to the loss value to obtain an updated license plate character detection network. For the updated license plate character detection network, the remaining license plate character sample images are used for further training. This process is iterated until a trained license plate character detection network is obtained.
[0135] The illegal parking detection module is used to detect illegally parked vehicles in the parking area, including two stages: vehicle detection and yellow line detection. (Refer to...) Figure 7 Specifically used for:
[0136] Step B1: Using a vehicle detection network, identify vehicle identification boxes from the inspection image. The height of the vehicle identification box is h.
[0137] With the top left corner of the inspection image as the origin O, the horizontal rightward direction as the positive x-axis, and the vertical downward direction as the positive y-axis, an image coordinate system is established to obtain the coordinates (x0, y0) of the center point P0 of the vehicle recognition frame.
[0138] Among them, a vehicle detection network with a high recognition rate was obtained by using a self-built high-altitude vehicle dataset to train the Darknet53 network.
[0139] Step B2: The intersection of the horizontal line passing through the center point P0 and the left boundary of the vehicle recognition frame is point P1(x1, y1). Let the length of P0P1 be d0. Shift point P1 to the left by the length d to obtain point P2. The value of d is as follows.
[0140]
[0141] Set the height with P0 and P2 as the axis of symmetry This rectangular ROI region can be used to narrow the detection range of the yellow line segment, improve the detection efficiency of the yellow line segment, reduce hardware requirements, and lower system costs.
[0142] Step B3: Perform HSV color gamut conversion and edge extraction on the rectangular ROI region to extract a yellow line segment with a length of L. If the length L of the yellow line segment is greater than the threshold, it is determined to be a no-parking space with a yellow line, and the vehicle is illegally parked; if it is less than the threshold, it is determined to be a normal parking space, and the vehicle is normally parked.
[0143] Step A4: Input the inspection image to be detected into the trained license plate image detection network. The trained license plate image detection network identifies and marks the license plate detection box in the inspection image, and then transmits the inspection image with the marked license plate detection box to the trained license plate character detection network.
[0144] Step A5: The trained license plate character detection network performs license plate character recognition within the license plate detection frame to obtain the license plate number information.
[0145] Step 3: The data processing module sends the vehicle inspection results to the user terminal system in real time.
[0146] The following is an example:
[0147] Example:
[0148] like Figure 2 The diagram shows the structural framework of a routine static traffic inspection system based on unmanned aerial vehicle (UAV) airport networking. The system comprises three main parts: a front-end data acquisition system, a cloud server processing platform, and a user terminal system. The front-end data acquisition system includes multiple inspection UAVs and UAV airports; the cloud server processing platform includes a UAV networking management module and a data processing module; and the user terminal system includes traffic management departments and parking management departments.
[0149] The inspection drone is equipped with a gimbal camera and a high-precision positioning module. It conducts regular inspections in designated areas, collects information on parking spaces, illegal parking, and road conditions, and transmits the images captured by the inspection drone to the drone airport. The drone airport then transmits the images back to the cloud server processing platform, or the drone transmits the captured data to the cloud server processing platform in real time. After the inspection, the drone returns to the airport for charging and maintenance.
[0150] The inspection drone is equipped with a gimbal camera that takes pictures of the parking space from the drone's nose; the high-precision positioning module on the inspection drone is used to locate the drone and determine the specific parking location.
[0151] The data processing module of the cloud server processing platform analyzes the image data collected by the inspection drone to obtain vehicle detection information. The vehicle detection information is combined with GPS information, license plate number information and time information from license plate detection and recognition to annotate the images, form a formatted file, and send it to the management department.
[0152] When user terminals receive road traffic information from the data processing module of the cloud server processing platform, relevant departments process it promptly and resolve the corresponding issues immediately.
[0153] By combining and applying the above algorithms, illegal parking in parking areas can be detected and identified in a timely manner, thereby improving the efficiency of static traffic management.
[0154] The flowchart of the static traffic routine inspection based on unmanned airport networking in this invention is as follows: Figure 3 As shown.
[0155] Based on the established inspection area, target function and environmental conditions, the target patrol route of each inspection drone is planned through the drone networking management module of the cloud server. The inspection area can be divided into vehicle non-recognition area, illegal parking area and parking area.
[0156] Determine whether replanning is needed based on the planned route. Route planning generally involves the following steps: defining the aerial survey range, determining the flight altitude, confirming the overlap, and completing the route planning.
[0157] The drone airport transmits the target cruise route to the inspection drones parked at the airport.
[0158] The inspection drone patrols according to the received route and acquires information about parked vehicles in the inspection area using its onboard gimbal camera. This vehicle information includes video and images. Finally, the inspection drone transmits the collected image data to the drone farm; the drone farm then uploads the data to a cloud server processing platform, or the drone directly transmits video data to the cloud server processing platform. The data processing module then processes the collected data.
[0159] After the cloud server data processing module processes the information and performs comprehensive analysis, it sends the parking space status information to the parking management department and the traffic violation information to the traffic management department; that is, it obtains parking information and illegal parking information, and sends them to the parking management department and the traffic management department respectively.
[0160] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:
[0161] 1. Unmanned management: Drones can take off and land automatically at airports, recharge quickly, and return to their flight path to continue patrolling. This solves the problem of insufficient battery life of multi-rotor drones, which prevents them from continuously carrying out on-site operations, and also avoids the trouble of sending a large number of administrators to the site.
[0162] 2. Routine inspections: Drones paired with unmanned aerial vehicles (UAVs) can achieve rapid charging, and their high-precision remote control and navigation systems ensure accurate operation and positioning during inspections. Equipped with cameras or infrared thermal imagers, they can effectively acquire relevant data about the inspection area, ultimately enabling routine inspections of static traffic.
[0163] 3. Data transmission between drones and cloud servers is ensured to be real-time. The cloud server's data processing module then processes the data and makes it available to user terminals.
[0164] 4. Use drone grid management, set up drone airports in different areas, and have each drone manage an area. This makes drone inspections more efficient and reduces management costs.
[0165] 5. Remote control and management system: By using a cloud platform to schedule the working time of drones and adjust the working time interval between every two drones, it is possible to carry out inspections around the clock all day long.
[0166] 6. Automatic obstacle avoidance for drones: Drones autonomously avoid obstacles using obstacle avoidance sensors located on the upper, lower, left, and right sides of their fuselage.
[0167] 7. The visual inspection system can perform real-time inspection of images captured by drones, identify license plate numbers and combine them with location information, and finally generate a report, which is transmitted to the cloud server for user terminals to use;
[0168] 8. Compared to video patrol vehicles, inspection drones can operate from the air, have a wider shooting angle, can patrol a wider area, and avoid traffic congestion during peak hours;
[0169] 9. Compared to manual management, drones can operate in harsh environments such as severe winters and scorching summers, ensuring the safety of staff.
[0170] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for a static traffic routine inspection system based on unmanned airport networking, characterized in that, The system includes a front-end data acquisition system, a cloud server processing platform, and a user terminal system. The front-end acquisition system includes multiple front-end acquisition nodes; each front-end acquisition node includes an unmanned aerial vehicle (UAV) farm and multiple inspection UAVs. The UAV farm is used for automatic charging, maintenance, and data transmission of each inspection UAV. The inspection UAVs are used for periodic inspections according to planned routes to acquire inspection images. The cloud server processing platform includes a drone network management module and a data processing module. The drone network management module is used to deploy various drone airports to form a drone airport network. The data processing module is used to process the inspection images collected by the inspection drones, identify parking information and illegal parking information at different inspection locations, and send them to the user terminal system. The user terminal system includes a traffic management department terminal system and a parking management department terminal system, which are used to receive the parking information and illegal parking information sent by the cloud server processing platform; The method for the static traffic routine inspection system based on unmanned airport networking includes the following steps: Step 1: The inspection drone performs autonomous inspections at pre-planned inspection paths at regular intervals. During the inspection process, the data processing module of the cloud server processing platform receives the drone status information and collected inspection images of the inspection drone in real time. The drone status information includes the inspection drone's GPS information and drone flight parameters. Step 2: The data processing module pre-divides the inspection area map into three types of areas: non-identification area, illegal parking area, and parking area; wherein, the non-identification area is an area where vehicle detection is not required; the illegal parking area is a no-parking area; and the parking area includes normal parking spaces and yellow-lined no-parking spaces. The data processing module determines the current location of the inspection drone in real time based on its GPS information and the inspection area map. If it is an unrecognized area, the received inspection image is not processed. If it is an illegal parking area, the license plate detection module is activated to identify the vehicle's license plate number from the inspection image and mark it as illegally parked. If it is a parked area, both the license plate detection module and the illegal parking detection module are activated simultaneously. The license plate detection module identifies the vehicle's license plate number from the inspection image, and the illegal parking detection module further identifies whether the vehicle is in a normal parking space or a yellow-line no-parking space. If it is in a normal parking space, it is marked as normally parked; if it is in a yellow-line no-parking space, it is marked as illegally parked. Therefore, the data processing module analyzes the vehicle's license plate number and parking status information in real time. The parking status information includes normal parking status and illegal parking status. Then, it integrates the data collection time and GPS information of the inspection drone to obtain the vehicle inspection results. Step 3: The data processing module sends the vehicle inspection results to the user terminal system in real time. The license plate detection module is specifically used for: Step A1: Construct a cascaded license plate image detection network and a license plate character detection network; Step A2: Use the license plate image training sample set to train the license plate image detection network to obtain the trained license plate image detection network; Step A3: Use the license plate character training sample set to train the license plate character detection network to obtain the trained license plate character detection network; Step A4: Input the inspection image to be detected into the trained license plate image detection network. The trained license plate image detection network identifies and marks the license plate detection box in the inspection image, and then transmits the inspection image with the marked license plate detection box to the trained license plate character detection network. Step A5: The trained license plate character detection network performs license plate character recognition within the license plate detection box to obtain the license plate number information. Step A2 is as follows: Step A2.1: Obtain vehicle images taken from multiple drone perspectives and open-source vehicle images; Step A2.2: Establish an image quality evaluation function and use the image quality evaluation function to evaluate the quality of the vehicle images. If the image is of low quality, it is discarded; if the image is of high quality, it is retained, thereby obtaining multiple high-quality vehicle images. Step A2.3: In each high-quality vehicle image, mark the smallest bounding rectangle of the license plate as the license plate annotation box; Step A2.4: Preprocess each vehicle image with a license plate marking frame, adjust it to a uniform size, and then perform gridding to form multiple cells, thus obtaining the processed vehicle image. Step A2.5: Input the processed vehicle image into the license plate image detection network. The license plate image detection network predicts the probability value of each cell belonging to a license plate, denoted as Pr. Then, filter out cells with probability values Pr greater than a set threshold to obtain a set of filtered cells. Step A2.6: Identify the overall boundary of the area covered by the filtered cell set, which is the license plate prediction box; then, calculate the overlap between the license plate prediction box and the license plate annotation box, denoted as the overlap ratio IOU; based on the overlap ratio IOU, calculate the loss value using a loss function. If the loss value is less than a set value, the training of the license plate image detection network ends, and a trained license plate image detection network is obtained; otherwise, adjust the parameters of the license plate image detection network according to the loss value to obtain an updated license plate image detection network, and then continue to train the updated license plate image detection network using the remaining license plate image training samples, iterating continuously until a trained license plate image detection network is obtained. Step A3 is as follows: Step A3.1: Construct a license plate character training sample set; the license plate character training sample set includes multiple license plate character training samples, each license plate character training sample is a license plate character image with the smallest rectangular frame of the license plate character, and the license plate character content is labeled; Step A3.2: Input the license plate character training samples into the license plate character detection network. The license plate character detection network performs feature extraction and text transcription on the license plate character training samples, specifically as follows: Step A3.2.1, the license plate character detection network converts the license plate character training samples into a license plate character image T1 with height H * width W * number of channels 3; In step A3.2.2, the license plate character detection network uses a patch partitioning method to divide the license plate character image T1 into n small image blocks T2 with a height of H / 4 and a width of W / 4. Step A3.2.3: The license plate character detection network extracts image features for each image patch T2 through convolutional layers; wherein, the image features include texture, shape, and edge features; then, a batch normalization processing module is used to normalize the image features of the image patch T2 to obtain normalized image features; then, an activation function is used to perform a non-linear mapping on the normalized image features to obtain the image patch feature vector; Step A3.2.4, the license plate character detection network adopts a three-stage feature module to extract local and global features of the feature vectors of each small image block, and then extracts the local features of the license plate characters and the dependencies between license plate characters to obtain the license plate character feature C; Specifically, positional information is embedded into the feature vector of each image patch, where the positional information refers to the relative position of the image patch feature vector within the license plate character image T1. Then, the first-stage feature module perceives the dependencies between all characters through the global fusion submodule and extracts global features; it perceives the correlations between all characters within a predefined receptive field through the local fusion submodule, extracts local features, encodes character morphological features, and establishes associations between characters, thereby simulating stroke features crucial for character recognition. The first-stage feature module extracts the first-stage character features. Then, the second-stage feature module further extracts character features from the first-stage features to obtain the second-stage features. Finally, the third-stage feature module further extracts character features from the second-stage features to obtain the third-stage features. Through repeated application of the three-stage feature module, the license plate character feature C is finally extracted. Step A3.2.5: The license plate character detection network classifies the license plate character features C using a parallel linear classifier to identify the license plate characters. Each license plate character has a confidence level. Based on the confidence level, a loss function is used to calculate the loss value. If the loss value is less than a set value, the training of the license plate character detection network ends, and a trained license plate character detection network is obtained. Otherwise, the parameters of the license plate character detection network are adjusted according to the loss value to obtain an updated license plate character detection network. For the updated license plate character detection network, the remaining license plate character sample images are used for further training. This process is iterated until a trained license plate character detection network is obtained. The illegal parking detection module includes two stages: vehicle detection and yellow line detection, and is specifically used for: Step B1: Using a vehicle detection network, identify vehicle identification boxes from the inspection image. The height of the vehicle identification box is h. With the top left corner of the inspection image as the origin O, the horizontal rightward direction as the positive x-axis, and the vertical downward direction as the positive y-axis, an image coordinate system is established to obtain the coordinates (x0, y0) of the center point P0 of the vehicle recognition frame. Step B2: The intersection of the horizontal line passing through the center point P0 and the left boundary of the vehicle recognition frame is point P1 (x1, y1). Let the length of P0P1 be d0. Shift point P1 to the left by the length d to obtain point P2. The value of d is as follows. ; Set the height with P0 and P2 as the axis of symmetry The rectangular ROI region; Step B3: Perform HSV color gamut conversion and edge extraction on the rectangular ROI region to extract a yellow line segment with a length of L. If the length L of the yellow line segment is greater than the threshold, it is determined to be a no-parking space with a yellow line, and the vehicle is illegally parked; if it is less than the threshold, it is determined to be a normal parking space, and the vehicle is normally parked.
2. The method for a static traffic routine inspection system based on unmanned airport networking according to claim 1, characterized in that, The unmanned airport includes a fast charging module, a monitoring camera module, a weather station module, a network module, an omnidirectional antenna module, and an RTK module; The fast charging module is used to quickly charge the inspection drone; The monitoring camera module is used to monitor the environment around the unmanned airport; The weather station module is used to monitor wind speed, rainfall, and temperature information around the unmanned airport; The network module can be connected to the network via a network cable or by installing a 4G / 5G module. The omnidirectional antenna module is used to communicate with the inspection drone and receive or send signals; The RTK module is used for high-precision navigation and positioning of the inspection drone; Using unmanned parking lots as forward nodes in the inspection system, these unmanned parking lots are deployed in different areas. By networking these unmanned parking lots, parking spaces can be managed in a grid-like manner, thereby achieving full coverage.
3. The method for a static traffic routine inspection system based on unmanned airport networking according to claim 1, characterized in that, The inspection drone is equipped with a gimbal camera and a high-precision positioning module.
Citation Information
Patent Citations
Parking management system combining high-position camera and parking space lock
CN116486496A
Vehicle illegal parking detection method based on urban high complexity detection scene
CN116503833B
A geomagnetic parking space detector and a smart parking system
CN218825850U
Roadside parking management method and system based on unmanned aerial vehicle, and storage medium
CN110766978A
Illegal parking vehicle license plate detection method and device, storage medium and computer equipment
CN115457532A
