A dual-mode GNSS-signal-free underground garage unmanned trolley patrol management system and method
By combining SLAM mapping and visible light recognition modules with passive RFID anchoring technology, the problems of information lag and low positioning accuracy in underground parking garages have been solved, enabling precise parking space management and efficient garage patrols.
Patent Information
- Application Number
- CN202611585977.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-10
AI Technical Summary
Underground parking garage management suffers from problems such as information lag, low positioning accuracy, and parking space mismatch. Existing technologies are insufficient for accurate patrolling and management in environments without GNSS.
By employing a SLAM mapping module combined with a visible light recognition module, a global coordinate system is established through passive RFID spatial anchoring. Combined with robust feature enhancement technology, accurate identification and dynamic management of vehicles and landmarks are achieved, forming a closed-loop management system.
It achieves a unified global coordinate benchmark for underground parking garages, improves positioning accuracy to ≤0.05m, achieves a parking space and license plate matching accuracy of ≥99.9%, reduces parking space status update delay to ≤200ms, and improves efficiency by more than 90%.
Smart Images

Figure CN123067484A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of smart parking, unmanned inspection and multi-sensor fusion technology, specifically a dual-modal GNSS-free underground parking garage unmanned vehicle patrol management system and method. Background Technology
[0002] Underground parking garages, as core facilities for urban transportation and building infrastructure, are growing in scale and complexity, making traditional management models inadequate for modern needs. Current underground parking garage management faces multiple technical and operational bottlenecks, specifically: On the one hand, the traditional manual patrol model has fundamental flaws: the status of parking spaces depends on manual recording and updating, which results in significant information lag, leading to low efficiency for car owners to find parking spaces and insufficient utilization of parking spaces; manual querying and statistics are time-consuming and labor-intensive, and administrators need to check each area to obtain the overall status of parking spaces, and the data is prone to errors due to human negligence; in long-term operation, labor costs account for more than 40% of the total management costs, and the management pressure of large-scale parking garages is particularly prominent.
[0003] On the other hand, the enclosed environment of underground parking garages poses unique challenges to technical solutions: GNSS signals are completely blocked, rendering navigation systems that rely on satellite positioning completely ineffective. Among existing alternatives, single-odometer navigation causes unmanned equipment to frequently deviate from the route due to accumulated errors (error ≥ 1m per 100m of driving), making it impossible to complete accurate patrols. While fixed camera solutions can achieve partial parking space recognition, their coverage is limited: a single camera can only cover a small portion of parking spaces, resulting in blind spots and making it difficult to adapt to changes in the garage structure (such as the need to redeploy hardware when rearranging parking spaces), resulting in extremely poor scalability.
[0004] Secondly, the lack of synergy in existing technical solutions exacerbates the management dilemma: some unmanned vehicle solutions only focus on mapping functions and lack linkage with target recognition, making it impossible to obtain parking space occupancy status and vehicle information in real time; single visual recognition technology suffers a sharp drop in recognition accuracy in the complex lighting environment of underground parking garages (where the light intensity varies greatly between day and night, and there are strong direct sunlight or completely dark scenes in some areas), making it difficult to support accurate recognition and management; while traditional SLAM mapping solutions can solve the problem of map building in environments without GNSS, they are not combined with functions such as parking space recognition and vehicle information collection, and cannot form a closed loop of "mapping-navigation-recognition-management".
[0005] Furthermore, with the upgrading of smart parking needs, users are increasingly demanding functions such as real-time parking space status inquiry, vehicle parking history tracking, and vacant parking space inquiry. Traditional solutions, due to the separate storage of navigation and recognition data and lagging processing (data not being updated in real time), cannot meet the requirements of intelligent management. Therefore, there is an urgent need for an integrated solution that combines autonomous mapping, accurate recognition, and dynamic management to solve the problems of low efficiency, data lag, and poor collaboration in traditional models.
[0006] This invention combines SLAM technology to achieve autonomous mapping in environments without GNSS signals, providing a foundation for underground navigation for unmanned vehicles; visible light target recognition technology can accurately collect vehicle and landmark information and update the backend database. The two are integrated into a dual-modal inspection system, which, combined with the mobility of unmanned vehicles, can effectively solve the core problem of patrol management in underground parking garages. Summary of the Invention
[0007] The purpose of this invention is to provide a dual-modal, GNSS-free underground parking garage unmanned vehicle patrol management system and method to solve the problems mentioned in the background art.
[0008] To achieve the above objectives, the present invention provides the following technical solution: A dual-modal, GNSS-free underground parking garage unmanned vehicle patrol management system includes a SLAM mapping module, a visible light recognition module, a central processing module, a parking garage management module, a passive RFID spatial anchoring module, and a spatial association module. The SLAM mapping module outputs the pose information of the unmanned patrol vehicle in the world coordinate system in real time. The visible light recognition module is used to identify vehicles in underground parking garages and their corresponding garage landmarks; The spatial association module receives the pose information from the SLAM mapping module and the pixel coordinates from the visible light recognition module. Combined with the pre-calibrated camera intrinsic parameters and camera-vehicle extrinsic parameters, the recognition results are mapped to the unified world coordinate system of the underground parking garage through pixel-to-world coordinate back projection. The central processing module receives pose, target recognition, and 3D spatial coordinate data, and establishes a dynamic binding relationship of "license plate-landmark-spatial coordinate-timestamp" using millisecond-level timestamps as unique association IDs; The garage management module stores garage data and updates and maintains the database in real time. Based on the administrator's instructions, it returns real-time information such as the garage's available parking space number, vehicle parking location, specific parking space and corresponding license plate number. The passive RFID spatial anchoring module includes several passive electronic tags that are fixedly deployed at designated locations in the underground parking garage. The passive electronic tags serve as the initial spatial anchor points of the global unified world coordinate system of the underground parking garage, used to calibrate the coordinate reference of the global coordinate system, perform initial alignment, and perform cumulative drift correction for the SLAM mapping module.
[0009] Furthermore, the visible light recognition module includes an image preprocessing unit, a robust feature enhancement submodule, a license plate recognition unit, and a garage landmark recognition unit. The robust feature enhancement submodule is used to perform multi-scale edge enhancement, adaptive contrast enhancement, and frequency domain ghosting removal on the preprocessed image. The robust feature enhancement submodule includes a multi-scale edge perception unit, an adaptive contrast enhancement unit, a frequency domain deghosting unit, and a cross-feature attention fusion unit. The multi-scale edge perception unit extracts edge features at different scales using multiple dilated convolution branches with different dilation rates and adaptively fuses features at each scale through a channel attention mechanism. The adaptive contrast enhancement unit performs illumination-reflection decomposition based on the four-scale Retinex theory and achieves nonlinear contrast enhancement through a learnable γ parameter. The frequency domain deghosting unit detects symmetric conjugate peaks in the frequency domain using fast Fourier transform and suppresses ghosting frequency components using a learnable mask. The cross-feature attention fusion unit generates dynamic weights through global feature perception and adaptively weights and fuses the outputs of multiple image processing branches. The image preprocessing unit has low-light enhancement, image denoising and distortion correction functions. After the image preprocessing unit completes the processing, it sends the image to the robust feature enhancement submodule for feature enhancement, while retaining the distortion parameters in the camera. The robust feature enhancement submodule outputs the enhanced feature map.
[0010] Furthermore, the operation of the spatial association module specifically includes: (1) Depth acquisition: The global real-time pose of the unmanned patrol vehicle output by the SLAM mapping module is used as the conversion reference, and the depth information of the target is obtained by combining the lidar point cloud data; (2) Pixel to camera coordinate back projection: Combining the aforementioned target depth information and camera intrinsic parameter matrix, the pixel center point coordinates (u,v) of the target in the image are back projected into a three-dimensional point in the camera coordinate system; (3) Camera to vehicle body rigid body transformation: The three-dimensional points are transformed from the camera coordinate system to the vehicle coordinate system through the camera-vehicle external parameter matrix; (4) Transformation from vehicle body to world rigid body: The three-dimensional points are transformed from the vehicle body coordinate system to the global world coordinate system of the underground garage through the real-time pose output by the SLAM mapping module.
[0011] Furthermore, the operation of the central processing module specifically includes: Spatiotemporal correlation data synchronization and merging: The spatiotemporal correlation records output by the spatial correlation module are transmitted to the central processing module and then synchronized to the main database service in real time. This triggers a two-way merging mechanism between the parking space spatial coordinate database and the license plate recognition database. Based on the timestamp, the spatial coordinates of the parking space landmark and the spatial coordinates of the license plate at the same time are correlated in real time to establish a one-to-one correspondence between "parking space and vehicle". At the same time, the landmark_detection_time, plate_detection_time and spatial coordinate fields of the parking_status table are supplemented to provide data support for subsequent calculation of vehicle parking time and parking space status tracking. Duplicate data cleanup is performed every few seconds, based on spatial coordinate overlap and timestamp to remove duplicates, and valid binding records are written to the detection_history table. It provides database query operations, supporting real-time retrieval of merged complete parking records by landmark ID, parking space coordinates, or license plate number; Timestamp synchronization and cross-service alignment are based on the main service's millisecond-level timestamp current_time, synchronizing the landmark detection time, license plate detection time, and SLAM pose output time returned by the sub-services.
[0012] Furthermore, the garage management module includes: The core database management unit initializes and maintains the `garage_merged.db` database, which includes the `parking_status` and `detection_history` tables. Atomic operations ensure data consistency under multi-threaded concurrency. The `parking_status` table pre-stores a fixed world coordinate range corresponding to each parking space landmark. By determining whether the world coordinates of the license plate fall within the coordinate range of the corresponding parking space, precise matching of parking spaces and vehicles is achieved. Furthermore, the spatiotemporal association records output by the spatial association module are synchronized in real time. Based on timestamps and world coordinates, landmarks and `license_plate` information are associated, and the `is_occupied` field is dynamically updated to mark the parking space occupancy status. The data interaction and command response unit provides multi-dimensional query interfaces: it supports precise querying of the real-time status of a single parking space by landmark ID, or batch return of the occupancy status of all parking spaces to generate a list of available parking spaces; it traces vehicle parking records by license plate number and calculates the actual parking time of the vehicle by combining landmark_detection_time and plate_detection_time. The historical data tracing and maintenance unit automatically stores several recent detection history records, including details of landmark / license plate recognition results, service response time, and recognition status, for troubleshooting and data auditing. Secondly, combined with the sub-service health monitoring mechanism, when an abnormal recognition is detected, the abnormal status is marked in the historical record.
[0013] Furthermore, the camera intrinsic parameters include the camera focal length, principal point coordinates, and distortion coefficients; the camera-to-vehicle extrinsic parameters include a 3×3 rotation matrix and a 3×1 translation vector between the camera coordinate system and the vehicle coordinate system; the coordinate transformation of the spatial association module includes two parts: pixel-to-world coordinate back projection and rigid body transformation. (1) Pixel to camera coordinate back projection: Combining the target depth value z obtained by the lidar and the pre-calibrated camera intrinsic parameter matrix K, the pixel center point coordinates (u,v) of the target in the image are back projected into a three-dimensional point in the camera coordinate system. (2) Rigid body transformation: including the external parameter transformation from the camera to the vehicle body and the pose transformation from the vehicle body to the world, both of which are rotation and translation transformations between three-dimensional coordinate systems; The coordinate transformation formula for the spatial association module is: , In the formula: P w =(X w ,Y w Z w ) T The target's three-dimensional coordinates are given in the global world coordinate system of the underground parking garage, with the superscript T indicating transpose. T wb The real-time pose transformation matrix from the vehicle coordinate system to the world coordinate system output by the SLAM mapping module; T bc This is the extrinsic parameter matrix from the pre-calibrated camera coordinate system to the vehicle coordinate system; z represents the target depth value obtained by the lidar; (u,v) represents the coordinates of the center point of the pixel in the image.
[0014] This invention also provides a dual-mode GNSS-free underground parking garage unmanned vehicle patrol management method, implemented through the dual-mode GNSS-free underground parking garage unmanned vehicle patrol management system described above, comprising: S1, to perform hardware deployment and global coordinate system calibration, to calibrate the global unified world coordinate system of the underground garage through passive RFID electronic tags, to complete the accurate calibration of the camera's internal parameters and the camera-vehicle external parameters, and to deploy the equipment at the underground garage site; S2 is used for software deployment and environment configuration. S3 performs dual-modal data acquisition and preprocessing, simultaneously acquiring LiDAR data required for SLAM mapping and image data required for visible light recognition; S4 performs visible light target recognition and target spatial coordinate calculation. Based on the real-time pose output by the SLAM mapping module, the recognized target in the two-dimensional image is mapped to the global world coordinate system, and the three-dimensional spatial coordinates of the target are calculated. S5 performs central data processing and dynamic association binding, using millisecond-level timestamps as unique IDs to establish spatial coordinate associations between license plates and parking spaces. S6 enables garage management and data interaction, allowing for dynamic updates and queries of parking space status based on spatial coordinate relationships.
[0015] Further, S3 includes: S3.1, Multi-source data acquisition initiated, including: S3.1.1, Log in to the unmanned patrol vehicle terminal and start the 360° RGB camera; S3.1.2 The unmanned patrol vehicle patrols along a preset path, with the camera collecting images, the lidar collecting point cloud data, and the inertial navigation unit collecting attitude data. S3.1.3, image data is transmitted to edge nodes via the network, while point cloud data and attitude data are stored in the vehicle's embedded unit for SLAM positioning updates; S3.2, Image preprocessing is performed, including: S3.2.1, Image scaling: The edge node calls the img_first_pre function to scale the acquired image proportionally using the INTER_AREA interpolation method; S3.2.2, Gaussian Denoising: The scaled image is denoised using the cv2.GaussianBlur function; S3.2.3, Grayscale Conversion and Binarization: First, the color image is converted to a grayscale image, and then the Otsu adaptive thresholding algorithm is used to achieve automatic thresholding segmentation, thereby improving the contrast between the target area and the background. S3.3, SLAM localization and map update; S3.3.1, the SLAM mapping module integrates LiDAR point cloud and inertial navigation data to estimate the position and attitude of the unmanned patrol vehicle on the map in real time; S3.3.2, combined with dynamic environmental changes, optimizes map accuracy in real time to ensure that the vehicle patrols stably along the planned route without deviation.
[0016] Further, S4 includes: S4.1 The preprocessed image is input into the robust feature enhancement submodule, where multi-scale edge perception, adaptive contrast enhancement and frequency domain ghosting are performed sequentially, and an enhanced feature map is generated through cross-feature attention fusion. S4.2, input the enhanced feature map into the YOLO series model, output the license plate bounding box (x1, y1, x2, y2), and filter the confidence score V. label Candidate boxes with a value ≥ μ, where 0.7 ≤ μ ≤ 1.0; S4.3, crop the license plate ROI sub-image, input it into the visual text understanding model, extract the license plate number, and require a recognition confidence score V. c ≥Ω, 0.85≤Ω≤1.0; S4.4, convert the license plate ROI sub-image to HSV space, and count the proportion of blue and green pixels respectively. Based on the proportion of blue and green pixels, determine whether the vehicle is a gasoline vehicle or an electric vehicle. S4.5 calls the PlateDetectionManager class to count license plate detection results within a time window of several seconds, accumulating the number of valid tickets V. sum If the value is greater than or equal to φ, then keep the license plate result and delete duplicate data; 3 ≤ φ ≤ 20. S4.6 inputs the enhanced feature map into the YOLO series model, outputs landmark bounding boxes, and filters the confidence scores V. area Candidate boxes with a value ≥0, where 0.995 ≤ 0 ≤ 1.0; S4.7, crop the landmark ROI sub-image, input the fine-tuned visual text understanding model, and extract the landmark number; S4.8 calls the LandmarkAnomalyDetector class to record several recent valid landmarks and calculate the deviation between the new landmark and the historical average; when the landmark recognition deviation value △M≤T dev If the landmark identification deviation value △M > T, it is considered valid; dev When marked as a potential anomaly, 80≤T dev ≤200.
[0017] Further, S5 includes: The main service generates a millisecond-level timestamp current_time for each frame of image, and synchronizes the license plate detection time plate_detection_time and the landmark detection time landmark_detection_time. Using current_time as a unique ID, and based on the overlap between the license plate and the world coordinates of the landmark, the license plate and landmark data are linked to establish a one-to-one mapping relationship of "landmark number - parking space fixed spatial coordinates - license plate world coordinates - license plate number"; The main service performs duplicate data cleanup every few seconds, using a voting deduplication algorithm based on a few-second time window: , In the formula, V weighted For weighted votes, T win Let I be the time window, and L be the indicator function. I takes the value 1 if the condition within the parentheses is true, and 0 if it is false. t Let w(t) be the license plate number detected in the t-th test, and L be the weight of the t-th test. consensus For the consensus license plate number, Where Dist(·) is the function for calculating the distance between detection boxes. max B is the preset maximum distance threshold for the detection frame; t For the t-th detection box, B avg For the average detection frame, when the weighted vote count V weighted The result is retained when the value is ≥3; Call the `clean_duplicate_data` function to delete duplicate detection records, retaining only the best voting result; The parking_status table is updated using atomic operations: when a valid license plate number is detected and the landmark is valid, is_occupied=1 is set to indicate that the parking space is occupied; when no license plate number is detected but the landmark is valid, is_occupied=0 is set to indicate that the parking space is vacant. Call the save_detection_history function to write the deduplicated valid records into the detection_history table, automatically retain the most recent few records, and archive expired records in descending order of timestamp.
[0018] Compared with the prior art, the beneficial effects of the present invention are: 1) Solving the problem of unifying the global coordinate reference in underground parking garages without GNSS: This invention uses pre-deployed passive RFID tags as spatial anchor points of the global world coordinate system, providing a unique global coordinate reference for SLAM mapping, target recognition and parking space management, overcoming the defects of existing technologies such as coordinate drift during multiple patrols and multiple device operation, and inability to achieve global alignment; the positioning error throughout the process is ≤0.05m, and the positioning accuracy is significantly better than existing technologies.
[0019] 2) Achieving deep integration of SLAM pose and visual recognition, breaking through the bottleneck of their separation: This invention completes precise sensor calibration through Kalibr and FAST-Calib. Based on the real-time pose of SLAM, the two-dimensional recognition results are accurately mapped to three-dimensional world coordinates through pixel-to-world coordinate back projection, achieving a technological leap from image recognition to target spatial positioning; the accuracy rate of parking space and license plate matching is ≥99.9%, completely solving the industry pain point of mismatch between adjacent parking spaces. This integration solution is not a conventional technical means in this field and cannot be easily inspired by existing technologies.
[0020] 3) Construct a multi-dimensional dynamic association parking space management mechanism to achieve intelligent closed-loop management: This invention relies on a globally unified coordinate system to establish a dynamic binding relationship between license plate information, spatial location, timestamp, and parking space status. Parking space status updates rely on precise spatial coordinate matching, no longer solely depending on visual recognition results. The accuracy rate of parking space status matching is ≥99.9%, fundamentally avoiding the problem of mismatch between recognition and parking space location. This multi-dimensional linkage management is a customized spatial coordinate association mechanism, not a conventional database function, and has outstanding technical advantages.
[0021] 4) A robust feature enhancement scheme specifically designed for underground parking garage scenarios is proposed, breaking through the bottleneck of YOLO detection accuracy: This invention is the first to organically integrate multi-scale edge perception, learnable Retinex contrast enhancement, and frequency domain ghosting removal, and designs an end-to-end trainable RFEM module to specifically solve the three major detection problems of paint chipping, ghosting, and dim lighting in underground parking garages. Compared with the traditional YOLO direct detection scheme, the detection accuracy of paint chipping landmarks is improved from 0.52 to 0.79 (+51.9%), the detection accuracy of ghosting scenes is improved from 0.48 to 0.78 (+62.5%), and the detection accuracy of low light is improved from 0.35 to 0.72 (+105.7%), with an overall detection accuracy improvement of 80.9%, providing a reliable foundation for subsequent spatial coordinate calculation and parking space matching. The module is lightweight and efficient, with only 0.5M parameters in the unmanned vehicle-specific version and an inference time of ≤15ms, fully meeting the real-time inference requirements of edge devices.
[0022] 5) This invention forms a complete technical closed loop of global coordinate anchoring, real-time pose calculation, target space mapping, dynamic association binding, and intelligent garage management. Each module works together to effectively solve core problems such as low positioning accuracy, mismatched parking space identification, and poor management reliability in underground garage patrol management. The parking space status update delay is ≤200ms, which improves efficiency by more than 90% compared with traditional manual patrol management. Attached Figure Description
[0023] Figure 1 This is a block diagram of a dual-modal, GNSS-free underground parking garage unmanned vehicle patrol management system according to the present invention. This diagram is a simplified schematic diagram of the system architecture, showing only the core data processing link. The passive RFID spatial anchoring module and spatial association module are underlying support modules and are not shown. Figure 2 This is a flowchart illustrating the overall process of a dual-modal, GNSS-free underground parking garage unmanned vehicle patrol management method according to the present invention. Figure 3 This is a flowchart of the license plate recognition port of the present invention; Figure 4 This is a flowchart of the garage landmark recognition process of the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] Please see Figure 1A dual-modal, GNSS-free underground parking garage unmanned vehicle patrol management system includes an unmanned patrol vehicle, a SLAM mapping module, a visible light recognition module, a central processing module, a parking garage management module, a passive RFID spatial anchoring module, and a spatial association module.
[0026] The unmanned patrol vehicle is equipped with dual-modal sensors, an internal processor, an RFID reader / writer unit, an inertial navigation unit, a 360° RGB camera, and a lidar, and is used to autonomously patrol along a preset path inside an underground parking garage.
[0027] The SLAM mapping module is used to create a global unified coordinate system based on the passive RFID spatial anchoring module. It integrates point cloud data from the lidar module on the unmanned patrol vehicle with image information from the visible light camera module to achieve real-time mapping, localization, and attitude estimation of the underground parking garage environment. Both lidar and visual information are provided by sensors on the unmanned patrol vehicle: point cloud data from the lidar and image data from the 360° RGB camera. The fused data is then input into the SLAM mapping module for processing.
[0028] The visible light recognition module is mounted on an unmanned patrol vehicle to identify vehicles and corresponding parking garage landmarks in underground parking garages. The visible light recognition module includes an image preprocessing unit, a robust feature enhancement submodule (RFEM), a license plate recognition unit, and a parking garage landmark recognition unit.
[0029] The license plate recognition unit uses the YOLO series model to first select the license plate location. Then, it calls the visual text understanding model to perform text recognition within the selected area. Next, it performs pixel value statistics on the selected area and determines the specific vehicle type, such as an electric vehicle or a gasoline vehicle, based on the pixel value range. The HSV color range for the blue license plate of a gasoline vehicle is [95,60,30] to [145,255,255]. HSV is a color space representation that decomposes color into three channels: Hue (H), Saturation (S), and Value (V). Hue (H): represents the type of color, such as blue, green, etc., and the value range is usually 0–180 or 0–360. Saturation (S): represents the vividness of the color, and the value range is 0–255. The larger the value, the more vivid the color. Value (V): represents the brightness of the color, and the value range is 0–255. The larger the value, the brighter the color. The HSV color range of the green license plate of the tram is [30,60,30] to [90,255,255]. Calculate the ratio of the number of pixels within the blue license plate HSV color range to the total number of pixels in the license plate area (blue_ratio) and the ratio of the number of pixels within the green license plate HSV color range to the total number of pixels in the license plate area (green_ratio):
[0030] In the formula, ratio color For blue_ratio or green_ratio, Num is the number of pixels whose HSV three channels are all within the target range. total H represents the total number of pixels in the license plate area. max H min S represents the maximum and minimum values of H. max ,S min V represents the maximum and minimum values of S. max V min Represents the maximum and minimum values of V; when green_ratio > blue_ratio and green_ratio > R (0.4 < R ≤ 1.0), it is determined to be an electric vehicle; otherwise, it is an oil vehicle.
[0031] The garage landmark recognition unit is used to identify the specific coordinates and content of garage landmarks. First, it calls the YOLO series model to select the garage landmark area, and then calls the fine-tuned visual text understanding model to recognize the garage landmark in the selected area.
[0032] The image preprocessing unit has low-light enhancement, image denoising and distortion correction functions. After completing low-light enhancement and distortion correction, the image preprocessing unit sends the processed image to the license plate recognition service port and the landmark recognition service port for parallel processing.
[0033] The robust feature enhancement submodule performs multi-scale edge enhancement, adaptive contrast enhancement, and frequency domain ghosting removal on preprocessed images. It comprises a multi-scale edge perception unit, an adaptive contrast enhancement unit, a frequency domain ghosting removal unit, and a cross-feature attention fusion unit. The multi-scale edge perception unit extracts edge features at different scales using four dilated convolution branches with different dilation rates (1, 2, 4, 8), and adaptively fuses these features through a channel attention mechanism. The adaptive contrast enhancement unit performs illumination-reflection decomposition based on four-scale Retinex theory and achieves nonlinear contrast enhancement through a learnable γ parameter. The frequency domain ghosting removal unit detects symmetric conjugate peaks in the frequency domain using FFT transformation and suppresses ghosting frequency components using a learnable mask. The cross-feature attention fusion unit generates dynamic weights through global feature perception and adaptively weights and fuses the outputs of the three branches. Specifically designed for three special scenarios—paint peeling, ghosting, and dim lighting—in underground parking garages, this robust feature enhancement submodule works in conjunction with YOLO series models to significantly improve target detection accuracy.
[0034] The spatial association module receives the pose information output by the SLAM mapping module and the pixel coordinates output by the visible light recognition module. Combined with the camera intrinsic parameters calibrated by the Kalibr tool and the camera-to-vehicle extrinsic parameters calibrated by the FAST-Calib tool, the identified license plate and garage landmark are mapped to the unified world coordinate system of the underground garage. Specifically: (1) Depth acquisition: Using the global real-time pose of the unmanned patrol car output by the SLAM mapping module as the transformation reference, the depth information of the target is obtained by combining the laser radar point cloud data; (2) Pixel-to-camera coordinate back projection: Combined with the laser radar point cloud data to obtain the depth information of the target, the pixel center point coordinates (u,v) of the target in the image are back projected into a three-dimensional point in the camera coordinate system; (3) Camera-to-vehicle rigid body transformation: The three-dimensional point is transformed from the camera coordinate system to the vehicle coordinate system through the camera-vehicle extrinsic matrix; (4) Vehicle-to-world rigid body transformation: The three-dimensional point is transformed from the vehicle coordinate system to the global world coordinate system of the underground garage through the real-time pose output by the SLAM mapping module.
[0035] The central processing module receives pose data from the SLAM mapping module and target recognition data and 3D spatial coordinate data from the spatial association module in real time. Using millisecond-level timestamps as unique association IDs, it establishes a dynamic binding relationship between "license plate information - license plate spatial coordinates - garage landmark information - landmark spatial coordinates - timestamp". The specific operations of the central processing module include: Real-time database synchronization and merging: The spatiotemporal correlation records output by the spatial correlation module are transmitted to the central processing module. The core of the central processing module is to fuse and correlate license plate recognition data and garage landmark recognition data. It establishes a "parking space-vehicle" mapping relationship through timestamp + landmark number, synchronizes it to the database, and supplements the detection time field to support parking space status management and vehicle parking duration calculation. It triggers the bidirectional merging mechanism of 'garage_merged.db' and 'garage_landmark.db', and correlates the timestamps of 'recognition_records' and 'license_plate_records' in real time, thereby recording whether a vehicle is parked in a specific parking space at the same time, and the license plate number of the parked vehicle; at the same time, it supplements the 'landmark_detection_time' and 'plate_detection_time' fields of the 'parking_status' table to provide data support for subsequent calculation of vehicle parking time.
[0036] Duplicate data cleanup is performed every few seconds, based on spatial coordinate overlap and timestamp to remove duplicates, and valid binding records are written to the detection_history table. It provides database query operations, supporting real-time retrieval of merged complete parking records by landmark ID, parking space coordinates, or license plate number; Timestamp synchronization and cross-service alignment are based on the main service's millisecond-level timestamp current_time, synchronizing the landmark detection time, license plate detection time, and SLAM pose output time returned by the sub-services to ensure that the time alignment error is ≤300ms.
[0037] The garage management module receives the spatiotemporal correlation records, associates parking space locations using world coordinates as an index, stores garage data, and updates and maintains the database in real time. Based on administrator commands, it returns real-time information such as available parking space numbers, vehicle parking locations, specific parking spaces, and corresponding license plate numbers. The garage management module includes a core database management unit, a data interaction and command response unit, and a historical data tracing and maintenance unit. The core database management unit initializes and maintains the `garage_merged.db` database, which includes the `parking_status` and `detection_history` tables. Atomic operations ensure data consistency under multi-threaded concurrency. The `parking_status` table pre-stores a fixed world coordinate range corresponding to each parking space landmark. By determining whether the license plate's world coordinates fall within the corresponding parking space's coordinate range, precise matching of parking spaces and vehicles is achieved. Furthermore, the spatiotemporal association records output by the spatial association module are synchronized in real time. Based on timestamps and world coordinates, landmarks and `license_plate` information are associated, and the `is_occupied` field is dynamically updated to mark the parking space occupancy status.
[0038] The data interaction and command response unit provides multi-dimensional query interfaces: it supports precise querying of the real-time status of a single parking space by landmark ID, or batch return of the occupancy status of all parking spaces to generate a list of available parking spaces; it traces vehicle parking records by license plate number and calculates the actual parking time of the vehicle by combining landmark_detection_time and plate_detection_time.
[0039] The historical data tracing and maintenance unit automatically stores several recent detection history records, including details of landmark / license plate recognition results, service response time, and recognition status, for troubleshooting and data auditing. Secondly, combined with the sub-service health monitoring mechanism, when an abnormality is detected, the abnormal status is marked in the historical record, providing a basis for optimizing the patrol management of unmanned vehicles in dual-modal underground parking garages without GNSS signals.
[0040] The camera's intrinsic parameters include the camera focal length, principal point coordinates, and distortion coefficients; the camera-to-vehicle extrinsic parameters include a 3×3 rotation matrix and a 3×1 translation vector between the camera coordinate system and the vehicle coordinate system; the coordinate transformation of the spatial association module includes two parts: pixel-to-world coordinate back projection and rigid body transformation. (1) Pixel to camera coordinate back projection: Combining the target depth value z obtained by the lidar and the pre-calibrated camera intrinsic parameter matrix K, the pixel center point coordinates (u,v) of the target in the image are back projected into a three-dimensional point in the camera coordinate system. (2) Rigid body transformation: including the external parameter transformation from the camera to the vehicle body and the pose transformation from the vehicle body to the world, both of which are rotation and translation transformations between three-dimensional coordinate systems; The coordinate transformation formula for the spatial association module is:
[0041] In the formula: P w =(X w ,Y w Z w ) T The target's three-dimensional coordinates are given in the global world coordinate system of the underground parking garage, with the superscript T indicating transpose. T wb The real-time pose transformation matrix from the vehicle coordinate system to the world coordinate system output by the SLAM mapping module; T bc This is the extrinsic parameter matrix from the pre-calibrated camera coordinate system to the vehicle coordinate system; z represents the target depth value obtained by the lidar; (u,v) represents the coordinates of the center point of the pixel in the image.
[0042] Please see Figure 2 A dual-modal, GNSS-free underground parking garage unmanned vehicle patrol management method, comprising: S1 involves hardware deployment and global coordinate system calibration. A passive RFID electronic tag is used to calibrate the unified global coordinate system of the underground parking garage, achieving precise calibration of camera-to-vehicle external parameters. The deployment location is the underground parking garage site (deployment location: underground parking garage site), including: S1.1, The supporting framework and basic hardware are fixed, including: S1.1.1 Select a flat area in the underground garage (such as an open space near the entrance) to place the unmanned vehicle base. Use M5×10mm hex screws (4 screws) to fix the aluminum alloy support frame (30cm×20cm×15cm) to the center of the chassis. Control the screw torque at 2.5N・m to ensure that the frame does not wobble (wobble ≤0.5mm).
[0043] S1.1.2 Install a 360° RGB camera (≥2 million pixels) at the top center of the support frame. The camera lens should be vertically downward (90° tilt angle), with the center point 1.5m above the ground. The deployment position should ensure that the observation range covers a 2m×2m area around the vehicle without being obstructed by the support frame.
[0044] S1.1.3 Install a lidar (scanning frequency ≥10Hz, ranging accuracy ±2cm) at the front end of the support frame. The deployment position should be at least 30cm away from the camera horizontally to avoid overlapping of their fields of view (overlapping area ≤5%). The lidar should be oriented towards the direction of the vehicle's movement to ensure that the scanning range covers the passage in front (radius ≥5m).
[0045] S1.1.4. Fix the inertial navigation unit to the rear of the car chassis (away from the motor vibration source) using a 5mm thick shock-absorbing pad. The deployment position must be close to the chassis and not loose to avoid vibration interfering with the positioning accuracy.
[0046] S1.2, Hardware parameter calibration and link testing, including: S1.2.1 Use a level to check the levelness of the camera and the lidar along the direction of travel of the car (X-axis) and perpendicular to the direction of travel (Y-axis), respectively. The level error should be ≤0.1°. If the error exceeds the standard, adjust it to meet the standard by adding or removing shock-absorbing pads.
[0047] S1.2.2, connect the camera to the host computer (deployed in the garage duty room), access the camera management interface, preview images in real time, adjust the camera angle to ensure no blind spots, and test the stability of image transmission (no packet loss for 5 minutes).
[0048] S1.2.3 Use a network cable tester to verify the network connection between the camera, LiDAR and edge node (deployed in the garage duty room) to ensure that the data transmission link is smooth and the packet loss rate is ≤0.1%.
[0049] S1.3, Global coordinate system calibration and RFID anchor point deployment, including: S1.3.1, Global coordinate system origin calibration: Select an open and fixed location at the entrance of the underground parking garage as the origin of the global coordinate system, deploy the first passive RFID electronic tag, and calibrate its world coordinates as (0,0,0) as the initial anchor point of the global coordinate system.
[0050] S1.3.2, Auxiliary Anchor Point Deployment: Along the main passage of the underground parking garage, deploy an auxiliary passive RFID electronic tag every 50m. Use a total station to calibrate the precise world coordinates of each tag and pre-store them in the database of the parking garage management module for global coordinate correction and cumulative drift suppression during the SLAM mapping process.
[0051] S1.3.3, Anchor Point Coordinate Fixing: The world coordinates of all RFID anchor points are written into the configuration file of the vehicle's embedded unit, serving as the global reference for SLAM mapping and spatial coordinate mapping. After reading the anchor point coordinates, six-degree-of-freedom difference compensation is performed between the standard global coordinates and the real-time SLAM solution coordinates, and the vehicle's pose transformation matrix is offset-corrected to continuously eliminate the cumulative drift in positioning caused by the vehicle's long-distance travel, ensuring that the global coordinate system is aligned throughout the entire process.
[0052] S1.3.4, Fixed coordinate interval calibration of parking spaces: Based on the global unified world coordinate system anchored by RFID, a total station is used to accurately map the landmark and boundary of each parking space in the underground garage, calibrate the world coordinate interval corresponding to each parking space, and pre-store it in the parking_status table of the parking_merged.db database to establish a one-to-one correspondence between "landmark number - fixed coordinate interval of parking space" as the benchmark for subsequent matching of license plates and parking spaces.
[0053] S1.4, Sensor calibration, including: S1.4.1, Camera Intrinsic Parameter Calibration: Using the Kalibr classic calibration tool, an AprilGrid calibration board was created. The unmanned patrol vehicle was controlled to acquire ≥200 frames of images of the calibration board at different angles and distances in an underground parking garage environment. The camera intrinsic parameter matrix K and radial and tangential distortion coefficients were obtained by solving the nonlinear optimization algorithm of Kalibr. The reprojection error of the intrinsic parameter calibration was ≤0.1 pixels.
[0054] S1.4.2, Camera-Vehicle Extrinsic Parameter Calibration: Using the FAST-Calib LiDAR-Camera Rapid Calibration Tool, with the calibration board deployed in the underground parking garage as the target, LiDAR point cloud data and camera images are acquired simultaneously. Without initial extrinsic parameters, the extrinsic parameters are solved within 1 second, yielding the transformation matrix from the camera coordinate system to the vehicle coordinate system. (Including a 3×3 rotation matrix and a 3×1 translation vector), calibration reprojection error ≤ 0.5 pixels.
[0055] S1.4.3, Calibration Parameter Fixation: The calibrated camera intrinsic and extrinsic parameters are written into the configuration files of the vehicle's embedded unit and edge nodes as fixed reference parameters for spatial coordinate mapping.
[0056] S2, software deployment and environment configuration (deployment locations: vehicle embedded unit, edge node, main server), including:
[0057] S2.1, Vehicle-side software deployment (deployment location: embedded unit of the unmanned vehicle), including: S2.1.1, install the Ubuntu 20.04 LTS operating system on the NVIDIA Jetson Nano embedded unit of the unmanned vehicle, and configure the static IP as 192.168.1.100.
[0058] S2.1.2, install ROS Noetic and Cartographer 1.0.0, run the command to start SLAM, push the car to drive one lap (≥500m) along the garage passage, generate an initial environment map with a resolution of 0.05m and save it.
[0059] S2.1.3, install OpenCV 4.5.5, configure the image acquisition environment, test the camera and the preview window displays normally.
[0060] S2.2, Edge Node Software Deployment (Deployment Location: Garage Guard Room Edge Node), includes: S2.2.1 Install PyTorch 1.13.1 and PaddlePaddle 2.4.2 on the Raspberry Pi 4B edge node (static IP: 192.168.1.102) and configure the Python 3.8 runtime environment.
[0061] S2.2.2 Deploy the target detection and recognition model: Place the pre-trained YOLOv8 (license plate detection), YOLOv11 (landmark detection) models, visual text understanding model, visual text understanding model fine-tuned with 2700 garage landmark images, and robust feature enhancement submodule (RFEM) model files in the corresponding locations.
[0062] S2.2.3 Deploy the license plate recognition service script, switch to the script directory to test the service startup status, and ensure that the terminal outputs that the service has started successfully.
[0063] S2.2.4, create a new SSH window, switch to the landmark recognition script directory, test the service startup status, and ensure that the terminal output shows the service has started successfully.
[0064] S2.3, Main server software deployment (deployment location: Garage Management Center Main Server), includes: S2.3.1, install the Ubuntu 20.04 LTS operating system, Flask 2.2.3, and SQLite 3.40.1 on the main server.
[0065] S2.3.2 Initialize the database, create the parking_status table (fields include landmark number, license plate number, vehicle type, update time, occupancy status, and detection timestamp) and the detection_history table (fields include detection time, response time, recognition status, confidence, etc.), and create a composite index for the landmark (landmark ID) and current_time (timestamp) fields.
[0066] S2.3.3 Deploy the main service script, test the connection status between the main service and the sub-services, and ensure successful connection.
[0067] S3 performs dual-modal data acquisition and preprocessing, simultaneously acquiring LiDAR data required for SLAM mapping and image data required for visible light recognition (operating location: embedded unit of the vehicle, edge node), including: S3.1, Multi-source data acquisition startup (running location: vehicle embedded unit), including: S3.1.1 Log in to the vehicle terminal, start the 360° RGB camera, and confirm that the image topic is / usb_cam / image_raw.
[0068] S3.1.2 The unmanned vehicle patrols along a preset path (patrol interval 15 minutes / time). The camera collects one frame of image (resolution 1280×720) every 400ms, the lidar collects point cloud data once every 200ms, and the inertial navigation unit collects attitude data once every 100ms.
[0069] In S3.1.3, image data is transmitted to edge nodes via Gigabit Ethernet (transmission delay ≤15ms), while point cloud data and attitude data are stored in the vehicle's embedded unit for SLAM positioning updates.
[0070] S3.2, Image preprocessing execution (running location: edge node), includes: S3.2.1, Image scaling: The edge node calls the img_first_pre function to scale the acquired image to 640×640 using the INTER_AREA interpolation method, reducing image processing latency.
[0071] S3.2.2, Gaussian Denoising: The scaled image is denoised using the cv2.GaussianBlur function with a 5×5 convolution kernel and a standard deviation σ=1.0 to reduce camera sensor noise.
[0072] S3.2.3, Grayscale Conversion and Binarization: First, the color image is converted to a grayscale image, and then the Otsu adaptive thresholding algorithm is used to achieve automatic thresholding segmentation, thereby improving the contrast between the target area and the background.
[0073] S3.3, SLAM localization and map update (running location: vehicle embedded unit), includes: S3.3.1 Initial coordinate alignment: After the unmanned patrol vehicle starts, it reads the RFID tag of the initial anchor point through the on-board RFID reader and writer unit to obtain the coordinates of the origin of the global coordinate system, completes the initial coordinate alignment of the SLAM system, and binds the SLAM local coordinate system with the global unified world coordinate system. S3.3.2, Real-time pose calculation: The SLAM mapping module integrates LiDAR point cloud data, visual image data and attitude data from the inertial navigation unit. Based on a tightly coupled fusion algorithm, it calculates the six-degree-of-freedom pose of the unmanned patrol vehicle in a globally unified world coordinate system in real time. The pose output frequency is ≥10Hz and the positioning error is ≤0.05m. S3.3.3, Global Coordinate Correction: During the operation of the unmanned patrol vehicle, when the on-board RFID reader reads the auxiliary RFID anchor tag, it compares the pre-calibrated world coordinates of the tag with the current coordinates calculated by SLAM to correct the cumulative drift of the SLAM system and ensure the consistency and stability of the global coordinate system throughout the entire process. S3.3.4, Real-time Map Optimization: Based on dynamic environmental changes (such as temporary obstacles), the map accuracy is optimized in real time to ensure that the vehicle patrols stably along the planned route without deviation.
[0074] S3.3.5, Real-time Pose Synchronization: The real-time pose of the vehicle body calculated by the SLAM mapping module is published to the spatial association module in real time through ROS topics. The pose publication frequency is consistent with the camera image acquisition frequency (25Hz). Each frame of image corresponds to unique vehicle body pose data, ensuring that the timestamp alignment error is ≤10ms.
[0075] S4 performs visible light target recognition and target spatial coordinate calculation. Using the real-time pose output by the SLAM mapping module as a reference, the recognized target in the 2D image is mapped to the global world coordinate system, and the 3D spatial coordinates of the target (running position: edge node) are calculated, including:
[0076] S4.1 Robust Feature Enhancement: The preprocessed image is input into the robust feature enhancement submodule, where multi-scale edge detection, adaptive contrast enhancement, and frequency domain ghosting removal are performed sequentially. An enhanced feature map is generated through cross-feature attention fusion, including: S4.1.1, Multi-scale edge perception: The preprocessed image is input into the multi-scale edge perception unit, and edge features of different receptive fields are extracted through four 3×3 dilation convolution branches with dilation rates of 1, 2, 4 and 8 respectively; the features of the four branches are concatenated in the channel dimension and input into the channel attention module to automatically learn the weights of the features at each scale and output the fused edge feature map.
[0077] S4.1.2, Adaptive Contrast Enhancement: The preprocessed image is input into the adaptive contrast enhancement unit. The multi-scale illumination components are estimated by Gaussian blurring with four different σ values (1,2,4,8), and the reflection components are obtained by illumination-reflection decomposition. The reflection components are nonlinearly enhanced by learnable γ parameters. Combined with the adaptive gain generated by channel attention, the contrast-enhanced feature map is output.
[0078] S4.1.3, Frequency Domain Deghosting: The preprocessed image is input into the frequency domain deghosting unit, and the image is transformed to the frequency domain through FFT transformation. Symmetrical conjugate peaks (ghosting features) in the frequency domain are detected. Suppression weights are generated through learnable Mask parameters to suppress the frequency components corresponding to the ghosting. The deghosted feature map is reconstructed through inverse FFT transformation.
[0079] S4.1.4, Cross-feature attention fusion: The edge feature map, contrast enhancement feature map, and ghosting removal feature map are concatenated in the channel dimension. Spatial information is compressed by global average pooling and input into a two-layer convolutional neural network to generate dynamic weights for the three branches. After Softmax normalization, the feature maps of the three branches are weighted and fused to output the final enhanced feature map, which is sent to the license plate recognition unit and the parking lot landmark recognition unit.
[0080] The robust feature enhancement submodule adopts a dedicated optimized version for unmanned vehicles, with only 0.5M parameters and a single-frame inference time of ≤15ms, which fully meets the real-time processing requirements of edge devices and will not increase the overall system latency.
[0081] S4.2, License Plate Area Detection: (e.g., ...) Figure 3 As shown, the enhanced feature map is input into the YOLO series model, and the output license plate bounding box (x1, y1, x2, y2) is used to filter the confidence score V. label Candidate boxes with a value ≥ μ, where 0.7 ≤ μ ≤ 1.0, and μ is preferably 0.7.
[0082] S4.3, Character Recognition: Crop the license plate ROI sub-image, input it into the visual text understanding model, extract the license plate number (including Chinese characters, letters, and numbers), and require a recognition confidence score V. c ≥Ω, 0.85≤Ω≤1.0, with Ω preferably being 0.85.
[0083] S4.4, Vehicle type determination: Convert the license plate ROI sub-image to HSV space, and count the percentage of blue (H:95-145, S:60-255, V:30-255) and green (H:30-90, S:60-255, V:30-255) pixels respectively; when the percentage of blue pixels is greater than ρ and greater than the percentage of green pixels, it is determined to be a gasoline vehicle; when the percentage of green pixels is greater than ρ and greater than the percentage of blue pixels, it is determined to be an electric vehicle, 0.4≤ρ≤1.0, and ρ is preferably 0.4.
[0084] S4.5, License Plate Voting Deduplication: The PlateDetectionManager class is called to count license plate detection results within a 5-second time window. If the cumulative number of valid votes is ≥φ, the license plate result is retained, and duplicate data is deleted. 3≤φ≤20, with φ preferably being 3. The above steps S4.2 to S4.5 complete the entire license plate recognition process. The garage landmark recognition process is as follows: Figure 4 As shown, the corresponding sub-steps S4.6 to S4.8 are as follows: S4.6, Landmark Region Detection: Input the enhanced feature map into the YOLO series model, output the landmark bounding box, filter the candidate box with confidence ≥ 0, 0.995≤0≤1.0, with 0 preferably being 0.995, and the average detection accuracy is ≥ 99.7%.
[0085] S4.7, Landmark Text Recognition: Crop the landmark ROI sub-image, input the fine-tuned visual text understanding model, extract the landmark number (e.g., "B1-025"), and the recognition accuracy is ≥99.4%.
[0086] S4.8, Landmark Anomaly Filtering: The LandmarkAnomalyDetector class is called to record the 25 most recent valid landmarks and calculate the deviation between the new landmark and the historical average. A landmark is considered valid when the deviation value △M ≤ 80; otherwise, it is considered invalid. dev When marked as a potential anomaly, 80 < T dev ≤200, T dev The preferred value is 80:
[0087] In the formula, K is the number of historical records, and M is the number of records. new For the new detection landmark value, M hist,i Let T be the value of the i-th historical landmark. dev This is the deviation threshold; when ΔM > T dev The time marker is marked as a potential anomaly. If the same anomaly range landmark is detected C times consecutively (with a deviation ≤ 20), it is determined to be a reasonable location jump. 10 ≤ C ≤ 30, and C is preferably 10.
[0088] After completing all visible light recognition processes from S4.1 to S4.8, the process of converting two-dimensional pixels to global three-dimensional world coordinates of the garage is executed, converting the license plate and landmark pixel positions into real spatial coordinates, which is divided into 5 execution steps: (1) Alignment of pose with image timestamp: When the spatial association module receives each frame of image, it matches the SLAM vehicle pose data that is closest to the image timestamp to ensure that the pose data and image data are synchronized in time and to eliminate coordinate calculation errors during the motion process.
[0089] (2) Acquisition of basic parameters: The real-time pose transformation matrix T from the vehicle coordinate system to the global world coordinate system of the underground parking garage at the current moment is obtained from the SLAM mapping module. wb Read the pre-calibrated camera intrinsic parameter matrix K and camera-vehicle extrinsic parameter matrix T from the configuration file. bc .
[0090] (3) Target depth value acquisition: Match the timestamps of the LiDAR point cloud and the image frame to extract the laser point cloud depth value z corresponding to the pixel coordinates of the license plate / landmark target.
[0091] (4) Coordinate transformation solution: The three-dimensional coordinates P of the target in the global world coordinate system are calculated by using the complete formula of pixel-to-world coordinate back projection. w The formula is as follows:
[0092] In the formula, (u,v) are the pixel center coordinates of the license plate / landmark target in the image; the pose and coordinate data of multiple frames before and after synchronous matching are averaged and smoothed to offset the conversion error caused by the instantaneous ranging jitter of the lidar and the millisecond-level time offset of image acquisition. The coordinate synchronization fault tolerance window is set to ±10ms.
[0093] (5) Spatiotemporal correlation record generation: The target identification content (license plate number / landmark number), the calculated world coordinates, and the current frame millisecond-level timestamp are bound together to generate a complete spatiotemporal correlation record, which is then sent to the central processing module.
[0094] S5 performs central data processing and dynamic association binding, establishing a spatial coordinate association between license plates and parking spaces using millisecond-level timestamps as unique IDs (running location: main server), including:
[0095] S5.1, timestamp synchronization and data association, including: In S5.1.1, the main service generates a millisecond-level timestamp current_time for each frame of image, synchronizes the license plate detection time plate_detection_time and the garage landmark detection time landmark_detection_time, and ensures that the time alignment error is ≤300ms.
[0096] S5.1.2 uses current_time as a unique ID and establishes a one-to-one mapping relationship between license plate and landmark data based on the overlap of world coordinates between license plate and landmark.
[0097] S5.2, Deduplication cleaning, including: S5.2.1, the main service performs duplicate data cleanup every 30 seconds, using a voting deduplication algorithm based on a 5-second time window:
[0098] In the formula, V weighted For weighted votes, T win =5s is the time window, I is the indicator function, L t Let L be the license plate number detected in the t-th test. consensus For the consensus license plate number, Where Dist(·) is the function for calculating the distance between detection boxes. max B is the preset maximum distance threshold for the detection box. t For the t-th detection box, B avg For the average detection frame, when the weighted vote count V weighted The results should be retained if there are at least 3 attempts.
[0099] S5.2.2, call the clean_duplicate_data function to delete duplicate detection records and retain only the best voting result.
[0100] S5.3, database update, including: S5.3.1 Update the parking_status table through atomic operations: when a valid license plate number is detected and the landmark is valid, mark is_occupied=1 (parking space occupied); when no license plate number is detected and the landmark is valid, mark is_occupied=0 (parking space vacant).
[0101] S5.3.2 calls the save_detection_history function to write the deduplicated valid records (including metadata such as detection time, confidence level, and recognition status) into the detection_history table, automatically retaining the most recent 100 records (which can be adjusted by parameters), and archiving expired records in descending order of timestamp.
[0102] S6 performs garage management and data interaction, dynamically updating and querying parking space status based on spatial coordinate relationships (running location: main server, management terminal), including:
[0103] S6.1, multi-dimensional query service, including: S6.1.1, Single parking space status query: The administrator enters the landmark ID on the management terminal of the parking garage management center (deployed in the parking garage management center), calls the function to query the parking_status table, and returns the real-time status of the parking space, such as the license plate number, vehicle type, and update time, with a response time of ≤50ms.
[0104] S6.1.2, Generation of Available Parking Spaces: The management terminal reads the parking_status table in batches, filters records with is_occupied=0, and generates a list of available parking spaces. Batch query time is ≤200ms, and front-end visualization is supported.
[0105] S6.1.3, Parking duration calculation: The management terminal inputs the license plate number, queries the detection_history table, and calculates the parking duration based on plate_detection_time and the current time (or the last detection time when the vehicle leaves the site), with an error of ≤1 second.
[0106] S6.2, Anomaly Marking and Historical Tracing, including: S6.2.1, the main server, in conjunction with the sub-service health monitoring mechanism, marks the anomaly type and cause in the detection_history table when the identified status is abnormal (such as landmark_status=anomaly or plate_status=error).
[0107] S6.2.2, through the management terminal, managers can query historical detection records by license plate number, time range, or anomaly type, providing data basis for fault diagnosis and system optimization.
[0108] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A dual-modal, GNSS-signal-free underground parking garage unmanned vehicle patrol management system, characterized in that, It includes a SLAM mapping module, a visible light recognition module, a central processing module, a garage management module, a passive RFID spatial anchoring module, and a spatial association module; The SLAM mapping module outputs the pose information of the unmanned patrol vehicle in the world coordinate system in real time. The visible light recognition module is used to identify vehicles in underground parking garages and their corresponding garage landmarks; The spatial association module receives the pose information from the SLAM mapping module and the pixel coordinates from the visible light recognition module. Combined with the pre-calibrated camera intrinsic parameters and camera-vehicle extrinsic parameters, the recognition results are mapped to the unified world coordinate system of the underground parking garage through pixel-to-world coordinate back projection. The central processing module receives pose, target recognition, and 3D spatial coordinate data, and establishes a dynamic binding relationship of "license plate-landmark-spatial coordinate-timestamp" with a millisecond-level timestamp as a unique association ID; The garage management module stores garage data and updates and maintains the database in real time. Based on the administrator's instructions, it returns real-time information such as the garage's available parking space number, vehicle parking location, specific parking space and corresponding license plate number. The passive RFID spatial anchoring module includes several passive electronic tags that are fixedly deployed at designated locations in the underground parking garage. The passive electronic tags serve as the initial spatial anchor points of the global unified world coordinate system of the underground parking garage, used to calibrate the coordinate reference of the global coordinate system, perform initial alignment, and perform cumulative drift correction for the SLAM mapping module.
2. The dual-modal GNSS-free underground parking garage unmanned vehicle patrol management system according to claim 1, characterized in that, The visible light recognition module includes an image preprocessing unit, a robust feature enhancement submodule, a license plate recognition unit, and a garage landmark recognition unit. The robust feature enhancement submodule is used to perform multi-scale edge enhancement, adaptive contrast enhancement, and frequency domain ghosting removal on the preprocessed image. The robust feature enhancement submodule includes a multi-scale edge perception unit, an adaptive contrast enhancement unit, a frequency domain ghosting removal unit, and a cross-feature attention fusion unit. The multi-scale edge perception unit uses multiple dilated convolution branches with different dilation rates to extract edge features at different scales and adaptively fuses features at each scale through a channel attention mechanism. The adaptive contrast enhancement unit performs illumination-reflection decomposition based on the four-scale Retinex theory and achieves nonlinear contrast enhancement through learnable γ parameters; the frequency domain deghosting unit detects symmetric conjugate peaks in the frequency domain through fast Fourier transform and uses learnable mask to suppress ghosting frequency components. The cross-feature attention fusion unit generates dynamic weights through global feature perception and adaptively weights and fuses the outputs of multiple image processing branches. The image preprocessing unit has low-light enhancement, image denoising and distortion correction functions. After the image preprocessing unit completes the processing, it sends the image to the robust feature enhancement submodule for feature enhancement, while retaining the distortion parameters in the camera. The robust feature enhancement submodule outputs the enhanced feature map.
3. The dual-modal GNSS-free underground parking garage unmanned vehicle patrol management system according to claim 1, characterized in that, The operation of the spatial association module specifically includes: (1) Depth acquisition: The global real-time pose of the unmanned patrol vehicle output by the SLAM mapping module is used as the conversion reference, and the depth information of the target is obtained by combining the lidar point cloud data; (2) Pixel to camera coordinate back projection: Combining the aforementioned target depth information and camera intrinsic parameter matrix, the pixel center point coordinates (u,v) of the target in the image are back projected into a three-dimensional point in the camera coordinate system; (3) Camera to vehicle body rigid body transformation: The three-dimensional points are transformed from the camera coordinate system to the vehicle coordinate system through the camera-vehicle external parameter matrix; (4) Transformation from vehicle body to world rigid body: The three-dimensional points are transformed from the vehicle body coordinate system to the global world coordinate system of the underground garage through the real-time pose output by the SLAM mapping module.
4. The dual-modal GNSS-free underground parking garage unmanned vehicle patrol management system according to claim 1, characterized in that, The operation of the central processing module specifically includes: Spatiotemporal correlation data synchronization and merging: The spatiotemporal correlation records output by the spatial correlation module are transmitted to the central processing module and then synchronized to the main database service in real time. This triggers a two-way merging mechanism between the parking space spatial coordinate database and the license plate recognition database. Based on the timestamp, the spatial coordinates of the parking space landmark and the spatial coordinates of the license plate at the same time are correlated in real time to establish a one-to-one correspondence between "parking space and vehicle". At the same time, the landmark_detection_time, plate_detection_time and spatial coordinate fields of the parking_status table are supplemented to provide data support for subsequent calculation of vehicle parking time and parking space status tracking. Duplicate data cleanup is performed every few seconds, based on spatial coordinate overlap and timestamp to remove duplicates, and valid binding records are written to the detection_history table. It provides database query operations, supporting real-time retrieval of merged complete parking records by landmark ID, parking space coordinates, or license plate number; Timestamp synchronization and cross-service alignment are based on the main service's millisecond-level timestamp current_time, synchronizing the landmark detection time, license plate detection time, and SLAM pose output time returned by the sub-services.
5. The dual-modal GNSS-free underground parking garage unmanned vehicle patrol management system according to claim 1, characterized in that, The garage management module includes: The core database management unit initializes and maintains the `garage_merged.db` database, which includes the `parking_status` and `detection_history` tables. Atomic operations ensure data consistency under multi-threaded concurrency. The `parking_status` table pre-stores a fixed world coordinate range corresponding to each parking space landmark. By determining whether the world coordinates of the license plate fall within the coordinate range of the corresponding parking space, precise matching of parking spaces and vehicles is achieved. Furthermore, the spatiotemporal association records output by the spatial association module are synchronized in real time. Based on timestamps and world coordinates, landmarks and `license_plate` information are associated, and the `is_occupied` field is dynamically updated to mark the parking space occupancy status. The data interaction and command response unit provides multi-dimensional query interfaces: it supports precise querying of the real-time status of a single parking space by landmark ID, or batch return of the occupancy status of all parking spaces to generate a list of available parking spaces; it traces vehicle parking records by license plate number and calculates the actual parking time of the vehicle by combining landmark_detection_time and plate_detection_time. The historical data tracing and maintenance unit automatically stores several recent detection history records, including details of landmark / license plate recognition results, service response time, and recognition status, for troubleshooting and data auditing. Secondly, combined with the sub-service health monitoring mechanism, when an abnormal recognition is detected, the abnormal status is marked in the historical record.
6. The dual-modal GNSS-free underground parking garage unmanned vehicle patrol management system according to claim 1, characterized in that, The camera intrinsic parameters include the camera focal length, principal point coordinates, and distortion coefficients; the camera-to-vehicle extrinsic parameters include a 3×3 rotation matrix and a 3×1 translation vector between the camera coordinate system and the vehicle coordinate system; the coordinate transformation of the spatial association module includes two parts: pixel-to-world coordinate back projection and rigid body transformation. (1) Pixel to camera coordinate back projection: Combining the target depth value z obtained by the lidar and the pre-calibrated camera intrinsic parameter matrix K, the pixel center point coordinates (u,v) of the target in the image are back projected into a three-dimensional point in the camera coordinate system. (2) Rigid body transformation: including the external parameter transformation from the camera to the vehicle body and the pose transformation from the vehicle body to the world, both of which are rotation and translation transformations between three-dimensional coordinate systems; The coordinate transformation formula for the spatial association module is: , In the formula: P w =(X w ,Y w Z w ) T The target's three-dimensional coordinates are given in the global world coordinate system of the underground parking garage, with the superscript T indicating transpose. T wb The real-time pose transformation matrix from the vehicle coordinate system to the world coordinate system output by the SLAM mapping module; T bc This is the extrinsic parameter matrix from the pre-calibrated camera coordinate system to the vehicle coordinate system; z represents the target depth value obtained by the lidar; (u,v) represents the coordinates of the center point of the pixel in the image.
7. A method for patrolling and managing unmanned vehicles in a dual-mode GNSS-free underground parking garage, implemented using the dual-mode GNSS-free underground parking garage unmanned vehicle patrolling and management system as described in any one of claims 1-6, characterized in that... include: S1, to perform hardware deployment and global coordinate system calibration, to calibrate the global unified world coordinate system of the underground garage through passive RFID electronic tags, to complete the accurate calibration of the camera's internal parameters and the camera-vehicle external parameters, and to deploy the equipment at the underground garage site; S2 is used for software deployment and environment configuration. S3 performs dual-modal data acquisition and preprocessing, simultaneously acquiring LiDAR data required for SLAM mapping and image data required for visible light recognition; S4 performs visible light target recognition and target spatial coordinate calculation. Based on the real-time pose output by the SLAM mapping module, the recognized target in the two-dimensional image is mapped to the global world coordinate system, and the three-dimensional spatial coordinates of the target are calculated. S5 performs central data processing and dynamic association binding, using millisecond-level timestamps as unique IDs to establish spatial coordinate associations between license plates and parking spaces. S6 enables garage management and data interaction, allowing for dynamic updates and queries of parking space status based on spatial coordinate relationships.
8. A method for patrolling and managing unmanned vehicles in a dual-modal underground parking garage without GNSS signals, as described in claim 7, is characterized in that... S3 includes: S3.1, Multi-source data acquisition initiated, including: S3.1.1, Log in to the unmanned patrol vehicle terminal and start the 360° RGB camera; S3.1.2 The unmanned patrol vehicle patrols along a preset path, with the camera collecting images, the lidar collecting point cloud data, and the inertial navigation unit collecting attitude data. S3.1.3, image data is transmitted to edge nodes via the network, while point cloud data and attitude data are stored in the vehicle's embedded unit for SLAM positioning updates; S3.2, Image preprocessing is performed, including: S3.2.1, Image scaling: The edge node calls the img_first_pre function to scale the acquired image proportionally using the INTER_AREA interpolation method; S3.2.2, Gaussian Denoising: The scaled image is denoised using the cv2.GaussianBlur function; S3.2.3, Grayscale Conversion and Binarization: First, the color image is converted to a grayscale image, and then the Otsu adaptive thresholding algorithm is used to achieve automatic thresholding segmentation, thereby improving the contrast between the target area and the background. S3.3, SLAM localization and map update; S3.3.1, the SLAM mapping module integrates LiDAR point cloud and inertial navigation data to estimate the position and attitude of the unmanned patrol vehicle on the map in real time; S3.3.2, combined with dynamic environmental changes, optimizes map accuracy in real time to ensure that the vehicle patrols stably along the planned route without deviation.
9. A method for patrolling and managing unmanned vehicles in a dual-modal underground parking garage without GNSS signals, as described in claim 7, is characterized in that... S4 includes: S4.1, the preprocessed image is input into the robust feature enhancement submodule, and multi-scale edge perception, adaptive contrast enhancement and frequency domain ghosting processing are performed in sequence. Enhanced feature map is generated by cross-feature attention fusion. S4.2, input the enhanced feature map into the YOLO series model, output the license plate bounding box (x1, y1, x2, y2), and filter the confidence score V. label Candidate boxes with a value ≥ μ, where 0.7 ≤ μ ≤ 1.0; S4.3, crop the license plate ROI sub-image, input it into the visual text understanding model, extract the license plate number, and require a recognition confidence score V. c ≥Ω, 0.85≤Ω≤1.0; S4.4, convert the license plate ROI sub-image to HSV space, and count the proportion of blue and green pixels respectively. Based on the proportion of blue and green pixels, determine whether the vehicle is a gasoline vehicle or an electric vehicle. S4.5 calls the PlateDetectionManager class to count license plate detection results within a time window of several seconds, accumulating the number of valid tickets V. sum If the value is greater than or equal to φ, then keep the license plate result and delete duplicate data; 3 ≤ φ ≤ 20. S4.6 inputs the enhanced feature map into the YOLO series model, outputs landmark bounding boxes, and filters the confidence scores V. area Candidate boxes with a value ≥0, where 0.995 ≤ 0 ≤ 1.0; S4.7, crop the landmark ROI sub-image, input the fine-tuned visual text understanding model, and extract the landmark number; S4.8 calls the LandmarkAnomalyDetector class to record several recent valid landmarks and calculate the deviation between the new landmark and the historical average; when the landmark recognition deviation value △M≤T dev If the landmark identification deviation value △M > T, it is considered valid; dev When marked as a potential anomaly, 80≤T dev ≤200.
10. A method for patrolling and managing unmanned vehicles in a dual-modal underground parking garage without GNSS signals, as described in claim 7, is characterized in that... S5 includes: The main service generates a millisecond-level timestamp current_time for each frame of image, and synchronizes the license plate detection time plate_detection_time and the landmark detection time landmark_detection_time. Using current_time as a unique ID, and based on the overlap between the license plate and the world coordinates of the landmark, the license plate and landmark data are linked to establish a one-to-one mapping relationship of "landmark number - parking space fixed spatial coordinates - license plate world coordinates - license plate number"; The main service performs duplicate data cleanup every few seconds, using a voting deduplication algorithm based on a few-second time window: , In the formula, V weighted For weighted votes, T win Let I be the time window, and L be the indicator function. I takes the value 1 if the condition within the parentheses is true, and 0 if it is false. t Let w(t) be the license plate number detected in the t-th test, and L be the weight of the t-th test. consensus For the consensus license plate number, Where Dist(·) is the function for calculating the distance between detection boxes. max B is the preset maximum distance threshold for the detection frame; t For the t-th detection box, B avg For the average detection frame, when the weighted vote count V weighted The result is retained when the value is ≥3; Call the `clean_duplicate_data` function to delete duplicate detection records, retaining only the best voting result; The parking_status table is updated using atomic operations: when a valid license plate number is detected and the landmark is valid, is_occupied=1 is set to indicate that the parking space is occupied; when no license plate number is detected but the landmark is valid, is_occupied=0 is set to indicate that the parking space is vacant. Call the save_detection_history function to write the deduplicated valid records into the detection_history table, automatically retain the most recent few records, and archive expired records in descending order of timestamp.
