Low-cost service area vehicle monitoring method based on AR eagle-eye panoramic camera

Abstract

The invention discloses a low-cost service area vehicle monitoring method based on an AR eagle-eye panorama camera. The method comprises the following steps: acquiring multiple video streams by deploying the AR eagle-eye panorama camera and splicing the video streams to generate a high-resolution panorama; a self-adaptive tile slicing strategy based on the vehicle type size is adopted to preprocess the panorama, and a lightweight target detection model is utilized to identify the vehicle target, the type and the orientation angle; integrating and outputting a panoramic detection result through inverse coordinate mapping and a cross-tile duplicate removal technology; carrying out multi-event real-time verification based on a pre-calibrated service area multifunctional partition so as to complete abnormal preliminary screening; and finally, the ball machine is driven to locally amplify the abnormal area and carry out secondary reasoning verification so as to improve the detection accuracy. And locally recording structured data of the abnormal event, continuously tracking and recording an event video, and pushing the event video to a service area operator terminal for handling. According to the invention, high-precision vehicle monitoring under single-device wide-area coverage is realized, and the system construction and maintenance cost is obviously reduced.

Description

Technical Field

[0001] This invention relates to the field of highway management technology, and in particular to a low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera. Background Technology

[0002] With the continuous densification of the national expressway network, service areas have evolved from rest stops into commercial complexes along transportation corridors, leading to a sustained increase in daily traffic volume. However, this surge in traffic has highlighted vehicle management issues within service areas, making it a key factor affecting operational efficiency and service quality. Therefore, implementing vehicle management in service areas is of significant practical importance. Currently, vehicle monitoring in service areas primarily relies on geomagnetic parking space detectors and traditional bullet / PTZ camera arrays, but these methods also face a series of challenges:

[0003] 1) Poor environmental adaptability: The geomagnetic detector buries a geomagnetic sensor in the center of the parking space and collects magnetic field or distance changes through an MCU. However, it is easily affected by interference in environments such as rain, snow, and high temperatures, causing magnetic field drift and resulting in false alarms or missed alarms. 2) Large single-point blind zone: The geomagnetic detector only covers 1m of the center of the parking space. 2 After a vehicle enters, the system outputs an "occupied / vacant" signal through magnetic field disturbance and distance sensing. However, it can only identify whether a parking space is occupied, and it is prone to false alarms and missed alarms in cases of interference from adjacent parking spaces or adjacent vehicles crossing the line. It cannot accurately determine irregular scenarios such as angled parking spaces, unmarked parking spaces, and multiple vehicles parked side by side. 3) High installation and maintenance costs: The installation of the geomagnetic detector requires burying a geomagnetic sensor in the center of the parking space, which has a long construction period and requires damage to the road surface. If the battery or sensor fails later, it is necessary to excavate and repair it again. Bullet or PTZ camera arrays require multiple cameras to be installed at a fixed height, and parking spaces need to be jointly calibrated among multiple cameras, resulting in high installation costs. 4) Inability to monitor parking spaces without spaces: The geomagnetic detector and bullet or PTZ camera arrays are mainly used to monitor marked parking spaces. However, in service areas with high traffic volume or congestion during holidays, and when there are insufficient existing parking spaces, vehicles illegally park and occupy driving lanes, or even overflow into green belts and ramp entrances, affecting the normal passage of service areas. The system cannot effectively monitor these situations, leading to congestion spreading without data feedback. 5) Poor scalability: Adding new monitoring areas requires repeated construction or the installation of additional cameras, and the parking spaces between cameras need to be recalibrated. Adding parking spaces also requires digging up the road surface to bury sensors and wiring, making it difficult to respond quickly to changes in demand. Summary of the Invention

[0004] The purpose of this invention is to provide a low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera.

[0005] The technical solution adopted in this invention is:

[0006] A low-cost service area vehicle monitoring method based on AR eagle-eye panoramic cameras includes the following steps:

[0007] S1. Obtain the real-time video stream from the AR Eagle Eye camera in the service area, and extract keyframe images from multiple wide-angle lenses periodically.

[0008] S2. Combine keyframe images from multiple wide-angle lenses to create a high-resolution panoramic image.

[0009] S3. Preprocess the high-resolution panoramic image by implementing an adaptive tile slicing strategy according to vehicle type to obtain several tile images of the vehicle target;

[0010] S4. Use a lightweight object detection model to infer the current vehicle detection result for each tile image. The current vehicle detection result includes vehicle type, confidence level and orientation angle.

[0011] S5. Perform coordinate inverse mapping and cross-tile deduplication on the vehicle detection results of each tile image, and output the final panoramic image detection result; the panoramic image detection result includes the number of vehicles, vehicle type, and vehicle orientation angle;

[0012] S6. Based on high-resolution panoramic images, the service area plaza is divided into functional zones, including vehicle counting areas, parking areas divided by vehicle type, refueling queue areas, dedicated charging pile areas, and no-parking areas.

[0013] S7. Based on the panoramic image detection results and the functional area zoning of the service area square, implement real-time verification of multiple events to complete the initial screening;

[0014] S8. For the initial screening of abnormal events, control the AR Eagle Eye camera to zoom in on the abnormal area locally, and use the same lightweight target detection model for secondary inference and verification to further improve the accuracy of event detection.

[0015] S9. After confirming that the abnormal event is correct, record the structured data of the abnormal event locally and continuously track and record the event video, and push it to the terminal of the service area duty personnel for handling.

[0016] Furthermore, in S1, the real-time sub-streams of multiple wide-angle lenses of the AR Eagle Eye camera are simultaneously pulled through the RTSP protocol, and the timestamps are calibrated using the NTP protocol to achieve frame-level synchronization; a timed frame extraction thread is set up to skip ordinary frames and cache the latest keyframe images of each lens according to the lens number; a synchronization timestamp, frame number and pre-calibrated lens distortion parameters are added to the latest keyframe image to form a complete data packet for subsequent stitching and coordinate mapping.

[0017] Specifically, when any RTSP interrupts, the corresponding thread automatically reconnects within a set time (3 seconds), and the previous frame is used to fill in the gap during the interruption to avoid a black screen.

[0018] Furthermore, S2 includes the following steps:

[0019] S2-1 performs color equalization and histogram matching on each keyframe image to eliminate lens vignetting and color difference;

[0020] S2-2, calibrating the internal parameters of each wide-angle lens extrinsic parameter matrix The distortion coefficients are used, where R represents the rotation extrinsic matrix between the two lenses and t represents the translation extrinsic matrix between the two lenses. The distortion is removed one by one from the keyframe images of each wide-angle lens obtained by synchronous frame extraction.

[0021] S2-3: Project the corrected image onto a spherical model, extract feature points using ORB / SURF, calculate the homography matrix using RANSAC, and complete sub-pixel level alignment.

[0022] S2-4 uses bundle adjustment to accurately adjust intrinsic and extrinsic parameters and distortion coefficients to ensure geometric consistency;

[0023] S2-5, construct a multi-band Laplacian pyramid, generate smooth blending weights in overlapping areas, and eliminate seams and brightness differences;

[0024] S2-6, synthesize and output a high-resolution panoramic image, and generate a reverse mapping table between global coordinates and the pixels of each lens, which can be directly called for subsequent detection and coordinate mapping;

[0025] Furthermore, S3 includes the following steps:

[0026] S3-1: Establish a mapping table based on different vehicle size-tiles, and set the corresponding tile size for different vehicle models; write the mapping table into the configuration file so that the corresponding tile can be dynamically selected according to the vehicle model label at runtime.

[0027] Specifically, the vehicle models are divided into microcars, medium-sized cars and large cars, and the corresponding tile sizes are 512×512, 640×640 and 768×768. The table is written into the configuration file and dynamically selected according to the vehicle model label at runtime.

[0028] S3-2, Tile slicing of the high-resolution panoramic image: Determine the grid to which the center point of the vehicle detection box belongs, and slice the image into tiles centered on the corresponding grid. Each tile expands outwards by a preset ratio to ensure that the feature overlap between adjacent tiles is no less than 40%. The vehicle detection box is generated by the target detection model, and the grid center coordinates are the center pixel coordinates of the vehicle detection box.

[0029] Furthermore, in S3-2, when two vehicle models appear in the same grid, the grid tile size is selected according to the principle of "largest vehicle model priority" to avoid cross-vehicle truncation.

[0030] Specifically, each tile expands outward by 20% on all four sides to ensure that the feature overlap between adjacent tiles is ≥40%, thereby reducing the probability of large vehicles being segmented.

[0031] S3-3: Calculate the ratio of the vehicle detection box to the tile side length in each tile image and determine if the ratio exceeds the set value. If it does, determine that the corresponding tile image has a complete target vehicle and execute S3-5. Otherwise, determine that there is a vehicle truncation in the tile image, merge the truncated tile image with the adjacent tiles into a new tile of 1.5 × the original size, and then execute S3-4.

[0032] Specifically, after the vehicle target detection model outputs the vehicle bounding box within the tile, it first determines the target as "complete target" if "bounding length / tile side length" > 0.6, and marks it as "potentially truncated" if ≤ 0.6.

[0033] S3-4: Determine if the tile image has reached the maximum tile size of 1024×1024; if yes, execute S3-5; otherwise, execute S3-3.

[0034] S3-5 records the row and column indices and four-point coordinates of each tile image in the high-resolution panoramic image, which can be directly reused for subsequent model inference and coordinate inverse mapping.

[0035] Furthermore, S4 includes the following steps:

[0036] S4-1, A lightweight object detection model is selected to infer the tile image, the tile image is adjusted to the model input size, and normalization is performed;

[0037] S4-2: Input the preprocessed image into the target detection model. After parallel inference for each tile image, output the coordinates of the 2D detection box of the vehicle. Category labels and confidence levels; among which, These represent the x and y coordinates of the top-left and bottom-right corners of the detection box, respectively.

[0038] S4-3, Calculate the coordinates of the center point of the detection box. ;

[0039] S4-4, based on the vehicle type and the aspect ratio of the vehicle detection frame, keep the direction of the vehicle's front end consistent with the direction of the long side of the vehicle detection frame, and obtain the orientation angle by calculating the angle between the long side of the vehicle's detection frame and the horizontal direction;

[0040] S4-5 integrates vehicle type and vehicle orientation angle information into the detection results to form structured output data; at the same time, output data with a confidence level greater than a set threshold is written to a cache so that it can be stitched back into the high-resolution panoramic image coordinate system according to the tile image row and column index and the global coordinate mapping table for subsequent NMS merging.

[0041] Furthermore, S5 includes the following steps:

[0042] S5-1 uses the coordinate mapping relationship between the high-resolution panoramic image and each tile image to inversely transform the vehicle detection box coordinates in each tile image to the global coordinate system of the panoramic image through a pre-stored transformation matrix.

[0043] S5-2 uses the Soft-NMS algorithm, sets the cross-union ratio threshold and attenuation parameters, and performs confidence attenuation and deduplication on the duplicate detection boxes across tiles according to the vehicle detection boxes with the highest confidence, to obtain the vehicle retrieval box retention list.

[0044] Specifically, the input detection box set is deduplicated across tiles using the threshold set by Soft-NMS. Each detection box contains coordinates and confidence level Simultaneously, set the intersection-over-union (IoU) threshold between vehicle detection frames. and attenuation parameters Among these factors, the selection of the IoU threshold affects the stringency of the detection results. A lower threshold retains more overlapping boxes, while a higher threshold more strictly filters boxes, requiring the removal of other boxes whose overlap exceeds the threshold. (Based on confidence level...) Sort the detection boxes from highest to lowest confidence level to obtain a sorted index list, and initialize an empty list to store the final retained detection box indices; select the detection box with the highest confidence level from the sorted detection boxes. Add its index to the reserved list; calculate the detection box. The Intersection over Union (IoU) with the remaining detection boxes is used to weight the confidence of the remaining detection boxes based on the IoU value. calculate:

[0045] ;

[0046] Wherein, IoU is the intersection-union ratio of two detection boxes. It is a parameter for adjusting the attenuation level. The smaller the value, the faster the confidence of high IoU detection boxes decreases; the larger the value, the smoother the decay.

[0047] Multiply the confidence scores of the remaining detection boxes by the corresponding decay weights, and remove detection boxes with confidence scores below a threshold. The threshold is a pre-defined minimum confidence standard used to filter out detection boxes that the vehicle target detection model considers unlikely to be target objects. Repeat the calculation of detection box confidence scores, compare the confidence scores with the threshold, and repeat the filtering and removal of low-confidence detection boxes until all detection boxes have been processed. Output a retention list, which contains the indices of the final retained detection boxes.

[0048] S5-3 integrates the vehicle detection box coordinates, vehicle type, and vehicle orientation angle information from the retained list after inverse mapping to form a structured detection result; finally, it outputs the detection result of a high-resolution panoramic image, including the number of vehicles, vehicle type, and vehicle orientation angle information.

[0049] Furthermore, S6 includes the following steps:

[0050] S6-1, based on high-resolution panoramic images, uses the corrected normalized spherical coordinates as a reference, then backprojects them to pixel coordinates, and uses the polygon ROI tool to calibrate area by area.

[0051] S6-2, Divide the overall vehicle counting area: Divide the service area plaza into an area for counting the overall number of vehicles, covering the entire parking area, refueling area, charging station area and all possible parking areas of vehicles;

[0052] S6-3: On a high-resolution panoramic image or parking area planning map, use the polygon tool to outline the boundaries of parking areas for each vehicle type and bind attributes to each area.

[0053] S6-4, based on vehicle size and usage requirements, divides the parking area into a small car parking area, a large bus parking area, a large truck parking area, and a hazardous chemical transport vehicle parking area;

[0054] S6-5, delineate refueling queue waiting areas near gas stations and mark the start and end points of the refueling queue area, as well as the width and direction of the queue lane, on a high-resolution panoramic image;

[0055] S6-6 marks the location of charging piles and the boundaries of dedicated parking areas for charging piles on a high-resolution panoramic map, while also delineating the vehicle queuing area.

[0056] S6-7, demarcate no-parking zones within the service area plaza. These zones include fire lanes, pedestrian crossings, and plaza entrances / exits. Use a polygon tool on a high-resolution panoramic image to delineate the boundaries of the no-parking zones.

[0057] Furthermore, the event types verified in S7 include at least one of the following: overloaded plaza vehicles when the total number of vehicles exceeds a preset threshold; vehicle type mismatch event when the vehicle type does not match the parking area binding attribute; abnormal posture event when the vehicle's orientation angle exceeds the allowed range of the parking area; no-parking zone occupancy event when the vehicle detection box falls into the no-parking zone; and congestion event when the number of vehicles in the refueling / charging queue area exceeds a threshold or the queuing time exceeds the time limit.

[0058] Specifically, all vehicle detection boxes detected by the high-resolution panoramic image are aggregated, and duplicate counts are eliminated by NMS to obtain the real-time total number of vehicles; when the total number of detected vehicles exceeds a preset threshold, it is marked as a "square vehicle overload" event.

[0059] Based on the vehicle type and permitted parking type rules, bind attributes to each parking space polygon area and write them to the hierarchical configuration file; save the parking area and the coordinates of the four vertices as JSON for direct use in subsequent anomaly detection.

[0060] For each parking area, extract the vehicle detection bounding box within the corresponding area from the panoramic image detection results, verify the following events in real time, and process them accordingly:

[0061] ;

[0062] When the detected vehicle type does not match the parking area, it is marked as a "vehicle type mismatch" event and detailed information of the vehicle type mismatch event is output, including vehicle type, parking area, and panoramic coordinates;

[0063] The attitude anomaly verification covers all parking areas and charging pile areas, and detects whether the vehicle's orientation angle conforms to the prescribed attitude of the parking area; it compares the detected vehicle orientation angle with the allowed attitude angle range of the parking area; when the vehicle orientation angle exceeds the allowed range, it is marked as an "attitude anomaly" event, and the detailed information of the attitude anomaly event is output, including vehicle type, parking area, vehicle orientation angle, and parking area orientation angle.

[0064] Virtually delineate no-parking zones in the panoramic image, including fire lanes, pedestrian crossings, and plaza entrances / exits; extract vehicle bounding boxes within the no-parking zones from the detection results, and compare the 2D bounding box coordinates (x0, y0) of each vehicle. 0, To determine if a vehicle is illegally parked, check if its coordinates (x1, y1) overlap with a no-parking zone. If a vehicle is found to be parked in a no-parking zone, mark it as a "no-parking zone occupancy" event and output detailed information about the event, including vehicle type, no-parking zone, and vehicle's panoramic coordinates.

[0065] A queuing area is defined at the entrance of the gas station. A threshold for the number of vehicles in the waiting area is set, as well as a threshold for the trigger time within the waiting area. A filtering time threshold is also set to avoid repeated alarms during prolonged congestion.

[0066] In the dedicated charging station area, the vehicle orientation is monitored in real time to ensure that the vehicle is parked in a standardized manner and does not encroach on the charging space. A queuing area is marked in the charging station area, and the number of vehicles in the queue is counted in real time. When the number of vehicles in the queue exceeds a preset threshold, it is marked as a "queuing area congestion" event.

[0067] Furthermore, in S8, the initial screening of abnormal events is partially magnified by the AR Eagle Eye camera and then verified through secondary inference using the same model, further improving the accuracy of event detection, including:

[0068] Mapping matrix from offline calibrated panoramic coordinate system to PTZ PTZ camera coordinate system Calculate the required horizontal angle, pitch angle and zoom of the PTZ camera, and drive the PTZ camera to turn to the abnormal area;

[0069] Once the PTZ camera has stabilized, capture keyframes as images of details with anomalies.

[0070] The detailed anomalous image is input into the same object detection model for secondary inference verification. The anomalous image, after being magnified, has richer details, which allows the model to more accurately identify and verify anomalous events.

[0071] Specifically, offline calibration of the panoramic-PTZ camera mapping matrix. After fixing the AR Eagle Eye camera, a laser rangefinder was used to measure the height and corresponding pixel coordinates of four ground control points. Find the 3×3 homography matrix that maps the panoramic spherical coordinate system to the PTZ coordinate system. Save at the same time ,focal length and size factor ; through the calibrated panoramic-PTZ camera mapping matrix Calculate the PTZ camera's horizontal angle, pitch angle, and zoom to ensure that abnormal areas occupy the entire frame. After the PTZ camera turns to the target and stabilizes for 1 second, it captures keyframes from the sub-stream and saves them as detailed anomaly images. These saved detailed anomaly images are then re-input into the same detection model for secondary inference. The magnified, more detailed anomaly images allow the model to more accurately identify and verify anomaly events. Based on the S8 control system, the PTZ camera continuously tracks the anomaly area, maintaining stable monitoring of the target region to ensure complete recording of anomalies. After tracking, keyframes from the sub-stream are captured and saved as detailed anomaly images.

[0072] Furthermore, after confirming that the event is correct in S9, the system first records the detailed information of the abnormal event locally and generates a JSON format log file. The log file includes the event type, trigger time, panoramic coordinates, and vehicle type. At the same time, the system pushes the abnormal event information to the terminal device of the service area duty personnel. After receiving the push, the duty personnel can respond quickly according to the abnormal details and go to the scene to handle the situation.

[0073] The present invention, employing the above technical solutions, has the following advantages and beneficial effects: 1) Low-cost installation and deployment: A single AR Eagle Eye panoramic camera can achieve vehicle monitoring and assessment of the service area's vehicle carrying capacity, significantly reducing construction costs and subsequent maintenance expenses. 2) Vehicle-type tile slicing strategy improves panoramic image detection accuracy: A mapping table is established based on different vehicle types and sizes, dynamically selecting corresponding tile sizes. Combined with a maximum vehicle type priority, edge expansion, and feature overlap mechanism, the integrity of vehicle detection in the panoramic image is ensured, effectively improving the accuracy of vehicle detection in the panoramic image. 3) Single-device panoramic image area hierarchical calibration: A single AR Eagle Eye panoramic camera achieves full coverage of the service area plaza and accurately calibrates different functional areas. This not only significantly reduces equipment procurement and maintenance costs but also solves the problem of multi-device data fusion, improving data consistency and real-time performance. Simultaneously, it simplifies the system architecture and enhances the convenience of service area management and the efficiency of event monitoring. 4) The "panoramic initial screening + detailed anomaly magnification" architecture improves the accuracy of abnormal event detection: Panoramic images are used to initially screen events and quickly locate abnormal event areas. The AR Eagle Eye camera is used to magnify the abnormal areas locally, and the same model is used for secondary inference verification. This architecture combines the efficient screening capability of panoramic images with the accurate recognition capability of detailed images, which can effectively improve the accuracy and reliability of abnormal event detection in service area squares. Attached Figure Description

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0075] Figure 1 This is a flowchart illustrating the low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to the present invention.

[0076] Figure 2 This is a schematic diagram of the structure of the AR Eagle Eye Panoramic Camera based on the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0078] This invention proposes a low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera. This method addresses five core shortcomings of existing service area vehicle monitoring systems: poor environmental adaptability, large blind spots in single-point monitoring, high installation and maintenance costs, missed detections, and poor scalability. Through a three-level architecture of "one camera with multiple lenses + lightweight model perception + anomaly amplification and secondary inference," a single AR eagle-eye camera can achieve panoramic coverage of the service area parking area, real-time detection of vehicle refueling and charging queues, identification of parking space anomalies, and assessment of the service area's vehicle carrying capacity. This significantly reduces construction costs and improves detection accuracy, providing a low-cost, high-precision, and easily deployable smart solution for service area vehicle management in highway digital service areas.

[0079] like Figure 1 As shown in Figure 2, this invention discloses a low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera, which includes the following steps:

[0080] S1. Obtain the real-time video stream from the AR Eagle Eye camera in the service area, and extract keyframe images from multiple wide-angle lenses periodically.

[0081] S2. Combine keyframe images from multiple wide-angle lenses to create a high-resolution panoramic image.

[0082] S3. Preprocess the high-resolution panoramic image by implementing an adaptive tile slicing strategy according to vehicle type to obtain several tile images of the vehicle target;

[0083] S4. Use a lightweight object detection model to infer the current vehicle detection result for each tile image. The current vehicle detection result includes vehicle type, confidence level and orientation angle.

[0084] S5. Perform coordinate inverse mapping and cross-tile deduplication on the vehicle detection results of each tile image, and output the final panoramic image detection result; the panoramic image detection result includes the number of vehicles, vehicle type, and vehicle orientation angle;

[0085] S6. Based on high-resolution panoramic images, the service area plaza is divided into functional zones, including vehicle counting areas, parking areas divided by vehicle type, refueling queue areas, dedicated charging pile areas, and no-parking areas.

[0086] S7. Based on the panoramic image detection results and the functional area zoning of the service area square, implement real-time verification of multiple events to complete the initial screening;

[0087] S8. For the initial screening of abnormal events, control the AR Eagle Eye camera to zoom in on the abnormal area locally, and use the same lightweight target detection model for secondary inference and verification to further improve the accuracy of event detection.

[0088] S9. After confirming that the abnormal event is correct, record the structured data of the abnormal event locally and continuously track and record the event video, and push it to the terminal of the service area duty personnel for handling.

[0089] Furthermore, in S1, the real-time sub-streams of multiple wide-angle lenses of the AR Eagle Eye camera are simultaneously pulled through the RTSP protocol, and the timestamps are calibrated using the NTP protocol to achieve frame-level synchronization; a timed frame extraction thread is set up to skip ordinary frames and cache the latest keyframe images of each lens according to the lens number; a synchronization timestamp, frame number and pre-calibrated lens distortion parameters are added to the latest keyframe image to form a complete data packet for subsequent stitching and coordinate mapping.

[0090] Specifically, when any RTSP interrupts, the corresponding thread automatically reconnects within a set time (3 seconds), and the previous frame is used to fill in the gap during the interruption to avoid a black screen.

[0091] Furthermore, S2 includes the following steps:

[0092] S2-1 performs color equalization and histogram matching on each keyframe image to eliminate lens vignetting and color difference;

[0093] S2-2, calibrating the internal parameters of each wide-angle lens extrinsic parameter matrix The distortion coefficients are used to remove distortion from the keyframe images of each wide-angle shot obtained through synchronous frame extraction; where R represents the rotation extrinsic parameter matrix between the two shots, and t represents the translation extrinsic parameter matrix between the two shots.

[0094] S2-3: Project the corrected image onto a spherical model, extract feature points using ORB / SURF, calculate the homography matrix using RANSAC, and complete sub-pixel level alignment.

[0095] S2-4 uses bundle adjustment to accurately adjust intrinsic and extrinsic parameters and distortion coefficients to ensure geometric consistency;

[0096] S2-5, construct a multi-band Laplacian pyramid, generate smooth blending weights in overlapping areas, and eliminate seams and brightness differences;

[0097] S2-6, synthesize and output a high-resolution panoramic image, and generate a reverse mapping table between global coordinates and the pixels of each lens, which can be directly called for subsequent detection and coordinate mapping;

[0098] Furthermore, S3 includes the following steps:

[0099] S3-1: Establish a mapping table based on different vehicle size-tiles, and set the corresponding tile size for different vehicle models; write the mapping table into the configuration file so that the corresponding tile can be dynamically selected according to the vehicle model label at runtime.

[0100] Specifically, the vehicle models are divided into microcars, medium-sized cars and large cars, and the corresponding tile sizes are 512×512, 640×640 and 768×768. The table is written into the configuration file and dynamically selected according to the vehicle model label at runtime.

[0101] S3-2, Tile slicing of the high-resolution panoramic image: Determine the grid to which the center point of the vehicle detection box belongs, and slice the image into tiles centered on the corresponding grid. Each tile expands outwards by a preset ratio to ensure that the feature overlap between adjacent tiles is no less than 40%. The vehicle detection box is generated by the target detection model, and the grid center coordinates are the center pixel coordinates of the vehicle detection box.

[0102] Furthermore, in S3-2, when two vehicle models appear in the same grid, the grid tile size is selected according to the principle of "largest vehicle model priority" to avoid cross-vehicle truncation.

[0103] Specifically, each tile expands outward by 20% on all four sides to ensure that the feature overlap between adjacent tiles is ≥40%, thereby reducing the probability of large vehicles being segmented.

[0104] S3-3: Calculate the ratio of the vehicle detection box to the tile side length in each tile image and determine if the ratio exceeds the set value. If it does, determine that the corresponding tile image has a complete target vehicle and execute S3-5. Otherwise, determine that there is a vehicle truncation in the tile image, merge the truncated tile image with the adjacent tiles into a new tile of 1.5 × the original size, and then execute S3-4.

[0105] Specifically, after the vehicle target detection model outputs the vehicle bounding box within the tile, it first determines the target as "complete target" if "bounding length / tile side length" > 0.6, and marks it as "potentially truncated" if ≤ 0.6.

[0106] S3-4: Determine if the tile image has reached the maximum tile size of 1024×1024; if yes, execute S3-5; otherwise, execute S3-3.

[0107] S3-5 records the row and column indices and four-point coordinates of each tile image in the high-resolution panoramic image, which can be directly reused for subsequent model inference and coordinate inverse mapping.

[0108] Furthermore, S4 includes the following steps:

[0109] S4-1, A lightweight object detection model is selected to infer the tile image, the tile image is adjusted to the model input size, and normalization is performed;

[0110] S4-2: Input the preprocessed image into the target detection model. After parallel inference for each tile image, output the coordinates of the 2D detection box of the vehicle. Category labels and confidence levels, among which, These represent the x and y coordinates of the top-left and bottom-right corners of the detection box, respectively.

[0111] S4-3, Calculate the coordinates of the center point of the detection box. ;

[0112] S4-4, based on the vehicle type and the aspect ratio of the vehicle detection frame, keep the direction of the vehicle's front end consistent with the direction of the long side of the vehicle detection frame, and obtain the orientation angle by calculating the angle between the long side of the vehicle's detection frame and the horizontal direction;

[0113] S4-5 integrates vehicle type and vehicle orientation angle information into the detection results to form structured output data; at the same time, output data with a confidence level greater than a set threshold is written to the cache (so that it can be pieced back into the high-resolution panoramic image coordinate system according to the tile image row and column index and the global coordinate mapping table for subsequent NMS merging).

[0114] Furthermore, S5 includes the following steps:

[0115] S5-1 uses the coordinate mapping relationship between the high-resolution panoramic image and each tile image to inversely transform the vehicle detection box coordinates in each tile image to the global coordinate system of the panoramic image through a pre-stored transformation matrix.

[0116] S5-2 uses the Soft-NMS algorithm, sets the cross-union ratio threshold and attenuation parameters, and performs confidence attenuation and deduplication on the duplicate detection boxes across tiles according to the vehicle detection boxes with the highest confidence, to obtain the vehicle retrieval box retention list.

[0117] Specifically, the input detection box set is deduplicated across tiles using the threshold set by Soft-NMS. Each detection box contains coordinates and confidence level Simultaneously, set the intersection-over-union (IoU) threshold between vehicle detection frames. and attenuation parameters Among these factors, the selection of the IoU threshold affects the stringency of the detection results. A lower threshold retains more overlapping boxes, while a higher threshold more strictly filters boxes, requiring the removal of other boxes whose overlap exceeds the threshold. (Based on confidence level...) Sort the detection boxes from highest to lowest confidence level to obtain a sorted index list, and initialize an empty list to store the final retained detection box indices; select the detection box with the highest confidence level from the sorted detection boxes. Add its index to the reserved list; calculate the detection box. The Intersection over Union (IoU) with the remaining detection boxes is used to weight the confidence of the remaining detection boxes based on the IoU value. calculate:

[0118] ;

[0119] Wherein, IoU is the intersection-union ratio of two detection boxes. It is a parameter for adjusting the attenuation level. The smaller the value, the faster the confidence of high IoU detection boxes decreases; the larger the value, the smoother the decay.

[0120] Wherein, IoU is the intersection-union ratio of two detection boxes. It is a parameter for adjusting the attenuation level. The smaller the value, the faster the confidence of high IoU detection boxes decreases; the larger the value, the smoother the decay.

[0121] Multiply the confidence scores of the remaining detection boxes by the corresponding decay weights, and remove detection boxes with confidence scores below a threshold. The threshold is a pre-defined minimum confidence standard used to filter out detection boxes that the vehicle target detection model considers unlikely to be target objects. Repeat the calculation of detection box confidence scores, compare the confidence scores with the threshold, and repeat the filtering and removal of low-confidence detection boxes until all detection boxes have been processed. Output a retention list, which contains the indices of the final retained detection boxes.

[0122] S5-3 integrates the vehicle detection box coordinates, vehicle type, and vehicle orientation angle information from the retained list after inverse mapping to form a structured detection result; finally, it outputs the detection result of a high-resolution panoramic image, including the number of vehicles, vehicle type, and vehicle orientation angle information.

[0123] Furthermore, S6 includes the following steps:

[0124] S6-1, based on high-resolution panoramic images, uses the corrected normalized spherical coordinates as a reference, then backprojects them to pixel coordinates, and uses the polygon ROI tool to calibrate area by area.

[0125] S6-2, Divide the overall vehicle counting area: Divide the service area plaza into an area for counting the overall number of vehicles, covering the entire parking area, refueling area, charging station area and all possible parking areas of vehicles;

[0126] S6-3: On a high-resolution panoramic image or parking area planning map, use the polygon tool to outline the boundaries of parking areas for each vehicle type and bind attributes to each area.

[0127] S6-4, based on vehicle size and usage requirements, divides the parking area into a small car parking area, a large bus parking area, a large truck parking area, and a hazardous chemical transport vehicle parking area;

[0128] S6-5, delineate refueling queue waiting areas near gas stations and mark the start and end points of the refueling queue area, as well as the width and direction of the queue lane, on a high-resolution panoramic image;

[0129] S6-6 marks the location of charging piles and the boundaries of dedicated parking areas for charging piles on a high-resolution panoramic map, while also delineating the vehicle queuing area.

[0130] S6-7, demarcate no-parking zones within the service area plaza. These zones include fire lanes, pedestrian crossings, and plaza entrances / exits. Use a polygon tool on a high-resolution panoramic image to delineate the boundaries of the no-parking zones.

[0131] Furthermore, the event types verified in S7 include at least one of the following: overloaded plaza vehicles when the total number of vehicles exceeds a preset threshold; vehicle type mismatch event when the vehicle type does not match the parking area binding attribute; abnormal posture event when the vehicle's orientation angle exceeds the allowed range of the parking area; no-parking zone occupancy event when the vehicle detection box falls into the no-parking zone; and congestion event when the number of vehicles in the refueling / charging queue area exceeds a threshold or the queuing time exceeds the time limit.

[0132] Specifically, all vehicle detection boxes detected by the high-resolution panoramic image are aggregated, and duplicate counts are eliminated by NMS to obtain the real-time total number of vehicles; when the total number of detected vehicles exceeds a preset threshold, it is marked as a "square vehicle overload" event.

[0133] Based on the vehicle type and permitted parking type rules, bind attributes to each parking space polygon area and write them to the hierarchical configuration file; save the parking area and the coordinates of the four vertices as JSON for direct use in subsequent anomaly detection.

[0134] For each parking area, extract the vehicle detection bounding box within the corresponding area from the panoramic image detection results, verify the following events in real time, and process them accordingly:

[0135] ;

[0136] When the detected vehicle type does not match the parking area, it is marked as a "vehicle type mismatch" event and detailed information of the vehicle type mismatch event is output, including vehicle type, parking area, and panoramic coordinates;

[0137] The attitude anomaly verification covers all parking areas and charging pile areas, and detects whether the vehicle's orientation angle conforms to the prescribed attitude of the parking area; it compares the detected vehicle orientation angle with the allowed attitude angle range of the parking area; when the vehicle orientation angle exceeds the allowed range, it is marked as an "attitude anomaly" event, and the detailed information of the attitude anomaly event is output, including vehicle type, parking area, vehicle orientation angle, and parking area orientation angle.

[0138] Virtually delineate no-parking zones in the panoramic image, including fire lanes, pedestrian crossings, and plaza entrances / exits; extract vehicle bounding boxes within the no-parking zones from the detection results, and compare the 2D bounding box coordinates (x0, y0) of each vehicle. 0, To determine if a vehicle is illegally parked, check if its coordinates (x1, y1) overlap with a no-parking zone. If a vehicle is found to be parked in a no-parking zone, mark it as a "no-parking zone occupancy" event and output detailed information about the event, including vehicle type, no-parking zone, and vehicle's panoramic coordinates.

[0139] A queuing area is defined at the entrance of the gas station. A threshold for the number of vehicles in the waiting area is set, as well as a threshold for the trigger time within the waiting area. A filtering time threshold is also set to avoid repeated alarms during prolonged congestion.

[0140] In the dedicated charging station area, the vehicle orientation is monitored in real time to ensure that the vehicle is parked in a standardized manner and does not encroach on the charging space. A queuing area is marked in the charging station area, and the number of vehicles in the queue is counted in real time. When the number of vehicles in the queue exceeds a preset threshold, it is marked as a "queuing area congestion" event.

[0141] Furthermore, in S8, the initial screening of abnormal events is partially magnified by the AR Eagle Eye camera and then verified through secondary inference using the same model, further improving the accuracy of event detection, including:

[0142] Mapping matrix from offline calibrated panoramic coordinate system to PTZ PTZ camera coordinate system Calculate the required horizontal angle, pitch angle and zoom of the PTZ camera, and drive the PTZ camera to turn to the abnormal area;

[0143] Once the PTZ camera has stabilized, capture keyframes as images of details with anomalies.

[0144] The detailed anomalous image is input into the same object detection model for secondary inference verification. The anomalous image, after being magnified, has richer details, which allows the model to more accurately identify and verify anomalous events.

[0145] Specifically, offline calibration of the panoramic-PTZ camera mapping matrix. After fixing the AR Eagle Eye camera, a laser rangefinder was used to measure the height and corresponding pixel coordinates of four ground control points. Find the 3×3 homography matrix that maps the panoramic spherical coordinate system to the PTZ coordinate system. Save at the same time ,focal length and size factor ; through the calibrated panoramic-PTZ camera mapping matrix Calculate the PTZ camera's horizontal angle, pitch angle, and zoom to ensure that abnormal areas occupy the entire frame. After the PTZ camera turns to the target and stabilizes for 1 second, it captures keyframes from the sub-stream and saves them as detailed anomaly images. These saved detailed anomaly images are then re-input into the same detection model for secondary inference. The magnified, more detailed anomaly images allow the model to more accurately identify and verify anomaly events. Based on the S8 control system, the PTZ camera continuously tracks the anomaly area, maintaining stable monitoring of the target region to ensure complete recording of anomalies. After tracking, keyframes from the sub-stream are captured and saved as detailed anomaly images.

[0146] Furthermore, after confirming that the event is correct in S9, the system first records the detailed information of the abnormal event locally and generates a JSON format log file. The log file includes the event type, trigger time, panoramic coordinates, and vehicle type. At the same time, the system pushes the abnormal event information to the terminal device of the service area duty personnel. After receiving the push, the duty personnel can respond quickly according to the abnormal details and go to the scene to handle the situation.

[0147] The specific principles of this invention will be explained in detail below:

[0148] This invention utilizes an AR eagle-eye panoramic camera deployment comprising multiple wide-angle lenses and an optical zoom PTZ camera. It acquires real-time RTSP streams and extracts frames for screenshotting, then stitches panoramic images from multiple lenses to generate a high-resolution image. A categorized tile slicing strategy is employed to establish a mapping table based on different vehicle size-tile dimensions. The corresponding tile size is dynamically selected according to the vehicle size, and combined with a maximum vehicle size priority, edge expansion, and feature overlap mechanism to ensure vehicle detection integrity. Target truncation is optimized through automatic merging and inference. This invention uses a lightweight object detection model to infer the tile images, obtaining the vehicle's detection bounding box, category label, confidence score, and orientation angle. Based on the Soft-NMS algorithm, it performs cross-tile deduplication and confidence attenuation processing, ultimately mapping the detection results back to the panoramic coordinate system to form a structured panoramic detection output. Based on high-resolution panoramic images, this invention uses a polygon ROI tool to define vehicle counting areas, parking areas divided by vehicle type, refueling queue areas, charging pile dedicated areas, and no-parking areas. Real-time multi-event verification is implemented through panoramic image detection results and functional partitioning. This invention employs a secondary reasoning technique to magnify details of abnormal events, offline calibration of the mapping matrix between panoramic and PTZ cameras, and the use of an AR eagle-eye camera to magnify the abnormal event area and perform secondary reasoning verification to improve detection accuracy. After confirming an abnormal event, the system locally records the abnormal structured data, continuously tracks and records event videos, and pushes them to the service area duty personnel's terminal for timely handling.

[0149] Example 1: A 6-meter-high pole is erected in the plaza outside the service area, with an AR eagle-eye panoramic camera (4-channel wide-angle + 1-channel 45x optical zoom PTZ camera) installed at the top. All video, control, and power are routed to a ground-based server chassis via an armored fiber optic cable and a PoE composite cable. The chassis houses a UPS, a gigabit switch, and a 4G / 5G dual-link router, enabling single-pole, single-line power and network supply. The target detection model is stored in read-only mode on a server with a total hard drive capacity of 4 TB. The server is equipped with two 24 GB GPUs, forming a 48 GB memory pool. 6 GB of the model resides in the GPU memory, responsible for batch tile inference. The specific workflow is as follows:

[0150] Step 1: First, the real-time sub-streams of the four wide-angle lenses of the AR Eagle Eye camera are simultaneously pulled via the RTSP protocol, and under the calibration of the local NTP clock source, the millisecond-level timestamps of all frames are aligned. The frame extraction thread caches the latest frame from each lens every 200 milliseconds, and automatically adds the lens number, frame number, and intrinsic and extrinsic parameters and distortion coefficients obtained from the offline calibration of the lens, so as to provide complete data for subsequent panoramic stitching.

[0151] Step 2: Calibrate the intrinsic parameters of each camera and the relative extrinsic parameters between cameras, perform distortion correction on each frame, and then project all the corrected images onto a spherical model. Use ORB feature point matching to calculate the homography matrix to achieve sub-pixel level alignment. Construct a Laplacian pyramid, generate smooth fusion weights in the overlapping areas, eliminate seams and brightness differences, and synthesize an 8192×4096 panoramic base map. At the same time, output the inverse mapping table of global coordinates to each lens pixel.

[0152] Step 3: Preprocess the high-resolution panoramic image by implementing an adaptive tile slicing strategy according to vehicle type to obtain several tile images of the vehicle target;

[0153] Specifically, a mapping table is established according to different vehicle size-tiles. Vehicles are divided into microcars, mid-size cars, and large cars, with corresponding tiles of 512×512, 640×640, and 768×768. The table is written into the configuration file and dynamically selected according to the vehicle type label at runtime. If two vehicle types appear in the same grid, the grid tile size is increased according to the "largest vehicle type priority" principle to avoid cross-vehicle truncation. Each tile expands outward by 20% on all four sides to ensure that the feature overlap between adjacent tiles is ≥40%, reducing the probability of large-sized vehicles being segmented. The center of the vehicle detection box falls into the grid, that is, the grid is used as the tile center to avoid edge holes. After the model outputs the vehicle frame within the tile, it first determines whether the "frame length / tile side length" is greater than 0.6 and is considered a "complete target". If it is less than or equal to 0.6, it is marked as "potentially truncated". For truncated frames, the tile is automatically merged with the adjacent tiles to form a new tile of 1.5 × the original size. The reasoning is repeated until the frame is complete or the maximum tile size of 1024 × 1024 is reached. The slice also records its row and column index and four-point coordinates in the panoramic image, which can be directly reused for subsequent model reasoning and coordinate inverse mapping.

[0154] Step 4: The tile images are fed into the INT8 lightweight object detection model for parallel inference; the tile images are adjusted to the model input size and normalized; the preprocessed images are input into the object detection model, and the coordinates of the 2D detection box of the vehicle are output after parallel inference for each tile. The system calculates the coordinates of the detection box center point, including its category label and confidence level. Based on the vehicle type and the aspect ratio of the detection box, the vehicle's front direction aligns with the long side of the detection box. The orientation angle is obtained by calculating the angle between the long side and the horizontal direction. Information such as vehicle type and orientation angle is integrated into the detection results to form structured output data. Results with a confidence level greater than a set threshold are written to a cache and then pieced back into the panoramic coordinate system using the tile row and column indices and the global coordinate mapping table for subsequent NMS merging.

[0155] Step 5: Through the coordinate mapping relationship between the panoramic image and each tile image, the coordinates of the vehicle detection box in each tile image are inversely transformed to the global coordinate system of the panoramic image using a pre-stored transformation matrix.

[0156] Specifically, the threshold set by Soft-NMS is used to remove duplicates across tiles. The input detection box set is provided, where each detection box contains coordinates and confidence. At the same time, the intersection-over-union (IoU) threshold and attenuation parameter between vehicle detection boxes are set.

[0157] The detection boxes are sorted from high to low confidence to obtain a sorted index list, and an empty list is initialized to store the indexes of the detection boxes that are ultimately retained.

[0158] Select the highest confidence score from the sorted detection boxes and add its index to the retention list;

[0159] Calculate the IoU between the detection box and the remaining detection boxes, and then decrease the confidence level of the remaining detection boxes based on the IoU value using the following formula:

[0160] ;

[0161] in, The intersection-over-union ratio between vehicle detection frames; For attenuation parameters;

[0162] Multiply the confidence of the remaining detection boxes by the corresponding decay weight, remove the detection boxes with confidence below the threshold, and repeat until all detection boxes have been processed.

[0163] Output a retention list, which contains the indexes of the ultimately retained detection boxes;

[0164] The inverse-mapped vehicle detection box coordinates, vehicle type, vehicle orientation angle, and other information are integrated to form a structured detection result;

[0165] Using the corrected normalized spherical coordinates as a reference, the coordinates are then back-projected to pixel coordinates, and the polygon ROI tool is used to calibrate each region.

[0166] Step 6, Divide the overall vehicle counting area: Divide the service area plaza into areas for counting the overall number of vehicles, covering the entire parking area, refueling area, charging station area and all possible parking areas of vehicles;

[0167] On the panoramic view or parking area planning map, use the polygon tool to outline the boundaries of the parking areas for each vehicle type, and bind attributes to each area;

[0168] Based on vehicle size and usage requirements, the parking area is divided into a small car parking area, a large passenger vehicle parking area, a large truck parking area, and a hazardous chemical transport vehicle parking area.

[0169] A refueling queue waiting area is designated near the gas station, and the starting and ending points of the refueling queue area, as well as the width and direction of the queue lane, are marked on a panoramic view.

[0170] Mark the location of charging stations and the boundaries of dedicated parking areas for charging stations on the panoramic map, and also delineate the vehicle queuing area.

[0171] No-parking zones were demarcated within the service area plaza, including fire lanes, pedestrian crossings, and plaza entrances and exits. The boundaries of the no-parking zones were then outlined using a polygon tool on the panoramic view.

[0172] Step 7: Based on the panoramic image detection results and the functional area zoning of the service area plaza, perform real-time verification of multiple events to complete the initial screening; specifically,

[0173] All vehicle detection boxes detected in the panoramic image are aggregated, and duplicate counts are eliminated by NMS to obtain the real-time total number of vehicles. If the total number of detected vehicles exceeds a preset threshold, it is marked as a "plaza vehicle overload" event. According to the vehicle type-allowed parking type rule, attributes are bound to each parking space polygon area and written into the hierarchical configuration file. The parking area and the coordinates of the four vertices are saved as JSON for direct use in subsequent anomaly detection.

[0174] For each parking area, extract the vehicle detection bounding box within that area from the panoramic image detection results, and verify the following events in real time:

[0175] (1) If the detected vehicle type does not match the parking area, mark it as a "vehicle type mismatch" event and output the event details, including vehicle type, parking area, and panoramic coordinates.

[0176] (2) Attitude anomaly verification covers all parking areas and charging pile areas, and detects whether the vehicle's orientation angle conforms to the prescribed attitude of the parking area. The detected vehicle orientation angle is compared with the allowable attitude angle range of the parking area. If the vehicle orientation angle exceeds the allowable range, it is marked as an "attitude anomaly" event, and the event details are output, including vehicle type, parking area, vehicle orientation angle, and parking area orientation angle.

[0177] (3) Virtually delineate no-parking zones in the panoramic image, including fire lanes, pedestrian crossings, and plaza entrances and exits. Extract vehicle detection boxes from the detection results within the no-parking zones. Determine whether a vehicle is illegally parked by comparing the coordinates of its 2D detection box with the no-parking zone. If a vehicle is found parked within the no-parking zone, mark it as a "no-parking zone occupancy" event and output detailed event information, including vehicle type, no-parking zone, and vehicle panoramic coordinates.

[0178] (4) Divide the queuing area at the entrance of the gas station, set the trigger threshold for the number of vehicles in the waiting area, set the trigger time threshold in the waiting area, and set the filtering time threshold to avoid repeated alarms when there is long-term congestion.

[0179] (5) In the dedicated charging station area, monitor in real time whether the vehicle orientation is consistent with the parking space direction to ensure that the vehicle is parked in a standardized manner and does not encroach on the charging space. Divide the charging station area into a queuing area and count the number of queuing vehicles in real time. When the number of queuing vehicles exceeds a preset threshold, mark it as a "queuing area congestion" event.

[0180] Step 8: For the initial screening of abnormal events, control the AR Eagle Eye camera to zoom in on the abnormal area locally, and use the same lightweight target detection model for secondary inference verification to further improve the accuracy of event detection.

[0181] Specifically, the panoramic-PTZ camera mapping matrix is ​​calibrated offline. After fixing the AR Eagle Eye camera, the height and corresponding pixel coordinates of four ground control points are measured using a laser rangefinder. The 3×3 homography matrix that maps the panoramic spherical coordinate system to the PTZ coordinate system is calculated, while saving the focal length and size factor. By comparing with the calibrated panoramic-PTZ camera mapping matrix, the horizontal angle, pitch angle and zoom of the PTZ camera are calculated to ensure that abnormal areas occupy the frame.

[0182] When an initial screening anomaly is detected, the AR Eagle Eye camera's PTZ camera is turned toward the target and stabilized for 1 second. Then, the key frame of the sub-stream is captured and saved as a detailed anomaly image. The saved detailed anomaly image is re-inputted into the same detection model for secondary inference. The more detailed anomaly image after magnification can enable the model to more accurately identify and verify the anomaly event.

[0183] S9. After confirming that the abnormal event is correct, record the structured data of the abnormal event locally and continuously track and record the event video, and push it to the terminal of the service area duty personnel for handling.

[0184] Specifically, after confirming the event is correct, the system first records the detailed information of the abnormal event locally, generating a JSON-formatted log file containing the event type, trigger time, panoramic coordinates, and vehicle type. The system then controls the PTZ camera to continuously track the abnormal area, maintaining stable monitoring of the target area to ensure the abnormal situation is fully recorded. After tracking is complete, keyframes in the sub-stream are captured and saved as a detailed anomaly image. Simultaneously, the system pushes the abnormal event information to the terminal devices of the service area staff. Upon receiving the push, the staff can quickly respond based on the anomaly details and proceed to the site for handling.

[0185] The present invention, employing the above technical solutions, has the following advantages and beneficial effects: 1) Low-cost installation and deployment: A single AR Eagle Eye panoramic camera can achieve vehicle monitoring and assessment of the service area's vehicle carrying capacity, significantly reducing construction costs and subsequent maintenance expenses. 2) Vehicle-type tile slicing strategy improves panoramic image detection accuracy: A mapping table is established based on different vehicle types and sizes, dynamically selecting corresponding tile sizes. Combined with a maximum vehicle type priority, edge expansion, and feature overlap mechanism, the integrity of vehicle detection in the panoramic image is ensured, effectively improving the accuracy of vehicle detection in the panoramic image. 3) Single-device panoramic image area hierarchical calibration: A single AR Eagle Eye panoramic camera achieves full coverage of the service area plaza and accurately calibrates different functional areas. This not only significantly reduces equipment procurement and maintenance costs but also solves the problem of multi-device data fusion, improving data consistency and real-time performance. Simultaneously, it simplifies the system architecture and enhances the convenience of service area management and the efficiency of event monitoring. 4) The "panoramic initial screening + detailed anomaly magnification" architecture improves the accuracy of abnormal event detection: Panoramic images are used to initially screen events and quickly locate abnormal event areas. The AR Eagle Eye camera is used to magnify the abnormal areas locally, and the same model is used for secondary inference verification. This architecture combines the efficient screening capability of panoramic images with the accurate recognition capability of detailed images, which can effectively improve the accuracy and reliability of abnormal event detection in service area squares.

[0186] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A low-cost service area vehicle monitoring method based on AR eagle-eye panoramic camera, characterized in that: It includes the following steps: S1. Obtain the real-time video stream from the AR Eagle Eye camera in the service area, and extract keyframe images from multiple wide-angle lenses periodically. S2. Combine keyframe images from multiple wide-angle lenses to create a high-resolution panoramic image. S3. Preprocess the high-resolution panoramic image by implementing an adaptive tile slicing strategy according to vehicle type to obtain several tile images of the vehicle target; S4. Use a lightweight object detection model to infer the current vehicle detection result for each tile image. The current vehicle detection result includes vehicle type, confidence level and orientation angle. S5. Perform coordinate inverse mapping and cross-tile deduplication on the vehicle detection results of each tile image, and output the final panoramic image detection result; the panoramic image detection result includes the number of vehicles, vehicle type, and vehicle orientation angle; S6. Based on high-resolution panoramic images, the service area plaza is divided into functional zones, including vehicle counting areas, parking areas divided by vehicle type, refueling queue areas, dedicated charging pile areas, and no-parking areas. S7. Based on the panoramic image detection results and the functional area zoning of the service area square, implement real-time verification of multiple events to complete the initial screening; S8. For the initial screening of abnormal events, control the AR Eagle Eye camera to zoom in on the abnormal area locally, and use the same lightweight target detection model to perform secondary inference to verify the abnormal events. S9. After confirming that the abnormal event is correct, record the structured data of the abnormal event locally and continuously track and record the event video, and push it to the terminal of the service area duty personnel for handling.

2. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: In S1, the real-time sub-streams of multiple wide-angle lenses of the AR Eagle Eye camera are simultaneously pulled through the RTSP protocol, and the timestamps are calibrated using the NTP protocol to achieve frame-level synchronization; a timed frame extraction thread is set up to skip ordinary frames and cache the latest keyframe images of each lens according to the lens number. The latest keyframe image is appended with a synchronization timestamp, frame number, and pre-calibrated lens distortion parameters to form a complete data packet for subsequent stitching and coordinate mapping.

3. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: S2 includes the following steps: S2-1 performs color equalization and histogram matching on each keyframe image to eliminate lens vignetting and color difference; S2-2, calibrating the internal parameters of each wide-angle lens extrinsic parameter matrix The distortion coefficients are used to remove distortion from the keyframe images of each wide-angle lens obtained by synchronous frame extraction; where R represents the rotation extrinsic parameter matrix between the two lenses, and t represents the translation extrinsic parameter matrix between the two lenses. S2-3: Project the corrected image onto a spherical model, extract feature points using ORB / SURF, calculate the homography matrix using RANSAC, and complete sub-pixel level alignment. S2-4 uses bundle adjustment to accurately adjust intrinsic and extrinsic parameters and distortion coefficients to ensure geometric consistency; S2-5, construct a multi-band Laplacian pyramid, generate smooth blending weights in overlapping areas, and eliminate seams and brightness differences; S2-6 synthesizes and outputs a high-resolution panoramic image, and generates a reverse mapping table between global coordinates and the pixels of each lens, which can be directly called for subsequent detection and coordinate mapping.

4. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: S3 includes the following steps: S3-1: Establish a mapping table based on different vehicle size-tiles, and set the corresponding tile size for different vehicle models; write the mapping table into the configuration file so that the corresponding tile can be dynamically selected according to the vehicle model label at runtime. S3-2, Tile slicing of high-resolution panoramic image: Determine the grid to which the center point of the vehicle detection box belongs, and slice the image into tiles with the corresponding grid as the center. Each tile is expanded outwards by a preset ratio to ensure that the feature overlap between adjacent tiles is not less than 40%. S3-3: Calculate the ratio of the vehicle detection box to the tile side length in each tile image and determine if the ratio exceeds the set value. If it does, determine that the corresponding tile image has a complete target vehicle and execute S3-5. Otherwise, determine that there is a vehicle truncation in the tile image, merge the truncated tile image with the adjacent tiles into a new tile of 1.5 × the original size, and then execute S3-4. S3-4: Determine if the tile image has reached the maximum tile size of 1024×1024; if yes, execute S3-5; otherwise, execute S3-3. S3-5 records the row and column indices and four-point coordinates of each tile image in the high-resolution panoramic image, which can be directly reused for subsequent model inference and coordinate inverse mapping.

5. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: S4 includes the following steps: S4-1, a lightweight object detection model is selected to infer the tile image, the tile image is adjusted to the model input size, and normalization is performed; S4-2: Input the preprocessed image into the target detection model. After parallel inference for each tile image, output the coordinates of the 2D detection box of the vehicle. Category labels and confidence levels; among which, ( ), ( ) represent the coordinates of the top left and bottom right corners of the detection box, respectively; S4-3, Calculate the coordinates of the center point of the detection box. ; S4-4, based on the vehicle type and the aspect ratio of the vehicle detection frame, keep the direction of the vehicle's front end consistent with the direction of the long side of the vehicle detection frame, and obtain the orientation angle by calculating the angle between the long side of the vehicle's detection frame and the horizontal direction; S4-5 integrates vehicle type and vehicle orientation angle information into the detection results to form structured output data; at the same time, output data with a confidence level greater than a set threshold is written into the cache as the detection results.

6. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: S5 includes the following steps: S5-1 uses the coordinate mapping relationship between the high-resolution panoramic image and each tile image to inversely transform the vehicle detection box coordinates in each tile image to the global coordinate system of the panoramic image through a pre-stored transformation matrix. S5-2 uses the Soft-NMS algorithm, sets the cross-union ratio threshold and attenuation parameters, and performs confidence attenuation and deduplication on the duplicate detection boxes across tiles according to the vehicle detection boxes with the highest confidence, to obtain the vehicle retrieval box retention list. S5-3 integrates the vehicle detection box coordinates, vehicle type, and vehicle orientation angle information from the retained list after inverse mapping to form a structured detection result; finally, it outputs the detection result of a high-resolution panoramic image, including the number of vehicles, vehicle type, and vehicle orientation angle information.

7. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: S6 includes the following steps: S6-1, based on high-resolution panoramic images, uses the corrected normalized spherical coordinates as a reference, then back-projects them to pixel coordinates, and uses the polygon ROI tool to calibrate area by area; S6-2, within the service area plaza, an area is designated for counting the total number of vehicles, covering the entire parking area, refueling area, charging station area, and all vehicle parking areas; S6-3: On a high-resolution panoramic image or parking area planning map, use the polygon tool to outline the boundaries of parking areas for each vehicle type and bind attributes to each area. S6-4, based on vehicle size and usage requirements, divides the parking area into a small car parking area, a large bus parking area, a large truck parking area, and a hazardous chemical transport vehicle parking area; S6-5, delineate refueling queue waiting areas near gas stations and mark the start and end points of the refueling queue area, as well as the width and direction of the queue lane, on a high-resolution panoramic image; S6-6 marks the location of charging piles and the boundaries of dedicated parking areas for charging piles on a high-resolution panoramic map, while also delineating the vehicle queuing area. S6-7, demarcate no-parking zones within the service area plaza. These zones include fire lanes, pedestrian crossings, and plaza entrances / exits. Use a polygon tool on a high-resolution panoramic image to delineate the boundaries of the no-parking zones.

8. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: The event types verified in S7 include at least one of the following: overloaded square vehicle event when the total number of vehicles exceeds a preset threshold; Vehicle type does not match the parking area's binding attribute - mismatch event; Abnormal vehicle orientation events (vehicle facing angle exceeding the permitted range of the parking area); no-parking zone occupancy events (vehicle detection frame falling into a no-parking zone); congestion events (number of vehicles in the refueling / charging queue area exceeding the threshold or queuing time exceeding the time limit).

9. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: S8 includes: Mapping matrix from offline calibrated panoramic coordinate system to PTZ PTZ camera coordinate system Calculate the required horizontal angle, pitch angle and zoom of the PTZ camera, and drive the PTZ camera to turn to the abnormal area; Once the PTZ camera has stabilized, capture keyframes as images of details with anomalies. The detailed abnormal images are input into the same target detection model for secondary inference verification, and the abnormal events are identified and verified by magnifying the details of the abnormal images.

10. The low-cost service area vehicle monitoring method based on an AR eagle-eye panoramic camera according to claim 1, characterized in that: After confirming that the event is correct in S9, the system first records the detailed information of the abnormal event locally and generates a JSON log file. The log file includes the event type, trigger time, panoramic coordinates, and vehicle type. At the same time, the system pushes the abnormal event information to the terminal device of the service area duty personnel. After receiving the push, the duty personnel respond quickly according to the abnormal details and go to the scene to handle the situation.

Images