Intelligent monitoring and warning system and method for parking process of airplane parking apron
By using computer vision technology during the berth of the aircraft apron, an intelligent monitoring and alarm system was designed, which solved the problem that the existing technology could not realize the automatic identification and collision avoidance of key nodes, and achieved the goal of aircraft and other targets. Fine positioning and real-time alarms improve the efficiency and safety of airport operations.
Patent Information
- Application Number
- CN202410807629.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-12
AI Technical Summary
The existing scene monitoring methods cannot achieve automated identification and collision avoidance of key nodes in the berth of the aircraft, especially due to the large size, irregular shape and complex movement state of the aircraft, making it difficult to accurately locate the easily impacted parts.
An intelligent monitoring and alarm system for berthing process based on computer vision was designed, including a scene target object detection module, a target association matching module, a key time node identification and acquisition module, a target object refinement positioning module and an alarm module, using The improved YOLOv7 target detector, Markov model, DG-STA model, SQPnP algorithm and PyQt interface realizes detection, trajectory formation, location refinement positioning and real-time alarms of target objects such as aircraft and passenger elevators.
It realizes refined positioning and real-time alarms of target objects in the airport scene, improves the efficiency and safety of airport operations, can timely identify and warn of potential collision risks, and improves the automated collection capabilities of apron operation.
Smart Images

Figure CN118942050A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image and video processing, and in particular to an intelligent monitoring and warning system for aircraft parking apron berthing process based on computer vision. Background Art
[0002] The "Action Outline for the Construction of Type 4 Civil Aviation Airports in China (2020-2035)" issued by the Civil Aviation Administration of China in 2020 clearly pointed out that full use should be made of technical means to improve the apron operation control capabilities, and breakthroughs should be made in the fields of intelligent video analysis and automatic collection of node time data, and gradually promoted to the entire industry. However, the existing scene monitoring methods are unable to complete the automatic collection of key nodes and avoid aircraft collision problems. Aircraft are large in size, irregular in shape, and have complex and changeable motion states. Existing monitoring methods such as scene surveillance radars and ADS-B cannot accurately locate the parts of the aircraft that are prone to collision. Therefore, there is an urgent need for an intelligent monitoring and warning system that can realize the automatic identification of key nodes and collision avoidance in the process of aircraft parking on the apron. Summary of the invention
[0003] The present invention provides an airport collaborative decision-making system based on field monitoring videos, aiming at improving the efficiency and safety of airport field operations.
[0004] An intelligent monitoring and warning system for aircraft parking apron parking process, comprising:
[0005] (1) Field object detection module, used to detect field objects such as aircraft, passenger elevators, refueling trucks, tractors, and catering trucks, and output the detection results;
[0006] (2) Target association matching module, which uses the Markov model to perform target association matching on the detection results to form the motion trajectory of the scene target;
[0007] (3) Critical time node identification and collection module, which uses the DG-STA model to identify and collect critical time nodes and output the categories of critical time nodes and the time intervals in which they occur;
[0008] (4) The target object fine positioning module performs multi-category key point detection based on the target detection results, uses the SQPnP algorithm for posture estimation, and outputs the three-dimensional spatial position of the target object;
[0009] (5) Alarm module, used to monitor the aircraft bridge connection and airport surface alarm area in real time and perform alarm operations;
[0010] (6) Data processing and display module, used to summarize the output results of the above modules and display and interact in real time on a graphical user interface.
[0011] Furthermore, the scene object detection module uses the improved YOLOv7 target detector, which improves detection accuracy and real-time performance by adding the BiFormer attention mechanism and replacing the confidence loss with FocalLoss.
[0012] Furthermore, the target association matching module adopts the BYTE algorithm, and through two association matching processes, the high-score detection frame and the low-score detection frame are matched with the prediction frame respectively to form the motion trajectory of the scene target.
[0013] Furthermore, the key time node identification and collection module uses the DG-STA model to extract the spatiotemporal motion characteristics of scene targets, and combines the sliding time window method to identify and collect key time nodes.
[0014] Furthermore, the target object fine-grained positioning module performs multi-category key point detection through the target detector and HRNet, and uses the SQPnP algorithm for posture estimation to output the three-dimensional spatial position of the target object.
[0015] Further, the alarm module includes:
[0016] (1) The aircraft bridge connection judgment and warning submodule is used to identify the position of the aircraft and the bridge, mark key parts, monitor whether the bridge position deviates from the movable area or there is a risk of collision, and determine whether the bridge returns to its original position after the aircraft leaves the airport and issue an alarm;
[0017] (2) The engine inhalation warning zone is divided into a submodule, which calculates the size of the engine safety zone based on the aircraft speed information, forms a circular safety zone, and issues an alarm when personnel or equipment approach;
[0018] (3) A submodule for passenger activity warning zoning is set up to detect whether passengers enter restricted areas by identifying the activity areas of passengers and staff, and to issue warnings;
[0019] (4) The submodule for verifying the number of personnel, vehicles, and reflective cones on the scene verifies the placement of reflective cones by detecting the number of personnel and equipment in the safety zone and issuing an alarm if an abnormality is found.
[0020] Furthermore, the data processing and display module is developed based on the PyQt interface for real-time display and interaction, providing a user-friendly operation interface.
[0021] Further, the following steps are included:
[0022] (1) Use the improved YOLOv7 target detector to detect scene targets;
[0023] (2) Use the Markov model to match the detection results to form a motion trajectory; (3) Use the DG-STA model to identify and collect key time nodes;
[0024] (4) Perform multi-category key point detection and use the SQPnP algorithm for posture estimation to output the three-dimensional spatial position;
[0025] (5) Real-time monitoring of aircraft bridge connections and airport surface warning areas, and execution of warning operations;
[0026] (6) Summarize the output results of the above steps and display and interact in real time on the graphical user interface.
[0027] Furthermore, the target association matching step specifically includes:
[0028] (1) Initialize trajectory;
[0029] (2) Traverse each frame of the video sequence and use the Markov model to estimate the new position of the target in the current frame;
[0030] (3) Through two association matching processes, the high-score detection frame and the low-score detection frame are matched with the prediction frame respectively to form the motion trajectory of the scene target.
[0031] Furthermore, the key time node identification and collection steps specifically include:
[0032] (1) Send the continuous detection boxes and their corresponding category information into the DG-STA model;
[0033] (2) Use spatiotemporal masking operations to reduce computational costs and classify key time nodes that appear in the video;
[0034] The steps of fine positioning of the target object specifically include:
[0035] (1) Use the object detector and HRNet to perform multi-category key point detection and obtain a high-resolution key point heat map;
[0036] (2) Using the SQPnP algorithm, the reprojection error is minimized through a nonlinear least squares problem to obtain the camera pose transformation matrix;
[0037] The alarm steps include:
[0038] (1) Use the three viewing angles of the processed video to identify the position of the aircraft and the bridge, mark the key parts, monitor whether the bridge position deviates from the movable area or there is a risk of collision, and determine whether the bridge returns to its original position after the aircraft leaves the airport, and issue an alarm;
[0039] (2) Calculate the size of the engine safety zone based on the aircraft speed information to form a circular safety zone and issue an alarm when personnel or equipment approach;
[0040] (3) By identifying the activity areas of passengers and staff, monitoring whether passengers enter restricted areas and issuing alarms;
[0041] (4) Detect the number of people and equipment in the safety zone, verify the placement of reflective cones, and issue an alarm if any abnormality is found.
[0042] The system acquires key time node information and key location information of targets by collecting and analyzing scene surveillance videos, thereby realizing refined positioning and early warning of various targets at the airport scene. The system consists of two main modules: a key time node acquisition module based on scene surveillance videos and a scene target refined positioning and warning module. Key time node information and target location information will be reported to the airport collaborative decision-making system in a timely manner to help it complete the corresponding implementation plan. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is the overall structural diagram of the system of the present invention.
[0044] Figure 2 It is a workflow diagram of the key time node acquisition module based on scene surveillance video.
[0045] Figure 3 It is the workflow diagram of the scene target refined positioning and alarm module.
[0046] Figure 4 It is a system block diagram of the present invention. DETAILED DESCRIPTION
[0047] 1. Overall idea of the system
[0048] The system of the present invention mainly comprises two modules: a key time node acquisition module based on scene monitoring video and a scene target object refined positioning and alarm module.
[0049] 2. Key time node acquisition module based on scene surveillance video
[0050] (1) Target detection and association:
[0051] The YOLOv7 target detector is used, combined with the efficient layer aggregation network (ELAN) and re-parameterized convolution (RepConv), to improve real-time performance while ensuring detection accuracy. The BYTE association algorithm is used to associate the target detection box with the prediction box, and the next frame position of the target is estimated through the Markov model to solve the problem that low-confidence detection boxes cannot be paired due to occlusion, motion blur, and scale changes.
[0052] (2) Identification and collection of key time nodes:
[0053] The dynamic graph spatiotemporal attention model (DG-STA) is combined with the sliding time window method to extract the spatiotemporal motion characteristics of scene targets in video frames and realize the classification and collection of key time nodes.
[0054] 3. Scene target refined positioning and warning module
[0055] (1) Target precise positioning:
[0056] The target detection algorithm is used to identify multi-category targets in the scene, and the HRNet key point detection network is used to extract the coordinates of key points in the two-dimensional image. The SQPnP posture estimation algorithm is used to obtain the camera posture transformation matrix, unify the point cloud coordinate system, and output the real-time positioning distance between the key points of the target.
[0057] (2) Extraction of key objects and posture judgment in images:
[0058] Image information datasets are collected from three sources: airport field monitoring equipment, operation and maintenance vehicles, and drones. This multi-source image information dataset includes aircraft, ground support vehicles (such as passenger elevators, tractors, refueling trucks, and dining trucks), and motion behavior image data of ground staff that may appear on the field. Using the trained target detection model, the category information and position information of each target object in each frame of video image, as well as the dangerous motion trend and dangerous behavior judgment report are obtained. According to the position information of each target object in each frame of video image and the size relationship between the plane of the frame of video image, the behavior subject of the target position in each frame of video image is judged. The Markov chain is used to judge the spatiotemporal information of the behavior subject extracted in each frame of video image to obtain the prediction result of the current frame.
[0059] (3) Combining the Markov model with the KM algorithm:
[0060] Using the Markov model, the prediction result of the current frame is obtained based on the behavior subject information extracted from each frame of the video image. The prediction result of the current frame is matched with the target detection result of the current frame, and the matching result is input into the KM (Kuhn-Munkres) algorithm to obtain the association result, which is used to represent the association relationship between the prediction result of the current frame and the target detection result. The airport scene image to be identified is obtained, and the motion behavior and the key nodes corresponding to the behavior in the airport scene image to be identified are determined based on the association result.
[0061] (4) Alarm system:
[0062] Aircraft bridge connection judgment and warning:
[0063] The three perspectives of the processed video are used to identify the position of the aircraft and the bridge, and the key parts where the aircraft is prone to collide with the bridge are marked. The drone and the maintenance vehicle solve the monitoring blind spot area, and the movable area of the bridge is constructed through the ranging function of the identification system. The position of the bridge is identified in real time in the movable area of the bridge, and the risk of deviation from the movable area and collision is monitored, otherwise an alarm is issued. After the aircraft leaves the port, it is determined whether the bridge position has returned to the original position, otherwise an alarm is issued.
[0064] Airport surface warning area division:
[0065] The size of the engine safety zone is calculated based on the aircraft speed information. The radius of the intake danger zone is preset with the engine air intake as the center, forming a circular safety zone that extends to the rear of the engine air intake cover lip. If a person or equipment approaches the warning zone, an alarm will be issued.
[0066] Define the restricted area for passengers, use work clothes to identify passengers in the scene, and synthesize the passenger activity area through the position of the passenger elevator and the shuttle bus. If a passenger moves into the restricted area, an alarm will be issued.
[0067] Determine whether there are other people or equipment intruding by the number of people and equipment in the safety zone. If found, issue an alarm. Determine whether reflective cones are completely placed at key points such as the nose, wheels, engine, wing corners, and tail. Issue an alarm for areas where reflective cones are not placed within a period of time when the aircraft is parked according to the relevant airport regulations.
[0068] 4. Spatial-temporal information analysis
[0069] Based on the Markov model, the temporal and spatial information of the subject in each frame of the video image is determined. Assume that the random variable X0 at time 0 follows the probability distribution π0, X t Depends only on X t-1 , and does not depend on the past state. This property is called Markov property. When predicting the direction of the scene target, it can be considered to satisfy this property. Using the transition probability matrix P, calculate the transition probability P of the Markov chain in state j at time t-1 and moving to state i at time t ij When a pixel signal in the image is brought in, the matrix of the pixel set converted into a vector set is used as the initial state distribution to form the transition probability matrix of the Markov chain.
[0070] 5. Correlation and Early Warning
[0071] The current frame prediction result of the Markov model is matched with the target detection result, and the KM (Kuhn-Munkres) algorithm is input to obtain the association result. The KM algorithm is a classic bipartite graph matching method, which is used to solve the optimal matching problem of the association matrix L. According to the association result, it is judged whether there is a collision risk. If there is, the alarm mechanism is triggered to inform the airport collaborative decision-making system in time.
Claims
1. An intelligent monitoring and warning system for aircraft parking apron parking process, comprising: (1) Field object detection module, used to detect field objects such as aircraft, passenger elevators, refueling trucks, tractors, and catering trucks, and output the detection results; (2) Target association matching module, which uses the Markov model to perform target association matching on the detection results to form the motion trajectory of the scene target; (3) Critical time node identification and collection module, which uses the DG-STA model to identify and collect critical time nodes and output the categories of critical time nodes and the time intervals in which they occur; (4) The target object fine positioning module performs multi-category key point detection based on the target detection results, uses the SQPnP algorithm for posture estimation, and outputs the three-dimensional spatial position of the target object; (5) Alarm module, used to monitor the aircraft bridge connection and airport surface alarm area in real time and perform alarm operations; (6) Data processing and display module, used to summarize the output results of the above modules and display and interact in real time on a graphical user interface.
2. According to the intelligent monitoring and warning system for aircraft apron parking process in claim 1, the scene target detection module uses an improved YOLOv7 target detector, and improves the detection accuracy and real-time performance by adding a BiFormer attention mechanism and replacing the confidence loss with FocalLoss.
3. According to the intelligent monitoring and warning system for aircraft apron parking process as described in claim 1, the target association matching module adopts BYTE algorithm, and through two association matching processes, the high-score detection frame and the low-score detection frame are matched with the prediction frame respectively to form the motion trajectory of the scene target.
4. According to the intelligent monitoring and warning system for aircraft apron parking process in claim 1, the key time node identification and collection module uses the DG-STA model to extract the spatiotemporal motion characteristics of scene targets, and combines the sliding time window method to identify and collect key time nodes.
5. According to the intelligent monitoring and warning system for aircraft apron parking process in claim 1, the target object fine positioning module performs multi-category key point detection through target detector and HRNet, and uses SQPnP algorithm for posture estimation to output the three-dimensional spatial position of the target object.
6. The intelligent monitoring and warning system for aircraft parking apron parking process according to claim 1, wherein the warning module comprises: (1) The aircraft bridge connection judgment and warning submodule is used to identify the position of the aircraft and the bridge, mark key parts, monitor whether the bridge position deviates from the movable area or there is a risk of collision, and determine whether the bridge returns to its original position after the aircraft leaves the airport and issue an alarm; (2) The engine inhalation warning zone is divided into a submodule, which calculates the size of the engine safety zone based on the aircraft speed information, forms a circular safety zone, and issues an alarm when personnel or equipment approach; (3) A submodule for passenger activity warning zoning is set up to detect whether passengers enter restricted areas by identifying the activity areas of passengers and staff, and to issue warnings; (4) The submodule for verifying the number of personnel, vehicles, and reflective cones on the scene verifies the placement of reflective cones by detecting the number of personnel and equipment in the safety zone and issuing an alarm if an abnormality is found.
7. According to claim 1, an intelligent monitoring and warning system for aircraft apron parking process, wherein the data processing and display module is developed based on the PyQt interface, used for real-time display and interaction, and provides a user-friendly operation interface.
8. An intelligent monitoring and warning method for an aircraft parking apron parking process, comprising the following steps: (1) Use the improved YOLOv7 target detector to detect scene targets; (2) Use the Markov model to match the detection results to form a motion trajectory; (3) Use the DG-STA model to identify and collect key time nodes; (4) Perform multi-category key point detection and use the SQPnP algorithm for posture estimation to output the three-dimensional spatial position; (5) Real-time monitoring of aircraft bridge connections and airport surface warning areas, and execution of warning operations; (6) Summarize the output results of the above steps and display and interact in real time on the graphical user interface.
9. The intelligent monitoring and warning method for aircraft parking apron parking process according to claim 8, wherein the target association matching step specifically comprises: (1) Initialize trajectory; (2) Traverse each frame of the video sequence and use the Markov model to estimate the new position of the target in the current frame; (3) Through two association matching processes, the high-score detection frame and the low-score detection frame are matched with the prediction frame respectively to form the motion trajectory of the scene target.
10. The intelligent monitoring and warning method for aircraft parking apron parking process according to claim 8, wherein the key time node identification and collection steps specifically include: (1) Send the continuous detection boxes and their corresponding category information into the DG-STA model; (2) Use spatiotemporal masking operations to reduce computational costs and classify key time nodes that appear in the video; The steps of fine positioning of the target object specifically include: (1) Use the object detector and HRNet to perform multi-category key point detection and obtain a high-resolution key point heat map; (2) Using the SQPnP algorithm, the reprojection error is minimized through a nonlinear least squares problem to obtain the camera pose transformation matrix; The alarm steps include: (1) Use the three viewing angles of the processed video to identify the position of the aircraft and the bridge, mark the key parts, monitor whether the bridge position deviates from the movable area or there is a risk of collision, and determine whether the bridge returns to its original position after the aircraft leaves the airport, and issue an alarm; (2) Calculate the size of the engine safety zone based on the aircraft speed information to form a circular safety zone and issue an alarm when personnel or equipment approach; (3) By identifying the activity areas of passengers and staff, monitoring whether passengers enter restricted areas and issuing alarms; (4) Detect the number of people and equipment in the safety zone, verify the placement of reflective cones, and issue an alarm if any abnormality is found.
Citation Information
Patent Citations
Gallery bridge working state detection system and method based on image processing technology
CN110210427A
Method for monitoring safe operation of airport apron through video analysis and deep learning
CN110610592A
Video-based airplane bridging event detection method and device
CN112528729A
Automatic identification method and system for key time nodes of airport scene
CN115797822A
Airplane in-and-out-of-position state identification method
CN115880646A
Cited By
ADS-B track and video fusion scene aircraft detection method and device
CN119723958A
Automatic parking system for aircraft
CN120802733A
Airport apron video image intelligent analysis and event automatic detection method
CN122135270A
An intelligent monitoring and alarming system and method for an aircraft apron
CN122906915A
