An intelligent monitoring method and system for aircraft take-off and landing process

Abstract

The application discloses an intelligent monitoring method and system for airplane taking-off and landing process, and the method comprises the following steps: acquiring a video stream in the direction of airplane taking-off and landing, and identifying the airplane in the video stream; continuously tracking the target airplane when the airplane is identified; identifying the number of airplane landing gears in the process of continuously tracking the target airplane, and giving an abnormal alarm when the number of landing gears is abnormal; and monitoring the flight track of the airplane in the process of continuously tracking to determine whether the airplane deviates from the runway, and giving a runway deviation alarm when the airplane deviates from the runway. The application has the advantages of improving the safety and efficiency of airplane taking-off and landing.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of aircraft take-off and landing monitoring, and in particular to an intelligent monitoring method and system for aircraft take-off and landing processes. Background Art

[0002] In recent years, advanced science and technology have driven the rapid development of civil aviation. Rapid economic growth has significantly improved people's living standards, leading to an increasing number of people choosing to travel by air. Aircraft safety is a strong safeguard for the safety of life and property. Takeoff and landing are the most dangerous stages of the entire flight process and are also the most prone to accidents. Therefore, monitoring aircraft takeoff and landing and keeping track of their flight status are crucial for preventing accidents.

[0003] Aircraft takeoff and landing monitoring technology has made significant progress in the civil aviation field. High-resolution cameras and thermal imaging equipment are widely used in airport monitoring systems. These devices can operate stably under various lighting and weather conditions, providing clear visual information, and are key components of airport monitoring systems.

[0004] Currently, aircraft takeoff and landing processes are typically monitored through manual observation or video surveillance. Tower personnel use telescopes, camera footage of the aircraft, or other sensors such as radar to monitor the aircraft's flight status. Through their own observations and analysis, they determine whether the aircraft is at risk of deviating from its flight path, whether the altitude and speed are appropriate, and whether the landing gear is in place. This manual judgment method relies heavily on human experience, resulting in ineffective and inaccurate judgments and a low level of intelligence for the entire system.

[0005] Traditional monitoring systems often only monitor aircraft from fixed angles, limiting their ability to comprehensively monitor the landing gear's status at various aircraft attitudes and reducing the accuracy and timeliness of monitoring. This fixed viewing angle particularly limits the system's ability to detect abnormal landing gear conditions during critical takeoff and landing. Even within visible range, the image resolution of fixed cameras may not be sufficient to capture subtle features of the landing gear in detail.

[0006] Existing monitoring methods often use single video monitoring when monitoring aircraft runway alignment. Due to the limitations of video sensors themselves, they cannot provide accurate aircraft position information, which increases the safety risks of aircraft takeoff and landing.

[0007] Patent application CN103287584B proposes an aircraft video landing auxiliary system, which is deployed with a fixed camera at multiple different positions on the center line of the runway, and real-time image collection is performed on the landing aircraft. Image information is transmitted to the aircraft through a wireless communication system and is displayed on different displays on the aircraft at the same time. Auxiliary lines are drawn on the display to facilitate the pilot to observe the position in time and find the landing direction and landing point, which can improve the accuracy and stability of landing. This method can only give a qualitative result and cannot accurately calculate the deviation angle and deviation distance. At the same time, this method depends on wireless communication technology, and the video data volume is large, which requires high stability of communication, otherwise it will cause problems such as large data delay and data loss, so that the pilot cannot make correct judgments in time.

[0008] Patent application CN106295695B proposes an aircraft take-off and landing process automatic tracking and shooting method and device, which realizes automatic tracking of the whole process of aircraft take-off and landing through a pan-tilt camera deployed on one side of the tower, and makes the aircraft always in the center position of the image frame with a zoom camera. This method assumes that the aircraft is always located on the vertical plane of the runway center line during take-off and landing, so it cannot detect and warn the aircraft's deviation from the runway.

[0009] In summary, the existing aircraft take-off and landing monitoring system often only monitors the flight height, flight speed, flight direction and other parameters of the aircraft to determine whether the aircraft can land safely, but does not effectively monitor the state of the aircraft landing gear. During the aircraft landing process, the timely deployment of the landing gear directly affects the safety of the aircraft after landing. The existing monitoring system often relies on a single perspective and cannot fully capture the key information of the aircraft in different attitudes. The existing technology has limitations in real-time analysis of runway alignment, which usually determines whether the aircraft deviates from the runway by the position of the aircraft, without measuring the distance of the aircraft, so the three-dimensional position information of the aircraft in the air cannot be obtained. SUMMARY

[0010] In view of the technical problems existing in the prior art, the present application provides an intelligent monitoring method and system for improving the safety of the aircraft take-off and landing process.

[0011] To solve the above technical problems, the technical solution provided by the present application is:

[0012] An intelligent monitoring method for the aircraft take-off and landing process, comprising the steps of:

[0013] Obtaining a video stream in the aircraft take-off direction and identifying the aircraft in the video stream; when the aircraft is identified, continuously tracking the target aircraft;

[0014] In the process of continuously tracking the target aircraft, the number of aircraft landing gears is identified, and when the number of landing gears is abnormal, an abnormality alarm is given;

[0015] and in the process of continuous tracking, the flight trajectory of the aircraft is monitored to determine whether it deviates from the runway, and an off-runway alarm is given when it deviates from the runway.

[0016] Preferably, the specific process of identifying the aircraft in the video stream is:

[0017] First stage: preliminary detection and region of interest extraction

[0018] Global detection is performed on the entire picture in the video stream, the aircraft in the image is identified and located, and preliminary identification of the landing gear is performed to obtain a detection result, and then the detection result is judged; if the detection result includes an aircraft, the number of landing gears in the detection result is calculated; if the number of landing gears meets the predetermined standard, the detection is ended and no subsequent second stage detection is performed; otherwise, a region of interest larger than the region where the aircraft is located is extracted with the region where the aircraft is located as the center, for the second stage detection;

[0019] Second stage: secondary refined detection and fusion

[0020] The image of the region of interest extracted in the first stage is sent to a preset landing gear identification model for further identification of the landing gear to obtain a detection result; after obtaining the detection result of the second stage, the landing gear detection result of the first stage is fused with the landing gear detection result of the second stage;

[0021] Third stage: detection result output and alarm

[0022] The fusion result of the previous two stages is output and alarmed.

[0023] Preferably, in the second stage, an intersection over union (IoU) calculation method is used to evaluate the overlap between the detection boxes in the two detection results; when the IoU value exceeds a set threshold, it is considered that the two detection boxes point to the same actual object, and the detection result with lower confidence is removed.

[0024] Preferably, in the third stage, the detection results in a plurality of consecutive frames in the video stream are continuously analyzed; if the landing gear is continuously detected in the plurality of image frames without meeting the preset detection standard, or if a bird, a balloon or an aerial floating object is continuously detected, it is considered that there is a potential landing risk, thereby automatically triggering an alarm mechanism.

[0025] Preferably, the process of continuously tracking the target aircraft is:

[0026] Target tracking trajectory generation: acquire video stream detection results, and associate the detection results in continuous multiple frames to realize tracking of the target;

[0027] Initial target selection: when there is only one aircraft target, continuously track the target and keep the target in the center of the screen; when there are multiple aircraft targets, select the aircraft with the largest image area as the main tracking target; for other aircraft targets except the main tracking target, add them to the filtering list;

[0028] Target tracking: after the main tracking target is selected, continuously track the main tracking target and keep the main tracking target always in the center of the screen;

[0029] Dynamic update of tracking position: during tracking of the aircraft, save and continuously update the position information of the current tracking target.

[0030] Preferably, during the initial target selection process, for other aircraft targets except the main tracking target, add them to the filtering list;

[0031] During target tracking, if the main tracking target fails, trigger the search mechanism to find the failed main tracking target again; exclude the aircrafts that have been added to the filtering list from the target trajectory set that can be considered; when a new target is found, if the target is not in the filtering list, try to find the original main tracking target among the newly appeared targets; compare the newly appeared target with the original main tracking target, the comparison content includes the closeness of the position when the original main tracking target fails and the position when the new target appears, and the trajectory similarity; if the newly appeared target is identified as the original main tracking target, continuously track the newly appeared target.

[0032] Preferably, the specific process of monitoring the flight trajectory of the aircraft to determine whether it deviates from the runway is as follows:

[0033] The first camera and the laser range finder are installed at point C on one side of the runway, point A is the left end point of the center axis of the runway, and point B is the right end point of the center axis of the runway;

[0034] The distance from point C to point P on the center axis of the runway is l1, i.e. the length of PC is l1; similarly, the distance from point P on the center axis of the runway to the right end point B of the center axis of the runway is l2, i.e. the length of PB is l2; the actual position of the aircraft in the air is point O, and the vertical projection to the ground is point O'; the laser range finder measures the straight-line distance d between point C and the aircraft O, i.e. the length of OC is d; the pitch angle of the first camera C is assumed to be α, i.e. ∠OCO' is α; the azimuth angle of the first camera C is set to β, i.e. ∠O'CN is β, where CN is an imaginary straight line parallel to the runway, and O'N is an imaginary straight line perpendicular to the runway;

[0035] The specific process for solving the yaw angle θ of the airplane relative to the center axis of the runway, i.e., solving ∠O'BM, and the distance value O'M of the deviation is as follows:

[0036] The distance O'C of the projection point O' of the airplane to the tower camera C is solved by a trigonometric function relationship:

[0037] O'C = d cos α

[0038] The distances O'N and CN are solved by the azimuth angle β of the point C and O'C:

[0039] O'N = O'C sin β = d cos α sin β

[0040] CN = O'C cos β = d cos α cos β

[0041] The lengths of O'M and BM are solved by the lengths of PC and PB, which are l1 and l2:

[0042] O'M = O'N - MN = O'N - PC = d cos α sin β - l1

[0043] BM = PM - PB = CN - PB = d cos α cos β - l2

[0044] O'M is the distance value of the deviation of the airplane from the center line of the runway; the yaw angle θ of the airplane is solved by a trigonometric function relationship by knowing the lengths of O'M and BM:

[0045]

[0046] Preferably, the specific process for monitoring the flight trajectory of the airplane to determine whether it deviates from the runway is as follows:

[0047] The yaw angle is estimated by deploying two second cameras at the left and right ends of the runway; the second cameras are located on the extension lines of the center axes of the two ends of the runway, and the pitch angles thereof are adjusted to ensure that the landing process of the airplane can be completely observed, and the azimuth angles thereof are adjusted to ensure that they are parallel to the center line of the runway;

[0048] The rectangular ABCD is the monitoring picture of the second camera, and a reference line is drawn on the monitoring picture to assist in determining whether the airplane deviates from the runway, which is represented by MM'; it is known that the installation position of the second camera is point O, the image resolution of the camera is W x H, and the horizontal field of view angle is FOV H ; the airplane target is detected, and its center position in the image is determined, assuming that the coordinates of the center of the airplane in the image are (x, y), wherein x and y are pixel coordinates; the yaw angle is defined as the included angle θ between the line connecting the center of the airplane target and the deployment point O of the second camera and the center line of the runway, and according to the similar relationship, it is obtained that:

[0049]

[0050] wherein the pixel width of the image W, the camera horizontal field of view FOV H are known quantities, the aircraft center coordinate x is obtained by image algorithm, and thus the estimated yaw angle is:

[0051]

[0052] Preferably, the sensor data in the history of take-off and landing process are recorded and analyzed to obtain the state that the sensor data should present in the normal take-off and landing state, which is recorded as the normal state; during the take-off and landing process of the aircraft, the real-time take-off and landing data of the aircraft measured by each sensor are compared with the data in the normal state, if the difference between the two is within the set threshold range, it indicates that the aircraft is in the normal take-off and landing state; if the difference between the two is outside the set threshold range, it indicates that the aircraft is likely to be in an abnormal take-off and landing state, there is a risk of take-off and landing, and an alarm needs to be issued.

[0053] The application also discloses an intelligent monitoring system for the take-off and landing process of an aircraft.

[0054] The camera monitoring module is composed of multiple cameras and is used for tracking and shooting the whole take-off and landing process of the aircraft.

[0055] The distance measuring module is installed at the same point as a certain camera and is used for measuring the distance between the aircraft and the monitoring device, and the camera at the point can give the azimuth angle and the pitch angle of the aircraft relative to the camera.

[0056] The intelligent recognition module is used for obtaining the real-time video stream of the camera and recognizing the aircraft and its landing gear in the image picture.

[0057] The intelligent tracking module is used for controlling the rotation and zoom of the camera according to the position of the aircraft in the camera picture given by the intelligent recognition module, so that the aircraft is always in the center of the camera picture with a proper size.

[0058] The runway deviation monitoring module is used for judging whether the aircraft deviates from the runway and the angle and distance of deviation according to the position information of the aircraft provided by the camera and the laser range finder.

[0059] The automatic alarm module is used for automatically issuing an alarm to remind the tower staff to take corresponding measures when the aircraft deviates from the runway or the number of landing gears is abnormal.

[0060] The display and man-machine interaction module is used for displaying the video picture of the camera and realizing man-machine interaction, so as to facilitate manual intervention.

[0061] Compared with the prior art, the application has the advantages that:

[0062] The intelligent monitoring method for aircraft take-off and landing process of the application can monitor the aircraft take-off and landing process from multiple angles and automatically identify the aircraft and its landing gear, determine whether the landing gear is in a normal state by automatically counting the number of landing gears, automatically track the entire take-off and landing process of the aircraft after identifying the aircraft, calculate the accurate position of the aircraft in real time through the distance, azimuth angle and pitch angle of the aircraft relative to the monitoring device, determine whether the aircraft deviates from the runway and calculate the angle and distance of deviation, and timely issue an alarm if the number of landing gears is abnormal or the aircraft deviates from the runway during the take-off and landing process.

[0063] The intelligent monitoring method and system for aircraft take-off and landing process of the application can analyze the motion trajectory of the aircraft and monitor the state of the landing gear through a deep learning algorithm after detecting the aircraft. When the aircraft is preparing to land, the system will also analyze the relative position of the aircraft and the center line of the runway to ensure that the aircraft accurately aligns with the runway during landing. If there is deviation, the system will issue a real-time warning to facilitate the pilot to adjust the heading in time. Finally, the monitoring data will be transmitted to the control center in real time and processed by the intelligent analysis module to quickly identify potential safety risks. Through this series of precise monitoring processes, the application can effectively improve the safety of aircraft take-off and landing, and also improve the efficiency and reliability of airport operation.

[0064] The application significantly improves the recognition accuracy of various states of the aircraft by fusing the camera views of multiple different positions and applying a deep learning algorithm for data fusion and analysis. This method can provide comprehensive visual coverage at the critical moments of aircraft take-off and landing, ensuring the capture and analysis of key information during the take-off and landing process.

[0065] The application can accurately measure and real-time process the deviation of the aircraft from the center axis of the runway by comprehensively using a multi-camera system and triangulation combined with real-time data flow. By deploying high-resolution cameras at both ends of the runway and using the tower camera and laser range finder to work synchronously, not only can the position of the aircraft be captured in real time, but also the yaw angle can be accurately calculated through the built-in sensor. This method enables the system to quickly respond and issue warnings in emergency situations and provide decision support, significantly enhancing the safety and efficiency of the entire aircraft take-off and landing process.

[0066] The present invention comprehensively monitors the aircraft's status through multiple perspectives and multiple sensors, and achieves comprehensive monitoring of the aircraft's various postures through the integration of multi-perspective cameras and laser rangefinders. The aircraft's operating status is monitored from multiple perspectives using cameras deployed at multiple locations, including locations near the control tower on both sides of the runway and the centerline at both ends of the runway. A laser rangefinder is deployed on one side of the runway at the same point as the camera to accurately measure the distance to the aircraft. Combined with the azimuth and pitch angle information provided by the camera, the aircraft's three-dimensional coordinates are accurately calculated, allowing for precise calculation of yaw angles and deviation distances, increasing the stability and reliability of the system and enabling more accurate and real-time flight safety monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the layout of the intelligent monitoring device for aircraft take-off and landing process in an embodiment of the present invention.

[0068] Figure 2 The figure is a flow chart of an embodiment of the method for intelligently monitoring the aircraft take-off and landing process of the present invention.

[0069] Figure 3 Schematic diagram of the runway deviation calculation method at the first point in the present invention.

[0070] Figure 4 Schematic diagram of the runway deviation calculation method at the second point in the present invention.

[0071] Figure 5 Schematic diagram of recognition frame overlap in the intersection-over-union (IoU) calculation method of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, a first camera and a laser rangefinder are deployed at the same point on one side of the runway. The laser rangefinder is integrated with the first camera or arranged separately, and the direction of the laser rangefinder is set to be consistent with the optical axis direction of the camera; in addition, a second camera is deployed on the extended line of the runway centerline at each end of the runway ( Figure 1 The second camera A and the second camera B in ).

[0074] like Figure 2 As shown, the intelligent monitoring method for aircraft take-off and landing process according to an embodiment of the present invention has the following specific processes:

[0075] First, the first camera is aimed at the direction of aircraft take-off and landing, and the real-time video image of the first camera is obtained and analyzed for intelligent target recognition. When an aircraft is recognized, it is immediately tracked continuously, and the laser rangefinder continuously measures the distance to the aircraft.

[0076] In the process of continuous tracking, on the one hand, the number of aircraft landing gears is identified and judged; if the number of landing gears is abnormal, an alarm is issued;

[0077] On the other hand, the aircraft pitch angle and azimuth angle information output by the first camera and the aircraft distance information output by the laser range finder are used to automatically calculate the accurate three-coordinate position of the aircraft, and then the position relationship between the aircraft and the runway is judged to determine whether the aircraft deviates from the runway; if the aircraft deviates from the runway, an alarm is issued, and the deviation angle and deviation distance are calculated.

[0078] In the above working process, no manual intervention is required, the intelligent degree is high, and the intelligent analysis and judgment of whether the aircraft deviates from the runway and whether the number of landing gears is abnormal can be performed at the same time.

[0079] The embodiment of the application also provides an intelligent monitoring system for aircraft take-off and landing, which comprises a camera monitoring module, a distance measuring module, an intelligent identification module, an intelligent tracking module, a runway deviation monitoring module, an automatic alarm module, a display and man-machine interaction module;

[0080] The camera monitoring module is composed of multiple cameras and tracks and photographs the entire process of aircraft take-off and landing;

[0081] The distance measuring module is installed at the same point as a certain camera and is used to measure the distance between the aircraft and the monitoring device; the camera at this point can give the azimuth angle and the pitch angle of the aircraft relative to the camera;

[0082] The intelligent identification module obtains the real-time video stream of the camera and identifies the aircraft and its landing gears in the image picture;

[0083] The intelligent tracking module controls the rotation and zoom of the camera according to the position of the aircraft in the camera picture given by the intelligent identification module, so that the aircraft is always in the center of the camera picture with an appropriate size;

[0084] The runway deviation monitoring module judges whether the aircraft deviates from the runway and the angle and distance of deviation according to the position information of the aircraft provided by the camera and the laser range finder;

[0085] When the system judges that the aircraft deviates from the runway or the number of landing gears is abnormal, the automatic alarm module automatically issues an alarm to remind the tower staff to take appropriate measures;

[0086] The display and man-machine interaction module is used to display the video picture of the camera and realize man-machine interaction, so as to facilitate manual intervention in the system.

[0087] In order to better understand the above technical solution, the following will combine the description of the drawings and the specific embodiments to explain each part of the above technical solution in detail.

[0088] Specifically, the specific process of the target intelligent identification and alarm is as follows:

[0089] The target detection algorithm is adopted to identify the target that may appear in the air during the process of the aircraft taking off and landing, including the aircraft, the aircraft landing gear, the flying bird, the balloon, and other possible floating objects in the air.

[0090] The first camera monitors the direction of the aircraft taking off and landing in real time by calling the preset position, the intelligent identification module obtains the video stream of the camera in real time, calls the target detection algorithm (such as the conventional YOLO algorithm) to analyze the video stream in real time, and issues an alien intrusion alarm if the flying bird, balloon or other floating object in the air is identified. If the aircraft and the aircraft landing gear are identified, the number of aircraft landing gears is calculated. If the number of landing gears does not reach the predetermined standard, an abnormal landing gear alarm is issued. The specific process is as follows:

[0091] 1. First stage - preliminary detection and extraction of region of interest

[0092] In this stage, the target detection algorithm is used to detect the entire image of the first camera globally, aiming to identify and locate the aircraft in the image and preliminarily identify the landing gear. The detection result of this stage is judged. If the detection result includes an aircraft, the number of landing gears in the detection result is calculated. If the number of landing gears meets the predetermined standard, this detection ends and the subsequent second stage detection is not performed; otherwise, the system extracts a region of interest slightly larger than the region where the aircraft is located as the center, which will be used for the second stage detection. The purpose of extracting the region of interest is to reduce the amount of image data for subsequent processing, and to focus on the key parts of the aircraft to improve processing efficiency and accuracy.

[0093] 2. Second stage - secondary fine detection and fusion

[0094] In this stage, the image of the region of interest extracted in the first stage is sent to a deep learning model (such as the conventional YOLO model) trained for landing gear identification. This model is optimized for the features of the landing gear and can accurately identify the landing gear in low-resolution or small-size images. After obtaining the detection result of the second stage, the landing gear detection result of the first stage is fused with the landing gear detection result of the second stage. Specifically, the present application uses the intersection over union (IoU) calculation method to evaluate the overlap between the detection boxes in the two detection results. For example, Figure 5As shown, assuming A and B represent the bounding boxes in two detection results respectively, when the two bounding boxes have a high degree of overlap, it is considered that they point to the same target object. At this time, in order to reduce redundant detection results, one of the target boxes needs to be removed. The measurement index of the degree of overlap is realized by calculating the IoU value, which is a ratio of the intersection area and the union area of the two bounding boxes. By pre-setting an IoU threshold, when the calculated IoU value exceeds the set threshold, it is considered that the two detection boxes point to the same actual object, and the detection result with lower confidence is removed. This deduplication strategy not only reduces redundant information, but also improves the accuracy and reliability of the final output.

[0095] 3. Third stage - detection result output and alarm

[0096] This stage will output and alarm the results of the previous two stages. In order to improve the overall robustness and reliability of the system, the present application introduces a technology based on continuous multi-frame analysis. Specifically, the system will continuously analyze the detection results in continuous multiple frames, and if the landing gear does not meet the preset detection standard or flying birds, balloons or other air floating objects are continuously detected in the multiple frames, it is considered that there is a potential landing risk and the alarm mechanism is automatically triggered.

[0097] This multi-frame analysis method not only improves the fault tolerance rate of single frame misjudgment, but also significantly enhances the continuous monitoring capability of the aircraft landing environment and landing gear state in dynamic environment.

[0098] The two-stage detection process of the present application includes primary detection and secondary refined detection; the primary detection is for the whole image, if the number of landing gears found in the primary detection does not meet the preset requirement, the secondary refined detection is started, the sub-image of the area where the aircraft is located is extracted and the sub-image is sent to a landing gear detection model, and finally the primary detection result and the secondary detection result are fused; if the fused result still indicates that the number of landing gears does not meet the preset, an abnormal landing gear alarm is issued, and the detection accuracy is high.

[0099] Specifically, the process of target intelligent tracking and searching is as follows:

[0100] The intelligent tracking module obtains the video stream detection result of the camera by the intelligent recognition module, and correlates the detection results in continuous multiple frames to realize target tracking. When there is only one aircraft target, the intelligent tracking module controls the first camera to continuously track the target and keeps the target in the center of the first camera screen; when there are multiple aircraft targets, the system automatically selects one of them for continuous tracking according to the set rules.

[0101] In addition, the application handles the target selection problem in the multi-aircraft environment through the tracking strategy of the target filtering and management mechanism; when the aircraft being continuously tracked disappears from the screen temporarily or is temporarily blocked, resulting in the failure of continuous tracking, the aircraft will be searched again.

[0102] The specific implementation steps are as follows:

[0103] 1. Target tracking trajectory generation

[0104] First, the intelligent tracking module obtains the detection result of the video stream of the first camera by the intelligent identification module, and correlates the detection results in continuous multiple frames to realize the tracking of the target. Each target generates a tracking trajectory and is assigned a unique tracking ID number.

[0105] 2. Initial target selection and filtering

[0106] When there is only one aircraft target, the tracking module controls the first camera to continuously track the target and keeps the target in the center of the screen of the first camera; when there are multiple aircraft targets, the aircraft with the largest image area is selected as the main tracking target. This selection is based on the fact that the aircraft with the largest image area is more likely to be the main aircraft landing or taking off in vision, and its dynamics is more worthy of attention. The aircraft targets other than the main tracking target will be added to a special filtering list.

[0107] 3. Target tracking and search

[0108] After the main tracking target is selected, the intelligent tracking module controls the first camera to continuously track the main tracking target and keeps the main tracking target always in the center of the screen of the first camera, so as to facilitate continuous observation and analysis of the landing and taking-off process.

[0109] Specifically, a real-time control algorithm automatically adjusts the camera's orientation and focal length to ensure the aircraft remains centered within the video frame and maintains a moderate size, preventing identification and tracking from being affected by aircraft that are too large or too small. The camera's pan and tilt angles are dynamically adjusted based on the aircraft's real-time position and predicted trajectory. Specifically, the system first determines the specific position of the currently tracked aircraft within the image frame based on the results of the target detection algorithm and calculates its offset relative to the center of the image frame. The system then adjusts the camera's pan and tilt angles based on this offset to center the tracked aircraft. For example, if the tracked aircraft is positioned slightly above the image frame, the camera is rotated upward; if the tracked aircraft is positioned slightly to the left, the camera is rotated left. The camera's focal length is adjusted based on the aircraft's proportion of the image frame. Specifically, the system first determines the proportion of the currently tracked aircraft within the image frame based on the results of the target detection algorithm. If the ratio is less than a certain set value, the camera focal length is increased to make the tracked aircraft occupy a larger proportion in the image; if the ratio is greater than another set value, the camera focal length is reduced to make the tracked aircraft occupy a smaller proportion in the image, thereby ensuring that the aircraft's key features such as landing gear are always clearly visible and ensuring good tracking effects.

[0110] If tracking of the primary target fails due to rapid movement or temporary obstruction during the tracking process, the system immediately triggers a search mechanism to re-locate the failed primary target. The system excludes aircraft that have been added to the filter list from the set of possible target trajectories to improve the success rate of re-locating the primary target. Because the filter list includes aircraft that were previously considered as non-primary targets due to being too far from the center of the image, these targets are not considered as new tracking targets at this stage. When a new target is found (not in the filter list), the system attempts to re-locate the original primary target among the newly discovered targets. The new target is compared with the original primary target, comparing its position at the time of failure to the position of the new target at the time of its appearance, as well as the degree of trajectory similarity. If the newly discovered target is identified as the original primary target, the intelligent tracking module controls the camera to continue tracking the new target. If the original target is not re-located within a specified period of time, the search is deemed a failure, the tracking mission ends, and the camera returns to the designated preset point. With this approach, monitoring can be quickly restored even if the target is lost, reducing the duration of monitoring interruption and potential security risks.

[0111] 4. Filter list and track location dynamic updates

[0112] In the tracking process of monitoring aircraft, the filtering list is dynamically updated according to the real-time data of each frame. At the same time, the system saves and continuously updates the position information of the current tracking main aircraft, which is not only crucial to maintaining tracking, but also a key basis for reselecting the tracking target after the aircraft is lost. By updating these data in real time, the risk of false tracking is greatly reduced, and the efficiency and reliability of the entire monitoring system are improved.

[0113] The method can significantly improve the stability of continuous tracking of the main tracking target.

[0114] Specifically, the specific process of monitoring the deviation of the aircraft from the runway is as follows:

[0115] The cameras at both ends of the runway, the auxiliary lines, are divided into multiple focal segments, and the positions are automatically changed. Each focal segment is divided into auxiliary lines, including vertical auxiliary lines and horizontal auxiliary lines. The vertical auxiliary lines are used for yaw detection, and the horizontal auxiliary lines are used for height early warning. The height cannot be too high or too low.

[0116] Two point positions are used to monitor and warn whether the aircraft deviates from the runway during takeoff and landing. The first point position is equipped with a first camera and a laser range finder, and the second point position is equipped with a second camera.

[0117] The runway deviation calculation method of the first point position is as shown in Figure 3 , wherein the first camera and the laser range finder are installed at point C (the height of the tower is ignored), point A is the left end point of the runway center line, and point B is the right end point of the runway center line. The runway deviation calculation method is described below with an example of an aircraft landing from right to left.

[0118] The distance from the first camera C to the point P on the runway center line can be measured, which is assumed to be l1, i.e. the length of PC is l1. Similarly, the distance from the point P on the runway center line to the right end point B of the runway center line can also be measured, which is assumed to be l2, i.e. the length of PB is l2. The actual position of the aircraft in the air is point O, and the vertical projection to the ground is point O'. The laser range finder can measure the straight-line distance d between the first camera C and the aircraft O, i.e. the length of OC is d. The pitch angle of the first camera C is assumed to be alpha, i.e. the angle of OCO' is alpha. The azimuth angle of the first camera C is set to beta, i.e. the angle of O'CN is beta, wherein CN is an imaginary straight line parallel to the runway, and O'N is an imaginary straight line perpendicular to the runway. Now we need to solve the yaw angle theta of the aircraft relative to the runway center line, i.e. solve the angle of O'BM, and the deviation distance value O'M.

[0119] The distance O'C from the projection point O' of the aircraft to the tower camera C can be solved by the trigonometric relationship:

[0120] O'C = dcosalpha

[0121] The distances of O'N and CN can be solved by O'C and the azimuth angle beta of the first camera C:

[0122] O'N = O'Cs in β = d cos α sin β

[0123] CN = O'Cs cos β = d cos α cos β

[0124] Now that the lengths of PC and PB are known as l1 and l2, the lengths of O'M and BM can be solved:

[0125] O'M = O'N - MN = O'N - PC = d cos α sin β - l1

[0126] BM = PM - PB = CN - PB = d cos α cos β - l2

[0127] O'M is the value of the distance of the aircraft from the centerline of the runway. Knowing the lengths of O'M and BM, the aircraft yaw angle θ can be solved by trigonometric functions:

[0128]

[0129] If the aircraft lands from the left, the yaw angle θ can be calculated according to the above process.

[0130] In another specific embodiment, as shown in FIG. 2, the yaw angle can also be estimated by two second cameras deployed at the left and right ends of the runway. The second cameras are located on the extension lines of the centerlines of the two ends of the runway, and their pitch angles are adjusted to ensure that the entire landing process of the aircraft can be observed, and their azimuth angles are adjusted to ensure that they are parallel to the centerline of the runway. Figure 4 In this embodiment, the rectangular ABCD is the monitoring screen of the second camera, and a reference line is drawn on the monitoring screen to assist in determining whether the aircraft deviates from the runway, which is represented by MM' in the figure. The installation position of the second camera is known as point O, the image resolution of the camera is W x H, and the horizontal field of view angle is FOV H The intelligent detection module can detect the aircraft target and determine its center position in the image. Assuming that the coordinates of the center of the aircraft in the image are (x, y), where x and y are pixel coordinates. The yaw angle is defined as the angle between the line connecting the center of the aircraft target and the deployment point O of the second camera and the centerline of the runway θ, and according to the similar relationship: Figure 4

[0131]

[0132] where the pixel width of the image W and the horizontal field of view angle FOV H of the camera are known quantities, and the center coordinates x of the aircraft can be obtained by image algorithm, so the estimated yaw angle is:

[0133] ​

[0134] The present application monitors and warns whether the aircraft deviates from the runway from multiple angles and using multiple sensors. On the one hand, the first camera and laser range finder deployed at the first point are used to measure and calculate the three-dimensional coordinate information of the aircraft in the air, and further calculate the angle and distance of the aircraft deviating from the center line of the runway. On the other hand, the second camera deployed at the second point is used to measure and calculate the angle of the aircraft deviating from the center line of the runway. If the angle or distance of the aircraft deviating from the center line of the runway exceeds a certain threshold, a runway deviation warning is issued.

[0135] In addition, in order to ensure the efficiency and accuracy of the aircraft take-off and landing monitoring system in data processing and analysis, the present application records and analyzes the image, the state of the landing gear, the runway deviation angle, the height of the aircraft from the ground, the sensor data of the laser range finder, etc. in real time during the take-off and landing process. When the historical take-off and landing data accumulates to a certain amount, it will be used as one of the bases for judging whether the subsequent take-off and landing process is normal.

[0136] Specifically, the data of each sensor during the historical take-off and landing process is recorded and analyzed to obtain the state that each sensor data should present under normal take-off and landing state, which is recorded as normal state. During the take-off and landing process of the aircraft, the real-time take-off and landing data of the aircraft measured by each sensor is compared with the data under the normal state. If the difference between the two is within the set threshold range, it means that the aircraft is in a normal take-off and landing state. If the difference between the two is outside the set threshold range, it means that the aircraft is likely to be in an abnormal take-off and landing state, there is a risk of take-off and landing, and an alarm needs to be issued. For example, in a normal take-off and landing process, the allowable runway deviation angle of the aircraft is within ±3°. When the horizontal angle of the aircraft detected by the camera deviates from this range, the system will judge that it is an abnormal state and prompt a possible take-off and landing risk. In addition, in a normal take-off and landing process, the distance between the aircraft and the ground presents a steady decreasing trend. Assuming that the rate of distance change is 40m / s to 50m / s, if the distance change rate detected by the laser range finder deviates significantly from this range, it means that there may be an abnormal situation and an alarm needs to be issued.

[0137] The intelligent monitoring method of the present application during the take-off and landing process of the aircraft. This method can monitor the take-off and landing process of the aircraft from multiple angles, automatically identify the aircraft and its landing gear, and determine whether the landing gear is in a normal state by automatically counting the number of landing gears. After identifying the aircraft, the take-off and landing process of the aircraft can be automatically tracked throughout the process. The accurate position of the aircraft is calculated in real time by the distance, azimuth angle, and pitch angle of the aircraft relative to the monitoring device, to determine whether the aircraft deviates from the runway and calculate the angle and distance of the deviation. If the number of landing gears is abnormal or the aircraft deviates from the runway during the take-off and landing process, the system will promptly issue an alarm.

[0138] The intelligent monitoring method and system for aircraft takeoff and landing of the present application, after detecting the aircraft, analyzes the motion trajectory of the aircraft and monitors the state of the landing gear through a deep learning algorithm. When the aircraft is preparing to land, the system will also analyze the relative position of the aircraft and the center line of the runway to ensure that the aircraft is accurately aligned with the runway during landing. If there is a deviation, the system will issue a real-time warning to facilitate the pilot to adjust the heading in time; finally, the monitoring data will be transmitted to the control center in real time for processing by the intelligent analysis module to quickly identify potential safety risks. Through this series of precise monitoring processes, the present application can effectively improve the safety of aircraft takeoff and landing, while also improving the efficiency and reliability of airport operations.

[0139] The present application significantly improves the recognition accuracy of various states of the aircraft by fusing the camera views of multiple different positions and applying deep learning algorithms for data fusion and analysis. This method can provide comprehensive visual coverage during the critical moments of aircraft takeoff and landing, ensuring the capture and analysis of key information during the landing process.

[0140] The present application can accurately measure and real-time process the deviation of the aircraft from the center line of the runway by comprehensively using a multi-camera system and triangulation combined with real-time data flow; by deploying high-resolution cameras at both ends of the runway and using tower cameras and laser range finders to work synchronously, not only can the position of the aircraft be captured in real time, but also the yaw angle can be accurately calculated through the built-in sensors; this method enables the system to respond quickly and issue warnings in emergency situations and provide decision support, significantly enhancing the safety and efficiency of the entire aircraft takeoff and landing process.

[0141] The present application comprehensively monitors the state of the aircraft through multiple perspectives and multiple sensors, and realizes comprehensive monitoring of the attitude of the aircraft through the fusion of multi-perspective cameras and laser range finders. The cameras are deployed at multiple points to monitor the running state of the aircraft from multiple perspectives, and the deployment points of the cameras include the positions near the tower on both sides of the runway and the center line positions at both ends of the runway. Laser range finders are deployed at the same point as the cameras on one side of the runway to accurately measure the distance of the aircraft, combined with the azimuth angle and pitch angle information provided by the cameras, to accurately calculate the three-dimensional coordinates of the aircraft, thereby accurately calculating the yaw angle and deviation distance, increasing the stability and reliability of the system, and realizing more accurate and real-time flight safety monitoring.

[0142] The present application is supported by the Hunan Provincial Science and Technology Innovation Plan, project number: 2023RC3230.

[0143] The above is only a preferred embodiment of the present application, and the protection scope of the present application is not limited to the above examples. Any technical solution that falls within the scope of the present application should be considered within the protection scope of the present application. It should be noted that for ordinary technical personnel in the technical field, some improvements and refinements without departing from the principles of the present application should be considered within the protection scope of the present application.

Claims

1. An intelligent monitoring method for aircraft take-off and landing process, characterized in that: Including steps: Obtain video streams of aircraft takeoff and landing directions and identify aircraft in the video streams; when an aircraft is identified, continuously track the target aircraft; During the continuous tracking of the target aircraft, the number of landing gears of the aircraft is identified, and an abnormal alarm is issued when the number of landing gears is abnormal; And during the continuous tracking process, the flight trajectory of the aircraft is monitored to determine whether it deviates from the runway, and a runway deviation warning is issued when it deviates from the runway; The specific process of identifying aircraft in the video stream is as follows: Phase 1: Preliminary detection and region of interest extraction Perform a global inspection of the entire video stream, identify and locate the aircraft in the image, and perform preliminary identification of the landing gear to obtain a detection result, which is then judged. If the detection result includes the aircraft, the number of landing gear in the detection result is calculated. If the number of landing gear meets the predetermined standard, the current inspection is completed and the subsequent second-stage inspection is not performed. Otherwise, a region of interest (ROI) larger than the aircraft area is extracted with the aircraft area as the center for the second-stage inspection. Phase 2: Secondary refined detection and fusion The image of the region of interest extracted in the first stage is fed into the preset landing gear recognition model to further identify the landing gear and obtain the detection result; After obtaining the second stage test results, the first stage landing gear test results are integrated with the second stage landing gear test results; Phase 3: Detection result output and alarm Output and issue alarms for the fusion results of the previous two stages.

2. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, characterized in that: In the second stage, the intersection-over-union (IoU) calculation method is used to evaluate the degree of overlap between the detection boxes in the two detection results. When the IoU value exceeds the set threshold, the two detection boxes are considered to point to the same actual object, and the detection results with lower confidence are removed.

3. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, characterized in that: In the third stage, the detection results in multiple consecutive frames in the video stream are continuously analyzed. If the landing gear is continuously detected in these multiple frames and fails to meet the preset detection standards, or if flying birds, balloons or floating objects are continuously detected in these multiple frames, it is considered that there is a potential landing risk and the alarm mechanism is automatically triggered.

4. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: The process of continuously tracking the target aircraft is as follows: Target tracking trajectory generation: Obtain the video stream detection results and associate the detection results in multiple consecutive frames to achieve target tracking; Initial target selection: When there is only one aircraft target, it will be continuously tracked and kept in the center of the image; when there are multiple aircraft targets, the aircraft occupying the largest image area will be selected as the primary tracking target; other aircraft targets besides the primary tracking target will be added to the filter list; Target tracking: Once the main tracking target is selected, it will be tracked continuously and kept in the center of the screen at all times. Dynamic update of tracking position: During the tracking process of the monitored aircraft, the position information of the current tracking target is saved and continuously updated.

5. The intelligent monitoring method for aircraft take-off and landing process according to claim 4, characterized in that: During the initial target selection process, aircraft targets other than the primary tracking target are added to the filter list; During the target tracking process, if the main tracking target fails to be tracked, the search mechanism is triggered to find the main tracking target that failed to be tracked again; the aircraft that has been added to the filter list is excluded from the set of target tracks that can be considered; When searching for a new target, if the target is not in the filter list, it will try to find the original tracking target among the new targets; Compare the newly appeared target with the original tracking target. The comparison includes the proximity between the position of the original tracking target when tracking fails and the position of the new target when it first appears, as well as the similarity of their trajectory. If the newly appeared target is identified as the original tracking target, the newly appeared target will be continuously tracked.

6. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: The specific process of monitoring the flight trajectory of an aircraft to determine whether it deviates from the runway is as follows: The first camera and laser rangefinder are installed at the same time at a point on one side of the runway. C ,point A is the left end point of the runway centerline. B It is the right end point of the runway centerline; point C To the point on the runway centerline P The distance is l 1, that is PC Length is l 1; Similarly, the point on the runway centerline P To the right end of the runway centerline B The distance is l 2, that is, the length of PB is l 2. The actual position of the aircraft in the air is a point O , vertically projected onto the ground as point ; Laser rangefinder measurement point C and airplanes O The straight-line distance between d ,Right now OC Length is d , the first camera C The pitch angle is assumed to be α , that is ∠ OCO' for α , the first camera C The azimuth angle is set to β , that is ∠ O'CN for β ,in CN is an imaginary straight line parallel to the runway, O'N It is an imaginary straight line perpendicular to the runway; Solve for the aircraft's yaw angle relative to the runway's centerline θ , that is, solving ∠ O'BM , and the deviation distance value O'M The specific process is: Solve the aircraft projection point through trigonometric function relationship O' To the tower camera C distance O'C : pass O'C and point C Azimuth β Solution O'N and CN distance: Known PC and PB The length is l 1 and l 2. Solution O'M and BM Length: O'M That is the distance the aircraft deviates from the runway centerline; O'M and BM The length of the aircraft is solved by trigonometric functions to obtain the yaw angle of the aircraft. θ : 。 7. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: The specific process of monitoring the flight trajectory of an aircraft to determine whether it deviates from the runway is as follows: The yaw angle is estimated using two secondary cameras deployed at the left and right ends of the runway. These cameras are located on the extended lines of the runway's centerline, with their pitch angles adjusted to ensure full observation of the aircraft's landing process and their azimuth angles adjusted to ensure they are aligned parallel to the runway's centerline. rectangle ABCD The second camera's monitoring screen is used to draw a reference line on the monitoring screen to help determine whether the aircraft has deviated from the runway. MM' Indicates that the second camera installation position is known to be point O , the camera image resolution is W × H , the horizontal field of view angle is FOV H ; Detect the aircraft target and determine its center position in the image. Assume that the coordinates of the center of the aircraft in the image are ( x , y ),in x and y All are pixel coordinates; the yaw angle is defined as the distance between the aircraft target center point and the second camera deployment point. O Angle between the connecting line and the runway centerline θ , according to the geometric relationship: The pixel width of the image W , camera horizontal field of view FOV H All are known quantities, the coordinates of the center of the aircraft x Obtained through image algorithm, the estimated yaw angle is: 。 8. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: Record and analyze the sensor data during historical takeoff and landing processes to determine the state that the sensor data should present under normal takeoff and landing conditions, which is recorded as the normal state; During the aircraft takeoff and landing process, the real-time takeoff and landing data of the aircraft measured by each sensor is compared with the data under normal conditions. If the difference between the two is within the set threshold range, it means that the aircraft is in a normal takeoff and landing state; if the difference between the two is outside the set threshold range, it means that the aircraft is likely to be in an abnormal takeoff and landing state, there is a takeoff and landing risk, and an alarm needs to be issued.

9. An intelligent monitoring system for aircraft take-off and landing processes, configured to execute the steps of the intelligent monitoring method for aircraft take-off and landing processes as claimed in any one of claims 1 to 8, characterized in that: It includes camera monitoring module, distance measurement module, intelligent recognition module, intelligent tracking module, runway deviation monitoring module, automatic alarm module, display and human-computer interaction module; The video monitoring module consists of multiple cameras, which are used to track and shoot the entire process of aircraft take-off and landing; The distance measurement module is installed at the same location as a camera to measure the distance between the aircraft and the monitoring equipment. The camera at this location can provide the azimuth and pitch angle of the aircraft relative to the camera. An intelligent recognition module, which is used to obtain the real-time video stream of the camera and identify the aircraft and its landing gear in the image; The intelligent tracking module is used to control the rotation and zoom of the camera according to the position of the aircraft in the camera image given by the intelligent recognition module, so that the aircraft is always in the center of the camera image at an appropriate size; The runway deviation monitoring module is used to determine whether the aircraft has deviated from the runway, as well as the angle and distance of the deviation, based on the aircraft position information provided by the camera and laser rangefinder; Automatic alarm module, used to automatically issue an alarm when the aircraft deviates from the runway or the number of landing gear is abnormal, so as to remind the tower staff to take corresponding measures; The display and human-computer interaction module is used to display the video images of the camera and realize human-computer interaction to facilitate manual intervention.

Images