Intelligent monitoring method and system for takeoff and landing process of airplane

Abstract

The invention discloses an intelligent monitoring method and system for the takeoff and landing process of an airplane, and the method comprises the steps: obtaining a video stream in the takeoff and landing direction of the airplane, and recognizing the airplane in the video stream; when the aircraft is identified, continuously tracking the target aircraft; in the continuous tracking process of the target aircraft, the number of undercarriages of the aircraft is identified, and when the number of the undercarriages is abnormal, an abnormity alarm is given; and in the continuous tracking process, monitoring the flight path of the aircraft to judge whether the aircraft deviates from the runway, and giving a runway deviation alarm when the aircraft deviates from the runway. The method has the advantages of improving the taking-off and landing safety and efficiency of the airplane and the like.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of aircraft take-off and landing monitoring, and in particular to an intelligent monitoring method and system for aircraft take-off and landing process. Background Art

[0002] In recent years, advanced science and technology have promoted the rapid development of civil aviation. The rapid economic development has significantly improved people's living standards. More and more people choose to travel by plane. The safe operation of aircraft is a strong guarantee for the safety of people's lives and property. The take-off and landing process of an aircraft is the most dangerous stage in the entire flight process, and it is also the stage most prone to accidents. Therefore, monitoring the take-off and landing process of an aircraft and timely tracking the flight status of the aircraft play an important role in preventing accidents.

[0003] Aircraft takeoff and landing monitoring technology has made significant progress in the civil aviation field. High-resolution cameras and thermal imaging equipment are widely used in airport monitoring systems. These devices can work stably under various lighting and weather conditions, provide clear visual information, and are key components of airport monitoring systems.

[0004] At present, the take-off and landing process of an aircraft is usually monitored by manual observation or video monitoring. The tower staff observes through a telescope, or observes through the aircraft images collected by the camera, or understands the flight status of the aircraft based on other sensors, such as radar. The staff judges whether there is a risk of deviation from the route during the take-off and landing process, whether the flight altitude and speed are appropriate, whether the landing gear is in place, etc. through their own observation and analysis. This manual judgment method is highly dependent on human experience, the timeliness of the judgment is insufficient, the accuracy is not high, and the intelligence level of the entire system is low.

[0005] Traditional monitoring systems can often only monitor aircraft from fixed angles, which limits the ability to fully monitor the state of the aircraft's landing gear in different postures, reducing the accuracy and timeliness of the monitoring effect. Especially at the critical moments of takeoff and landing, this fixed monitoring perspective limits the system's ability to capture abnormal states of the landing gear. Even within the visible range, the image resolution of the fixed camera may not be sufficient to capture the subtle features of the landing gear in detail.

[0006] Existing monitoring methods often use a single video monitoring method when monitoring aircraft runway alignment. Due to the limitations of the video sensor itself, it is unable to provide accurate aircraft location information, which increases the safety risk of aircraft takeoff and landing.

[0007] Patent application CN103287584B proposes an aircraft video landing assistance system, which deploys a fixed camera at multiple different positions on the center line of the runway to collect images of the landing aircraft in real time. The image information is transmitted to the aircraft through a wireless communication system and displayed simultaneously on different displays on the aircraft. Auxiliary lines are drawn on the display to facilitate the pilot to observe his own position in time, find the landing direction and landing point, and improve the accuracy and stability of landing. This method can only give qualitative results and cannot accurately calculate the deviation angle and deviation distance. At the same time, this method relies on wireless communication technology, the amount of video data is large, and the stability of communication is required to be high. Otherwise, it will cause large data delays, data loss and other problems, making it impossible for the pilot to make correct judgments in time.

[0008] Patent application CN106295695B proposes a method and device for automatically tracking and photographing an aircraft during takeoff and landing. This method uses a pan / tilt camera deployed on one side of the tower to automatically track the entire process of aircraft takeoff and landing, and uses a zoom camera to keep the aircraft in the center of the image at an appropriate size. This method assumes that the aircraft is always located on the vertical plane where the runway centerline is located during takeoff and landing, so it is impossible to detect and warn the aircraft of deviating from the runway.

[0009] In summary, the existing aircraft take-off and landing monitoring system often only monitors the aircraft's flight altitude, flight speed, flight direction and other parameters to determine whether the aircraft can land safely, but does not effectively monitor the status of the aircraft's landing gear. During the landing process, the timely deployment of the landing gear directly affects the safety of the aircraft after landing. Existing monitoring systems often rely on a single perspective and cannot fully capture the key information of the aircraft in different postures. Existing technologies have limitations in real-time analysis of runway alignment. Usually, the aircraft's orientation is used to determine whether the aircraft has deviated from the runway, without measuring the distance of the aircraft, so the aircraft's three-dimensional position information in the air cannot be obtained. Summary of the invention

[0010] In view of the technical problems existing in the prior art, the present invention provides an intelligent monitoring method and system for an aircraft take-off and landing process, which can improve the safety of the aircraft take-off and landing process.

[0011] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0012] An intelligent monitoring method for aircraft take-off and landing process, comprising the steps of:

[0013] Obtain the video stream of the aircraft take-off and landing direction, and identify the aircraft in the video stream; when the aircraft is identified, continue to track the target aircraft;

[0014] In the process of continuously tracking the target aircraft, the number of the aircraft landing gears is identified, and an abnormal alarm is issued when the number of the landing gears is abnormal;

[0015] And during the continuous tracking process, the flight trajectory of the aircraft is monitored to determine whether it deviates from the runway, and a runway deviation warning is issued when it deviates from the runway.

[0016] Preferably, the specific process of identifying the aircraft in the video stream is:

[0017] Phase 1: Preliminary detection and region of interest extraction

[0018] Perform global detection on the entire image in the video stream, identify and locate the aircraft in the image and perform preliminary identification of the landing gear to obtain the detection result, and then judge the detection result; if the detection result includes the aircraft, calculate the number of landing gears in the detection result; if the number of landing gears meets the predetermined standard, the current detection ends and the subsequent second stage detection is no longer performed; otherwise, extract an area of ​​interest larger than the area where the aircraft is located with the aircraft as the center for the second stage detection;

[0019] Phase 2: Secondary refined detection and fusion

[0020] The image of the region of interest extracted in the first stage is sent to the preset landing gear recognition model to further identify the landing gear and obtain the detection result; after obtaining the detection result of the second stage, the landing gear detection result of the first stage is fused with the landing gear detection result of the second stage;

[0021] Phase 3: Detection result output and alarm

[0022] The fusion results of the previous two stages are output and alarms are issued.

[0023] Preferably, in the second stage, the intersection-over-union (IoU) calculation method is used to evaluate the degree of overlap between the detection boxes in the two detection results; when the IoU value exceeds the set threshold, it is considered that the two detection boxes point to the same actual object, and the detection result with lower confidence is removed.

[0024] Preferably, in the third stage, the detection results in multiple consecutive frames in the video stream are continuously analyzed. If it is continuously detected in these multiple frames that the landing gear does not meet the preset detection standard, or if flying birds, balloons or floating objects in the air are continuously detected, it is considered that there is a potential landing risk and the alarm mechanism is automatically triggered.

[0025] Preferably, the process of continuously tracking the target aircraft is:

[0026] Target tracking trajectory generation: Obtain the video stream detection results and associate the detection results in multiple consecutive frames to achieve target tracking;

[0027] Initial target selection: When there is only one aircraft target, the target will be continuously tracked and kept in the center of the image; when there are multiple aircraft targets, the aircraft occupying the largest image area will be selected as the main tracking target; other aircraft targets except the main tracking target will be added to the filter list;

[0028] Target tracking: After the main tracking target is selected, it will be continuously tracked and kept in the center of the screen at all times;

[0029] Dynamic update of tracking position: During the tracking process of the monitoring aircraft, the position information of the current tracking target is saved and continuously updated.

[0030] Preferably, during the initial target selection process, other aircraft targets other than the primary tracking target are added to the filter list;

[0031] During the target tracking process, if the main tracking target fails to be tracked, the search mechanism is triggered to retrieve the main tracking target that failed to be tracked; the aircraft that has been added to the filter list is excluded from the set of target trajectories that can be considered; when a new target is searched, if the target is not in the filter list, an attempt will be made to retrieve the original main tracking target from the newly appeared target; the newly appeared target is compared with the original main tracking target, and the comparison content includes the proximity of the position of the original main tracking target when the tracking fails and the position when the new target just appears, as well as the similarity of the trajectory; if the newly appeared target is identified as the original main tracking target, the newly appeared target is continuously tracked.

[0032] Preferably, the specific process of monitoring the flight trajectory of the aircraft to determine whether it deviates from the runway is:

[0033] The first camera and the laser rangefinder are installed at point C on one side of the runway, point A is the left end point of the runway centerline, and point B is the right end point of the runway centerline;

[0034] The distance from point C to point P on the centerline of the runway is l1, that is, the length of PC is l1; similarly, the distance from point P on the centerline of the runway to the right end point B of the centerline of the runway is l2, that is, the length of PB is l2; the actual position of the aircraft in the air is point O, and the vertical projection to the ground is point O′; the laser rangefinder measures the straight-line distance d between point C and aircraft O, that is, the length of OC is d, the pitch angle of the first camera C is assumed to be α, that is, ∠OCO' is α, and the azimuth angle of the first camera C is set to β, that is, ∠O'CN is β, where CN is an imaginary straight line parallel to the runway, and O'N is an imaginary straight line perpendicular to the runway;

[0035] The specific process of solving the yaw angle θ of the aircraft relative to the centerline of the runway, that is, solving ∠O'BM, and the deviation distance value O'M is:

[0036] The distance O'C from the aircraft projection point O' to the tower camera C is solved by the trigonometric function relationship:

[0037] O′C=dcosα

[0038] Solve the distance between O'N and CN by the azimuth angle β of O'C and point C:

[0039] O′N=O′Csinβ=d cosαsinβ

[0040] CN=O′Ccosβ=d cosαcosβ

[0041] Given that the lengths of PC and PB are l1 and l2, find the lengths of O'M and BM:

[0042] O′M=O′N-MN=O′N-PC=dcosαsinβ-l1

[0043] BM=PM-PB=CN-PB=dcosαcosβ-l2

[0044] O'M is the distance the aircraft deviates from the runway centerline. Knowing the lengths of O'M and BM, the aircraft yaw angle θ can be solved by trigonometric functions:

[0045]

[0046] Preferably, the specific process of monitoring the flight trajectory of the aircraft to determine whether it deviates from the runway is:

[0047] The yaw angle is estimated by two second cameras deployed at the left and right ends of the runway; the second cameras are located on the extension lines of the center axis at both ends of the runway, and their pitch angles are adjusted to ensure that the landing process of the aircraft can be fully observed, and their azimuth angles are adjusted to ensure that their orientation is parallel to the center line of the runway;

[0048] Rectangle ABCD is the monitoring screen of the second camera, and a reference line is drawn on the monitoring screen to assist in judging whether the aircraft deviates from the runway, represented by MM'; the installation position of the second camera is known to be point O, the image resolution of the camera is W×H, and the horizontal field of view angle is FOV H ; Detect the aircraft target and determine its center position in the image. Assume that the coordinates of the center of the aircraft in the image are (x, y), where x and y are both pixel coordinates; define the yaw angle as the angle θ between the line connecting the center point of the aircraft target and the second camera deployment point O and the center line of the runway. According to the geometric relationship:

[0049]

[0050] The pixel width of the image is W, and the horizontal field of view of the camera is FOV. H All are known quantities. The center coordinate x of the aircraft is obtained through the image algorithm, so the estimated yaw angle is:

[0051]

[0052] Preferably, the sensor data in the historical take-off and landing process are recorded and analyzed to obtain the state that each sensor data should present in the normal take-off and landing state, which is recorded as the normal state; during the aircraft take-off and landing process, the real-time take-off and landing data of the aircraft measured by each sensor is compared with the data in the normal state. If the difference between the two is within the set threshold range, it means that the aircraft is in a normal take-off and landing state; if the difference between the two is outside the set threshold range, it means that the aircraft is likely to be in an abnormal take-off and landing state, there is a take-off and landing risk, and an alarm needs to be issued.

[0053] The present invention also discloses an intelligent monitoring system for aircraft take-off and landing process, including a video monitoring module, a distance measurement module, an intelligent identification module, an intelligent tracking module, a runway deviation monitoring module, an automatic alarm module, and a display and human-computer interaction module;

[0054] The video monitoring module consists of multiple cameras, which are used to track and shoot the entire process of aircraft take-off and landing;

[0055] The distance measurement module is installed at the same point as a camera to measure the distance between the aircraft and the monitoring equipment. The camera at this point can give the azimuth and pitch angle of the aircraft relative to the camera;

[0056] An intelligent recognition module, which is used to obtain the real-time video stream of the camera and identify the aircraft and its landing gear in the image;

[0057] The intelligent tracking module is used to control the rotation and zoom of the camera according to the position of the aircraft in the camera image given by the intelligent recognition module, so that the aircraft is always in the center of the camera image with an appropriate size;

[0058] The runway deviation monitoring module is used to determine whether the aircraft deviates from the runway, as well as the angle and distance of deviation based on the aircraft position information provided by the camera and the laser rangefinder;

[0059] Automatic warning module, used to automatically send out an alarm when the aircraft deviates from the runway or the number of landing gear is abnormal, so as to remind the tower staff to take corresponding measures;

[0060] The display and human-computer interaction module is used to display the video images of the camera and realize human-computer interaction to facilitate manual intervention.

[0061] Compared with the prior art, the advantages of the present invention are:

[0062] The intelligent monitoring method for the aircraft take-off and landing process of the present invention can monitor the aircraft take-off and landing process from multiple angles, automatically identify the aircraft and its landing gear, and judge whether the landing gear is in a normal state by automatically counting the number of landing gears. After identifying the aircraft, the whole take-off and landing process of the aircraft can be automatically tracked, and the accurate position of the aircraft can be calculated in real time by the distance, azimuth and pitch angle of the aircraft relative to the monitoring equipment, and whether it deviates from the runway and calculates the angle and distance of the deviation; during the take-off and landing process of the aircraft, if the number of landing gears is abnormal or the aircraft deviates from the runway, the system will issue an alarm in time.

[0063] The intelligent monitoring method and system for aircraft takeoff and landing of the present invention, after detecting the aircraft, the system analyzes the movement trajectory of the aircraft and monitors the status of the landing gear through a deep learning algorithm. When the aircraft is preparing to land, the system will also analyze the relative position of the aircraft and the centerline of the channel to ensure that the aircraft is accurately aligned with the runway when landing. If there is any deviation, the system will issue a warning in real time to facilitate the pilot to adjust the course in time; finally, the monitoring data will be transmitted to the control center in real time and processed by the intelligent analysis module to quickly identify potential safety risks. Through this series of precise monitoring processes, the present invention can effectively improve the safety of aircraft takeoff and landing, while also improving the efficiency and reliability of airport operations.

[0064] The present invention significantly improves the recognition accuracy of each state of the aircraft by fusing camera views from multiple different positions and applying deep learning algorithms for data fusion and analysis. This method can provide comprehensive view coverage at the critical moments of takeoff and landing, ensuring the capture and analysis of key information during takeoff and landing.

[0065] The present invention uses a multi-camera system and triangulation in combination with real-time data streams to accurately measure and process in real time the deviation of the aircraft from the runway centerline. By deploying high-resolution cameras at both ends of the runway and using tower cameras and laser rangefinders to work synchronously, not only can the position of the aircraft be captured in real time, but the yaw angle can also be accurately calculated through built-in sensors. This method enables the system to respond quickly in emergency situations and issue warnings in a timely manner and provide decision support, significantly enhancing the safety and efficiency of the entire aircraft take-off and landing process.

[0066] The present invention comprehensively monitors the aircraft status through multi-viewing angles and multi-sensors, and realizes comprehensive monitoring of all postures of the aircraft through the integration of multi-viewing angle cameras and laser rangefinders. Cameras deployed at multiple points are used to perform multi-viewing angle monitoring of the aircraft's operating status. The deployment points of the cameras include positions close to the tower on both sides of the runway, and the centerline positions at both ends of the runway. A laser rangefinder is deployed at the same point as the camera on one side of the runway to accurately measure the distance to the aircraft. Combined with the azimuth and pitch angle information given by the camera, the three-dimensional coordinates of the aircraft are accurately calculated, thereby accurately calculating the yaw angle and deviation distance, increasing the stability and reliability of the system, and realizing more accurate and real-time flight safety monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The figure is a schematic diagram of the arrangement of the intelligent monitoring device for aircraft take-off and landing process in an embodiment of the present invention.

[0068] Figure 2 The figure is a flow chart of an embodiment of the method for intelligently monitoring the aircraft take-off and landing process of the present invention.

[0069] Figure 3 It is a schematic diagram of the runway deviation calculation method of the first point in the present invention.

[0070] Figure 4 It is a schematic diagram of the runway deviation calculation method of the second point in the present invention.

[0071] Figure 5 It is a schematic diagram of recognition frame overlap in the intersection-over-union ratio calculation method of the present invention. DETAILED DESCRIPTION

[0072] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, a first camera and a laser rangefinder are deployed at the same point on one side of the runway, wherein the laser rangefinder is integrated with the first camera or arranged separately, and the direction of the laser rangefinder is set to be consistent with the optical axis direction of the camera; in addition, a second camera ( Figure 1 The second camera A and the second camera B in it).

[0074] like Figure 2 As shown, the intelligent monitoring method for aircraft take-off and landing process according to the embodiment of the present invention has the following specific process:

[0075] First, the first camera is aimed at the direction of the aircraft's take-off and landing, and the real-time video image of the first camera is obtained and analyzed to perform intelligent target recognition. When the aircraft is recognized, the aircraft is immediately tracked continuously, and the laser rangefinder continuously measures the distance to the aircraft;

[0076] During the continuous tracking process, on the one hand, the number of aircraft landing gears is identified and determined; if the number of landing gears is abnormal, an alarm is issued;

[0077] On the other hand, the aircraft's pitch angle and azimuth angle information are output by the first camera, and the aircraft's distance information is output by the laser rangefinder to automatically calculate the aircraft's accurate three-coordinate position, and then determine whether the aircraft deviates from the runway based on the positional relationship between the aircraft and the runway; if the aircraft deviates from the runway, an alarm is issued and the deviation angle and deviation distance are calculated.

[0078] The above work process does not require human intervention and has a high degree of intelligence. It can also perform intelligent analysis and judgment on whether the aircraft has deviated from the runway and whether the number of landing gear is abnormal.

[0079] The embodiment of the present invention also provides an intelligent monitoring system for aircraft take-off and landing, including a video monitoring module, a distance measurement module, an intelligent recognition module, an intelligent tracking module, a runway deviation monitoring module, an automatic alarm module, and a display and human-computer interaction module;

[0080] The video monitoring module consists of multiple cameras, which can track and shoot the entire process of aircraft take-off and landing;

[0081] The distance measurement module is installed at the same point as a camera to measure the distance between the aircraft and the monitoring equipment. The camera at this point can give the azimuth and pitch angle of the aircraft relative to the camera;

[0082] The intelligent recognition module obtains the real-time video stream of the camera and identifies the aircraft and its landing gear in the image;

[0083] The intelligent tracking module controls the rotation and zoom of the camera according to the position of the aircraft in the camera image given by the intelligent recognition module, so that the aircraft is always in the center of the camera image with an appropriate size;

[0084] The runway deviation monitoring module determines whether the aircraft deviates from the runway, as well as the angle and distance of deviation, based on the aircraft position information provided by the camera and laser rangefinder;

[0085] When the system determines that the aircraft has deviated from the runway, or the number of landing gear is abnormal, the automatic warning module will automatically issue an alarm to remind the tower staff to take corresponding measures;

[0086] The display and human-computer interaction module is used to display the camera's video images and realize human-computer interaction, making it convenient for humans to intervene in the system.

[0087] In order to better understand the above technical solution, each part of the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0088] Specifically, the specific process of target intelligent identification and alarm is as follows:

[0089] The present invention adopts a target detection algorithm to identify targets that may appear in the air during the take-off and landing of an aircraft, including aircraft, aircraft landing gear, flying birds, balloons, and other possible floating objects in the air.

[0090] The first camera monitors the take-off and landing direction of the aircraft in real time by calling the preset position. The intelligent recognition module obtains the video stream of the camera in real time, and calls the target detection algorithm (such as the conventional YOLO algorithm, etc.) to analyze the video stream in real time. If a flying bird, balloon or other floating object in the air is identified, a foreign object intrusion alarm is issued; if an aircraft and aircraft landing gear are identified, the number of aircraft landing gear is calculated. If the number of landing gear does not meet the preset standard, a landing gear abnormality alarm is issued. The specific process is as follows:

[0091] 1. Phase 1 - Preliminary Detection and Region of Interest Extraction

[0092] In this stage, the target detection algorithm is used to perform global detection on the entire screen of the first camera, aiming to identify and locate the aircraft in the image and perform preliminary identification of the landing gear. The detection results of this stage are judged. If the detection results include the aircraft, the number of landing gears in the detection results is calculated. If the number of landing gears meets the predetermined standard, the detection ends and the subsequent second stage detection is no longer performed; otherwise, the system will extract a region of interest slightly larger than the area where the aircraft is located with the aircraft as the center, and the region of interest will be used for the second stage detection. The purpose of extracting the region of interest is to reduce the amount of image data to be processed later, while focusing on the key parts of the aircraft to improve processing efficiency and accuracy.

[0093] 2. The second stage - secondary refined detection and fusion

[0094] In this stage, the image of the region of interest extracted in the first stage is fed into a deep learning model trained for landing gear recognition (such as a conventional YOLO model). This model is optimized for the features of the landing gear and can accurately identify the landing gear in low-resolution or small-size images. After obtaining the detection results of the second stage, the landing gear detection results of the first stage are fused with the landing gear detection results of the second stage. Specifically, the present invention uses the intersection over union (IoU) calculation method to evaluate the degree of overlap between the detection boxes in the two detection results. Figure 5As shown in the figure, assuming that A and B represent the recognition boxes in the two detection results respectively, when the overlap degree of the two recognition boxes is high, it is considered that they point to the same target object. At this time, in order to reduce redundant detection results, one of the target boxes needs to be removed. The overlap degree is measured by calculating the IoU value, which is a ratio of the intersection area and the union area of ​​the two recognition boxes. By pre-setting an IoU threshold, when the calculated IoU value exceeds the set threshold, it is considered that the two detection boxes point to the same actual object, and the detection results with lower confidence are removed. This deduplication strategy not only reduces redundant information, but also improves the accuracy and reliability of the final output.

[0095] 3. The third stage - test result output and alarm

[0096] This stage will output and warn the results of the previous two stages. In order to improve the overall robustness and reliability of the system, the present invention introduces a technology based on continuous multi-frame analysis. Specifically, the system will continuously analyze the detection results in multiple frames. If the landing gear is continuously detected in these multiple frames that does not meet the preset detection standards, or if flying birds, balloons or other floating objects in the air are continuously detected, it will be considered that there is a potential landing risk and the alarm mechanism will be automatically triggered.

[0097] This multi-frame analysis method not only improves the tolerance for single-frame misjudgment, but also significantly enhances the ability to continuously monitor the aircraft take-off and landing environment and landing gear status in a dynamic environment.

[0098] The two-stage detection process of the present invention includes an initial detection and a secondary refined detection; the initial detection is for the entire image, and if it is found in the initial detection that the number of landing gears does not meet the preset requirements, the secondary refined detection is started, a sub-image of the area where the aircraft is located is extracted and the sub-image is sent to a landing gear detection model, and finally the initial detection result is fused with the secondary detection result; if the fused result still shows that the number of landing gears does not meet the preset requirements, a landing gear abnormality alarm is issued, and the detection accuracy is high.

[0099] Specifically, the process of target intelligent tracking and searching is as follows:

[0100] The intelligent tracking module obtains the video stream detection results of the camera by the intelligent recognition module, and associates the detection results in multiple consecutive frames to achieve target tracking. When there is only one aircraft target, the intelligent tracking module controls the first camera to continuously track the target and keep the target in the center of the first camera's screen; when there are multiple aircraft targets, the system automatically selects one of them for continuous tracking according to the set rules.

[0101] In addition, the present invention handles the target selection problem in a multi-aircraft environment through a tracking strategy of a target filtering and management mechanism; when an aircraft being continuously tracked disappears from the screen temporarily or is temporarily blocked, resulting in a failure of continuous tracking, the aircraft will be searched again.

[0102] The specific implementation steps are as follows:

[0103] 1. Target tracking trajectory generation

[0104] First, the intelligent tracking module obtains the detection results of the video stream of the first camera by the intelligent recognition module, and associates the detection results in multiple consecutive frames to achieve target tracking. Each target will generate a tracking track and be assigned a unique tracking ID number.

[0105] 2. Initial target selection and filtering

[0106] When there is only one aircraft target, the tracking module controls the first camera to continuously track the target and keep the target in the center of the first camera's image; when there are multiple aircraft targets, the aircraft with the largest image area is selected as the main tracking target. This selection is based on the fact that the aircraft with the largest image area is more likely to be the main landing or taking-off aircraft visually, and its dynamics are more worthy of attention. For other aircraft targets except the main tracking target, they will be added to a special filter list.

[0107] 3. Target tracking and search

[0108] After the main tracking target is selected, the intelligent tracking module controls the first camera to continuously track the main tracking target and keep the main tracking target always in the center of the first camera's picture, so as to facilitate continuous observation and analysis of the take-off and landing process.

[0109] Specifically, the direction and focal length of the camera are automatically adjusted through a real-time control algorithm to ensure that the aircraft always remains in the center of the video frame and is of moderate size, so as to avoid affecting the recognition and tracking effect due to the aircraft being too large or too small. The horizontal and pitch angles of the camera will be dynamically adjusted according to the real-time position of the aircraft and the predicted motion trajectory. Specifically, the system will first obtain the specific position of the currently tracked aircraft in the image according to the results of the target detection algorithm, and calculate the offset of the position relative to the center of the image. The system will then adjust the horizontal and pitch angles of the camera according to the offset, so that the tracked aircraft is in the center of the image. For example, if the tracked aircraft is in the upper part of the image, the camera is controlled to rotate upward; if the tracked aircraft is in the left part of the image, the camera is controlled to rotate left. The focal length of the camera will be adjusted according to the proportion of the aircraft in the image. Specifically, the system will first obtain the proportion of the currently tracked aircraft in the image according to the results of the target detection algorithm. If the ratio is less than a certain set value, the camera focal length is increased to increase the proportion of the tracked aircraft in the image; if the ratio is greater than another set value, the camera focal length is reduced to reduce the proportion of the tracked aircraft in the image, thereby ensuring that the key features of the aircraft, such as the landing gear, are always clearly visible and that a good tracking effect is achieved.

[0110] If the main tracking target fails to be tracked due to reasons such as rapid movement or temporary occlusion during the tracking process, the system will immediately trigger the search mechanism to find the main tracking target that failed to be tracked. The system will exclude the aircraft that have been added to the filter list from the set of target trajectories that can be considered to improve the success rate of finding the main tracking target again. Because the filter list includes aircraft that were previously far away from the center of the screen or were judged as non-primary targets, these targets will not be considered as new tracking objects at this stage. When a new target is searched (the target is not in the filter list), the system will try to find the original main tracking target among the newly appeared targets. The newly appeared target is compared with the original main tracking target. The comparison content includes the proximity between the position of the original main tracking target when the tracking failed and the position when the new target just appeared, the similarity of the trajectory, etc. If the newly appeared target is identified as the original main tracking target, the intelligent tracking module controls the camera to continue tracking the newly appeared target; if the original tracking target is not found within a certain period of time, it is considered as a search failure, the tracking task is terminated, and the camera returns to the specified preset point again. With this approach, surveillance can be quickly restored even if the target is lost, reducing the duration of surveillance interruption and potential security risks.

[0111] 4. Filter list and track location dynamic updates

[0112] During the tracking process of the monitored aircraft, the filter list will be dynamically updated according to the real-time data of each frame. At the same time, the system will save and continuously update the position information of the currently tracked main aircraft, which is not only crucial for maintaining tracking, but also a key basis for reselecting the tracking target after the aircraft is lost. By updating this data in real time, the risk of mistracking is greatly reduced, and the efficiency and reliability of the entire monitoring system are improved.

[0113] The above method can significantly improve the stability of continuous tracking of the main tracking target.

[0114] Specifically, the specific process of aircraft deviation from the runway monitoring is as follows:

[0115] The cameras at both ends of the runway have auxiliary lines, which are divided into multiple focal segments and automatically change position. Each focal segment has auxiliary lines, including vertical auxiliary lines and horizontal auxiliary lines. The vertical auxiliary lines are used for yaw detection, and the horizontal auxiliary lines are used for height warning. They cannot be too high or too low.

[0116] Two points are used to monitor and warn whether the aircraft deviates from the runway during takeoff and landing. The first point is equipped with a first camera and a laser rangefinder, and the second point is equipped with a second camera.

[0117] The runway deviation calculation method for the first point is as follows Figure 3 As shown, the first camera and the laser rangefinder are installed at point C (the tower height is negligible), point A is the left end point of the runway centerline, and point B is the right end point of the runway centerline. The following takes the example of an aircraft landing from right to left to illustrate the runway deviation calculation method.

[0118] The distance from the first camera C to the point P on the runway centerline can be measured, assuming it is l1, that is, the length of PC is l1; similarly, the distance from the point P on the runway centerline to the right end point B of the runway centerline can also be measured, assuming it is l2, that is, the length of PB is l2. The actual position of the aircraft in the air is point O, and the vertical projection to the ground is point O'. The laser rangefinder can measure the straight-line distance d between the first camera C and the aircraft O, that is, the length of OC is d, the pitch angle of the first camera C is assumed to be α, that is, ∠OCO' is α, and the azimuth angle of the first camera C is set to β, that is, ∠O'CN is β, where CN is an imaginary straight line parallel to the runway, and O'N is an imaginary straight line perpendicular to the runway. Now it is necessary to solve the yaw angle θ of the aircraft relative to the runway centerline, that is, to solve ∠O'BM, and the deviation distance value O'M.

[0119] The distance O'C from the aircraft projection point O' to the tower camera C can be solved by the trigonometric function relationship:

[0120] O′C=dcosα

[0121] The distances O'N and CN can be solved by O'C and the azimuth angle β of the first camera C:

[0122] O′N=O′Csinβ=d cosαsinβ

[0123] CN=O′Ccosβ=d cosαcosβ

[0124] Now we know that the lengths of PC and PB are l1 and l2, so we can solve the lengths of O'M and BM:

[0125] O′M=O′N-MN=O′N-PC=dcosαsinβ-l1

[0126] BM=PM-PB=CN-PB=dcosαcosβ-l2

[0127] O'M is the distance the aircraft deviates from the centerline of the runway. If the lengths of O'M and BM are known, the aircraft yaw angle θ can be solved by trigonometric functions:

[0128]

[0129] If the aircraft lands from the left, the yaw angle θ can be calculated according to the above process.

[0130] In another specific embodiment, Figure 4 As shown, the yaw angle can also be estimated by two second cameras deployed at the left and right ends of the runway. The second cameras are respectively located on the extension lines of the center axis at both ends of the runway, and their pitch angles are adjusted to ensure that the landing process of the aircraft can be fully observed, and their azimuth angles are adjusted to ensure that their orientation is parallel to the center line of the runway. Figure 4 In the figure, rectangle ABCD is the monitoring screen of the second camera, and a reference line is drawn on the monitoring screen to assist in judging whether the aircraft has deviated from the runway, which is represented by MM' in the figure. It is known that the installation position of the second camera is point O, the image resolution of the camera is W×H, and the horizontal field of view angle is FOV H , the intelligent detection module can detect the aircraft target and determine its center position in the image. Assume that the coordinates of the center of the aircraft in the image are (x, y), where x and y are pixel coordinates. The yaw angle is defined as the angle θ between the line connecting the center point of the aircraft target and the second camera deployment point O and the center line of the runway. According to the geometric relationship:

[0131]

[0132] The pixel width of the image is W, and the horizontal field of view of the camera is FOV. H All are known quantities. The center coordinate x of the aircraft can be obtained through the image algorithm, so the estimated yaw angle is:

[0133]

[0134] The present invention uses multiple sensors to monitor and warn whether an aircraft deviates from the runway from multiple angles. On the one hand, the first camera and laser rangefinder deployed at the first point are used to measure and calculate the three-dimensional coordinate information of the aircraft in the air, and further calculate the angle and distance of the aircraft deviating from the runway centerline; on the other hand, the second camera deployed at the second point is used to measure and calculate the angle of the aircraft deviating from the runway centerline; if the angle or distance of the aircraft deviating from the runway centerline exceeds a certain threshold, a runway deviation alarm is issued.

[0135] In addition, to ensure the efficiency and accuracy of the aircraft take-off and landing monitoring system in data processing and analysis, the present invention records and analyzes the images and sensor data such as the landing gear status, runway deviation angle, aircraft height from the ground, laser rangefinder, etc. during the aircraft take-off and landing process in real time. When the historical take-off and landing data accumulates to a certain amount, it will be used as one of the bases for judging whether the subsequent aircraft take-off and landing process is normal.

[0136] Specifically, the sensor data in the historical take-off and landing process are recorded and analyzed to obtain the state that each sensor data should present in the normal take-off and landing state, which is recorded as the normal state. During the take-off and landing process of the aircraft, the real-time take-off and landing data of the aircraft measured by each sensor is compared with the data in the normal state. If the difference between the two is within the set threshold range, it means that the aircraft is in a normal take-off and landing state; if the difference between the two is outside the set threshold range, it means that the aircraft is likely to be in an abnormal take-off and landing state, and there is a risk of take-off and landing, and an alarm needs to be issued. For example, during normal take-off and landing, the runway deviation angle allowed by the aircraft is within the range of ±3°. When the horizontal angle of the aircraft detected by the camera in real time deviates from this range, the system will be judged as an abnormal state, indicating a possible take-off and landing risk; in addition, during normal take-off and landing, the distance between the aircraft and the ground presents a steady decreasing trend. Assuming that the rate of distance change is 40m / s~50m / s, if the distance change rate detected by the laser rangefinder obviously deviates from this range, it means that there may be an abnormal situation and an alarm needs to be issued.

[0137] The intelligent monitoring method for the aircraft take-off and landing process of the present invention can monitor the aircraft take-off and landing process from multiple angles, automatically identify the aircraft and its landing gear, and judge whether the landing gear is in a normal state by automatically counting the number of landing gears. After identifying the aircraft, the whole take-off and landing process of the aircraft can be automatically tracked, and the accurate position of the aircraft can be calculated in real time by the distance, azimuth and pitch angle of the aircraft relative to the monitoring equipment, and whether it deviates from the runway and calculates the angle and distance of the deviation; during the take-off and landing process of the aircraft, if the number of landing gears is abnormal or the aircraft deviates from the runway, the system will issue an alarm in time.

[0138] The intelligent monitoring method and system for aircraft takeoff and landing of the present invention, after detecting the aircraft, the system analyzes the movement trajectory of the aircraft and monitors the status of the landing gear through a deep learning algorithm. When the aircraft is preparing to land, the system will also analyze the relative position of the aircraft and the centerline of the channel to ensure that the aircraft is accurately aligned with the runway when landing. If there is any deviation, the system will issue a warning in real time to facilitate the pilot to adjust the course in time; finally, the monitoring data will be transmitted to the control center in real time and processed by the intelligent analysis module to quickly identify potential safety risks. Through this series of precise monitoring processes, the present invention can effectively improve the safety of aircraft takeoff and landing, while also improving the efficiency and reliability of airport operations.

[0139] The present invention significantly improves the recognition accuracy of each state of the aircraft by fusing camera views from multiple different positions and applying deep learning algorithms for data fusion and analysis. This method can provide comprehensive view coverage at the critical moments of takeoff and landing, ensuring the capture and analysis of key information during takeoff and landing.

[0140] The present invention uses a multi-camera system and triangulation in combination with real-time data streams to accurately measure and process in real time the deviation of the aircraft from the runway centerline. By deploying high-resolution cameras at both ends of the runway and using tower cameras and laser rangefinders to work synchronously, not only can the position of the aircraft be captured in real time, but the yaw angle can also be accurately calculated through built-in sensors. This method enables the system to respond quickly in emergency situations and issue warnings in a timely manner and provide decision support, significantly enhancing the safety and efficiency of the entire aircraft take-off and landing process.

[0141] The present invention comprehensively monitors the aircraft status through multi-viewing angles and multi-sensors, and realizes comprehensive monitoring of all postures of the aircraft through the integration of multi-viewing angle cameras and laser rangefinders. Cameras deployed at multiple points are used to perform multi-viewing angle monitoring of the aircraft's operating status. The deployment points of the cameras include positions close to the tower on both sides of the runway, and the centerline positions at both ends of the runway. A laser rangefinder is deployed at the same point as the camera on one side of the runway to accurately measure the distance to the aircraft. Combined with the azimuth and pitch angle information given by the camera, the three-dimensional coordinates of the aircraft are accurately calculated, thereby accurately calculating the yaw angle and deviation distance, increasing the stability and reliability of the system, and realizing more accurate and real-time flight safety monitoring.

[0142] This invention is funded by the Hunan Science and Technology Innovation Program, Project No. 2023RC3230.

[0143] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. An intelligent monitoring method for aircraft take-off and landing process, characterized in that: Includes steps: Obtain the video stream of the aircraft take-off and landing direction, and identify the aircraft in the video stream; when the aircraft is identified, continue to track the target aircraft; In the process of continuously tracking the target aircraft, the number of the aircraft landing gears is identified, and an abnormal alarm is issued when the number of the landing gears is abnormal; And during the continuous tracking process, the flight trajectory of the aircraft is monitored to determine whether it deviates from the runway, and a runway deviation warning is issued when it deviates from the runway.

2. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, characterized in that: The specific process of identifying the aircraft in the video stream is as follows: Phase 1: Preliminary detection and region of interest extraction Perform global detection on the entire image in the video stream, identify and locate the aircraft in the image and perform preliminary identification of the landing gear to obtain the detection result, and then judge the detection result; if the detection result includes the aircraft, calculate the number of landing gears in the detection result; if the number of landing gears meets the predetermined standard, the current detection ends and the subsequent second stage detection is no longer performed; otherwise, extract an area of ​​interest larger than the area where the aircraft is located with the aircraft as the center for the second stage detection; Phase 2: Secondary refined detection and fusion The image of the region of interest extracted in the first stage is sent to the preset landing gear recognition model to further identify the landing gear and obtain the detection result; After obtaining the second stage test results, the first stage landing gear test results are merged with the second stage landing gear test results; Phase 3: Detection result output and alarm The fusion results of the previous two stages are output and alarms are issued.

3. The intelligent monitoring method for aircraft take-off and landing process according to claim 2 is characterized in that: In the second stage, the intersection-over-union (IoU) calculation method is used to evaluate the degree of overlap between the detection boxes in the two detection results. When the IoU value exceeds the set threshold, it is considered that the two detection boxes point to the same actual object, and the detection results with lower confidence are removed.

4. The intelligent monitoring method for aircraft take-off and landing process according to claim 2 or 3, characterized in that: In the third stage, the detection results in multiple consecutive frames in the video stream are continuously analyzed. If the landing gear is continuously detected in these multiple frames and fails to meet the preset detection standards, or if flying birds, balloons or floating objects in the air are continuously detected, it is considered that there is a potential landing risk and the alarm mechanism is automatically triggered.

5. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: The process of continuous tracking of the target aircraft is as follows: Target tracking trajectory generation: Obtain the video stream detection results and associate the detection results in multiple consecutive frames to achieve target tracking; Initial target selection: When there is only one aircraft target, the target will be continuously tracked and kept in the center of the image; when there are multiple aircraft targets, the aircraft occupying the largest image area will be selected as the main tracking target; other aircraft targets except the main tracking target will be added to the filter list; Target tracking: After the main tracking target is selected, it will be continuously tracked and kept in the center of the screen at all times; Dynamic update of tracking position: During the tracking process of the monitoring aircraft, the position information of the current tracking target is saved and continuously updated.

6. The intelligent monitoring method for aircraft take-off and landing process according to claim 5 is characterized in that: During the initial target selection process, other aircraft targets besides the primary tracking target are added to the filter list; During the target tracking process, if the main tracking target fails to be tracked, the search mechanism is triggered to retrieve the main tracking target that failed to be tracked; the aircraft that has been added to the filter list is excluded from the set of target trajectories that can be considered; When searching for a new target, if the target is not in the filter list, it will try to find the original tracking target among the new targets; Compare the newly appeared target with the original tracking target, including the proximity between the position of the original tracking target when tracking fails and the position of the new target when it first appears, and the similarity of their trajectories; If the newly appeared target is identified as the original tracking target, the newly appeared target will be continuously tracked.

7. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: The specific process of monitoring the flight trajectory of an aircraft to determine whether it deviates from the runway is as follows: The first camera and the laser rangefinder are installed at point C on one side of the runway, point A is the left end point of the runway centerline, and point B is the right end point of the runway centerline; The distance from point C to point P on the centerline of the runway is I1, that is, the length of PC is I1; similarly, the distance from point P on the runway centerline to the right end point B of the runway centerline is I2, that is, the length of PB is I2; the actual position of the aircraft in the air is point O, and the vertical projection to the ground is point O′; the laser rangefinder measures the straight-line distance d between point C and aircraft O, that is, the length of OC is d, the pitch angle of the first camera C is assumed to be a, that is, ∠OCO′ is α, and the azimuth angle of the first camera C is set to β, that is, ∠O′CN is β, where CN is an imaginary straight line parallel to the runway, and O′N is an imaginary straight line perpendicular to the runway; The specific process of solving the yaw angle θ of the aircraft relative to the centerline of the runway, that is, solving ∠O′BM, and the deviation distance value O′M is: The distance O′C from the aircraft projection point O′ to the tower camera C is solved by the trigonometric function relationship: O′C=dcosα Solve the distance between O'N and CN by the azimuth angle β of O'C and point C: O′N=O′Csinβ=dcosαsinβ CN=O′Ccosβ=dcosαcosβ Given that the lengths of PC and PB are I1 and I2, find the lengths of O′M and BM: O′M=O′N-MN=O′N-PC=dcosαsinβ-l1 BM=PM-PB=CN-PB=dcosαcosα-l2 O′M is the distance the aircraft deviates from the runway centerline. Knowing the lengths of O′M and BM, the aircraft yaw angle θ can be solved by trigonometric functions:

8. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: The specific process of monitoring the flight trajectory of an aircraft to determine whether it deviates from the runway is as follows: The yaw angle is estimated by two second cameras deployed at the left and right ends of the runway; the second cameras are located on the extension lines of the center axis at both ends of the runway, and their pitch angles are adjusted to ensure that the landing process of the aircraft can be fully observed, and their azimuth angles are adjusted to ensure that their orientation is parallel to the center line of the runway; Rectangle ABCD is the monitoring screen of the second camera, and a reference line is drawn on the monitoring screen to assist in judging whether the aircraft deviates from the runway, represented by MM′; it is known that the installation position of the second camera is point 0, the image resolution of the camera is W×H, and the horizontal field of view angle is FOV H ; Detect the aircraft target and determine its center position in the image. Assume that the coordinates of the center of the aircraft in the image are (x, y), where x and y are both pixel coordinates; define the yaw angle as the angle θ between the line connecting the center point of the aircraft target and the second camera deployment point O and the center line of the runway. According to the geometric relationship: The pixel width of the image is W, and the horizontal field of view of the camera is FOV. H All are known quantities. The center coordinate x of the aircraft is obtained through the image algorithm, so the estimated yaw angle is:

9. The intelligent monitoring method for aircraft take-off and landing process according to claim 1, 2 or 3, characterized in that: Record and analyze the data of each sensor during the historical take-off and landing process, and derive the state that each sensor data should present under the normal take-off and landing state, which is recorded as the normal state; During the aircraft take-off and landing process, the real-time take-off and landing data of the aircraft measured by each sensor is compared with the data under normal conditions. If the difference between the two is within the set threshold range, it means that the aircraft is in a normal take-off and landing state; if the difference between the two is outside the set threshold range, it means that the aircraft is likely to be in an abnormal take-off and landing state, there is a take-off and landing risk, and an alarm needs to be issued.

10. An intelligent monitoring system for aircraft take-off and landing, characterized in that: It includes video monitoring module, distance measurement module, intelligent identification module, intelligent tracking module, runway deviation monitoring module, automatic alarm module, display and human-computer interaction module; The video monitoring module consists of multiple cameras, which are used to track and shoot the entire process of aircraft take-off and landing; The distance measurement module is installed at the same point as a camera to measure the distance between the aircraft and the monitoring equipment. The camera at this point can give the azimuth and pitch angle of the aircraft relative to the camera; An intelligent recognition module, which is used to obtain the real-time video stream of the camera and identify the aircraft and its landing gear in the image; The intelligent tracking module is used to control the rotation and zoom of the camera according to the position of the aircraft in the camera image given by the intelligent recognition module, so that the aircraft is always in the center of the camera image with an appropriate size; The runway deviation monitoring module is used to determine whether the aircraft deviates from the runway, as well as the angle and distance of deviation based on the aircraft position information provided by the camera and the laser rangefinder; Automatic warning module, used to automatically send out an alarm when the aircraft deviates from the runway or the number of landing gear is abnormal, so as to remind the tower staff to take corresponding measures; The display and human-computer interaction module is used to display the video images of the camera and realize human-computer interaction to facilitate manual intervention.

Images