Runway intrusion prevention method based on multi-source heterogeneous data fusion processing

Abstract

The invention discloses a runway intrusion prevention method based on multi-source heterogeneous data fusion processing, and the method comprises the steps: obtaining video stream data collected by a panoramic infrared camera in a ground protection area range, and forming cooperative target monitoring data and non-cooperative target monitoring data; performing consistency comparison on the formed control intention data based on the voice recognition and the collected control intention data based on the electronic process sheet to form definite control intention data; obtaining runway state data; generating runway intrusion early warning or alarm; and sending the target monitoring data and the runway intrusion early warning or alarm to a runway intrusion prevention related party in real time to realize situation monitoring and runway intrusion early warning. According to the method, the target monitoring data, the control intention data and the runway state data are combined for the runway intrusion risk, and alarm logic judgment processing is fused, so that compared with alarm processing by only depending on one type of data, the data is more comprehensive, mutual cross fusion authentication is realized, and the alarm is more comprehensive, timely, reliable and accurate.

Description

Technical Field

[0001] The invention belongs to the technical field of airport scene management, and in particular relates to a runway intrusion prevention method based on multi-source heterogeneous data fusion processing. Background Art

[0002] Airport runway intrusion seriously affects the safety of people's lives and property, and is the focus of airport operation risk prevention and control. In recent years, with the continuous growth in the number of civil aviation flights, the complexity of airport operations has increased, and the number of incidents due to aircraft, vehicles, and personnel intruding into runways has increased dramatically.

[0003] At present, there are many methods and systems for preventing runway intrusions. For example, unilateral target intrusion calculation and alarm are performed based on traditional surveillance target location data (MLAT, ADS-B, SMR), video target identification location data, and flight status data. No data cross-verification is performed, and no data fusion processing and alarm calculation are performed. There are problems such as incomplete, untimely, unreliable, and inaccurate prevention and control. Moreover, the main prevention and control targets are aircraft and surveillance cooperation vehicles. No prevention and control is carried out on non-cooperative vehicles, personnel and other targets, and no warning information is shared and sent to vehicles and personnel to remind them to evacuate in time to avoid runway intrusion incidents. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a runway intrusion prevention method based on multi-source heterogeneous data fusion processing, so as to solve the problems existing in the existing runway intrusion prevention technology, such as incomplete, untimely, unreliable, inaccurate prevention and control, incomplete prevention and control objects, unshared information, and untimely reminders.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] The present invention provides a runway intrusion prevention method based on multi-source heterogeneous data fusion processing, comprising the following steps:

[0007] 1) Obtaining video stream data collected by a panoramic infrared camera over the ground protected area, performing video recognition to generate target recognition data; and fusing the received traditional surveillance data with the target recognition data to form cooperative target surveillance data and non-cooperative target surveillance data;

[0008] 2) Collect the voice commands of the tower controller to conduct control voice recognition and repetition consistency check, form control intention data based on voice recognition, and compare it with the control intention data collected based on the electronic process sheet for consistency, to form confirmed control intention data;

[0009] 3) Obtain runway status data;

[0010] 4) Using the target monitoring data, confirmed control intention data, and runway status data generated in step 1), step 2), and step 3), a data combination logic judgment process is performed on various runway incursion scenarios to generate a runway incursion warning or alarm;

[0011] 5) The target monitoring data in step 1) and the runway intrusion warning or alarm in step 4) are sent in real time to the relevant parties of runway intrusion prevention to realize situation monitoring and runway intrusion warning.

[0012] Furthermore, the step 1) specifically includes the following steps:

[0013] 11) Obtain the video stream data collected by the panoramic infrared camera in the ground protection area, use the depth-guided monocular 3D video detection method to perform target recognition on the video stream data, and obtain the target identification, target type, target geometric center position coordinates, and target contour coordinates;

[0014] 12) Receive the traditional monitoring data output by the monitoring device, and fuse it with the video target recognition data in step 11) to form cooperative target monitoring data and non-cooperative target monitoring data.

[0015] Furthermore, the panoramic infrared camera is composed of a plurality of infrared cameras arranged and combined into one, and can cover the ground protection area to which the runway belongs.

[0016] Furthermore, the ground protection area includes the airport runway, the portion of the taxiway between the applicable runway holding position and the actual runway, the soil surface area within 75m on both sides of the runway centerline, the ILS sensitive area, the ILS critical area and the runway end safety area.

[0017] Furthermore, the targets include aircraft, vehicles, personnel, and animals; a cooperative target refers to a target that, when the monitoring device detects a target, requires the target's cooperation in sending information in order to achieve target detection; a non-cooperative target refers to a target that, when the monitoring device detects a target, does not require the target's cooperation in sending information in order to achieve target detection.

[0018] Furthermore, the target contour coordinates are the head position coordinates, tail position coordinates, left wingtip position coordinates, and right wingtip position coordinates of the aircraft or vehicle target.

[0019] Furthermore, the traditional surveillance data is surveillance data of aircraft and cooperative vehicles detected using surveillance equipment MLAT, ADS-B or SMR, wherein cooperative vehicles are vehicles equipped with ADS-B transmitters; the traditional surveillance data does not include non-cooperative vehicles, personnel, and animals.

[0020] Furthermore, the cooperative target and non-cooperative target monitoring data include airborne aircraft monitoring data, as well as aircraft, vehicles, personnel, and animals on the airport surface.

[0021] Furthermore, the aerial aircraft monitoring data includes target identification, type, location coordinates, altitude, heading, and speed.

[0022] Furthermore, the airport scene aircraft and vehicle monitoring data include target identification, type, target geometric center position coordinates, target contour coordinate data, heading, and speed.

[0023] Furthermore, the airport personnel and animal monitoring data includes target identification, type, target coordinates, heading, and speed.

[0024] Furthermore, the method of using a depth-guided monocular 3D video detection method to perform target recognition on video stream data specifically includes the following steps:

[0025] 111) Inputting the video stream data collected from the ground protection area by the panoramic infrared camera into the feature extraction network to extract the multi-scale feature map of the image;

[0026] 112) inputting the multi-scale feature map extracted in step 111) into a visual coding network to complete visual coding of the multi-scale feature map;

[0027] 113) inputting the multi-scale feature map extracted in step 111) into a deep feature extraction network to obtain a deep feature map of the image;

[0028] 114) inputting the depth feature map in step 113) into the depth coding network to complete the depth coding of the depth feature map;

[0029] 115) The visual encoding of the multi-scale feature map in step 112) and the depth encoding of the depth feature map in step 114) are input into a depth-guided decoding network to calculate the coordinates of the geometric center position of the target in the output image.

[0030] Furthermore, the visual coding network in step 112) is a DETR image coding network trained with image data.

[0031] Furthermore, the deep feature extraction network in step 113) is two 3*3 convolutional layers trained with deep data.

[0032] Furthermore, the deep coding network in step 114) is a DETR image coding network trained with deep data.

[0033] Further, the step 115) specifically includes the following steps:

[0034] 1151) Initialize multiple sequences queries denoted as q;

[0035] 1152) The initialized sequence q and the deep encoding of the deep feature map in step 114) are input into the deep decoding network, and the output result is the feature vector q 1 ;

[0036] 1153) The output result q of step 1152) is 1 The visual encoding of the multi-scale feature map in step 112) is input into the visual decoding network together, and the output result is the feature vector q 2 ;

[0037] 1154) The output result q of step 1153) is 2 The input is a labeled and trained feedforward neural network, and the output feature vector q′, q′ contains the coordinates of the target geometric center position.

[0038] Furthermore, the deep decoding network in step 1152) is a DETR image decoding network trained with deep data.

[0039] Furthermore, the visual decoding network in step 1152) is a DETR image decoding network trained with image data.

[0040] Furthermore, the step 12) specifically includes:

[0041] 121) clustering the target geometric center position coordinates and the position coordinates in the traditional track data obtained by the video recognition detection in step 11) using the nearest neighbor matching algorithm according to the target type to form an association graph of each target data containing multiple clusters;

[0042] 122) analyzing the data in each target data association graph; when the target data association graph contains the target geometric center position coordinates obtained by video recognition detection and the position coordinates in the traditional track data, the target geometric center position coordinates obtained by video recognition detection are selected as the target position coordinate data of the fused track; when the target data association graph contains only the position coordinates in the traditional track data, the position coordinates are selected as the target position coordinate data of the fused track;

[0043] 123) According to the target position coordinate data selected in step 122), the monitoring data of the corresponding target is associated to form cooperative target monitoring data and non-cooperative target monitoring data.

[0044] Furthermore, the step 2) specifically includes the following steps:

[0045] 21) Collect voice commands from tower controllers, pilots and vehicle drivers, perform control voice recognition and repetition consistency check, and form 6 key structured control command text data, namely, enter the runway, cross the runway, take off, land, blind landing is allowed, and vehicles can enter the runway, which are control intention data based on voice recognition;

[0046] 22) Collect flight status data modified by tower controllers in the electronic progress sheet system, including four types of flight status data, namely, runway entry, runway crossing, take-off, and landing, which are control intention data based on the electronic progress sheet;

[0047] 23) Compare the control intention data based on voice recognition in step 21) with the control intention data based on the electronic process sheet in step 22) for consistency to form confirmed control intention data.

[0048] Furthermore, the step 21) specifically includes the following steps:

[0049] 211) Collect audio data from the control voice recorder of the tower control unit and convert analog voice signals into digital voice signals;

[0050] 212) Preprocessing the digital voice signal by noise reduction and audio segmentation, and using a deep learning speech recognition method based on a neural network to recognize the preprocessed voice signal as an unstructured control instruction text;

[0051] 213) By establishing a control command voice corpus and a digital voice rule library, the control command intention is structured extracted to form structured control command text data, including five key structured control command text data for aircraft command: entering the runway, crossing the runway, taking off, landing, and blind landing, as well as key structured control command text data for vehicles that plan to enter the runway for operations;

[0052] 214) Compare the tower controller's instruction text with the pilot / vehicle driver's readback text to determine whether the two are consistent and give a readback consistency mark;

[0053] 215) Output the recognized voice text data, including: seat name, voice channel, up / down instruction mark, voice start time, voice end time, voice duration, Chinese / English mark, original voice text, structured control instruction text, target mark, and repetition consistency mark.

[0054] Furthermore, the repeat consistency mark includes a repeat inconsistent mark and a repeat consistent mark; when it is a repeat inconsistent mark, an alarm prompt is issued to remind the tower controller to find the cause and deal with it, and the method ends; when it is a repeat consistent mark, step 23 is executed.

[0055] Furthermore, the step 23) specifically includes the following steps:

[0056] 231) The control intention data based on speech recognition obtained in step 21) is compared with the control intention data based on the single process list in step 22) for consistency, and the comparison check data includes entering the runway, crossing the runway, taking off, and landing; when there is inconsistency, an alarm prompt is issued to remind the tower controller to find the cause and deal with it, and the method ends; when there is consistency, step 232) is executed;

[0057] 232) The four types of control intention data of entering the runway, crossing the runway, taking off and landing, and the two types of control intention data of allowing blind landing and vehicles being allowed to enter the runway that have been compared and found to be consistent are used as confirmed control intention data and output.

[0058] Furthermore, the runway status data in step 3) includes the runway status data input by the tower controller and the runway status data input by the electronic process sheet system.

[0059] Further, the runway status data in step 3) specifically includes: runway open status, runway closed status, runway inspection status, and third-party occupied status;

[0060] The runway open state means that the current runway is in normal operation, allowing aircraft to take off and land on the runway;

[0061] Runway closure status means that the runway is not suitable for aircraft takeoff and landing due to bad weather, navigation equipment failure or other events, and the runway is announced to be closed;

[0062] The runway inspection status refers to the status where the airport runway temporarily requires airport maintenance vehicles / personnel to enter the runway for runway inspection.

[0063] Furthermore, the runway intrusion scenarios in step 4) include: for ground protection zone intrusion, runway occupancy conflict, and runway use error, the target monitoring data, confirmed control intention data, and runway status data formed in steps 1), 2), and 3) are used to perform data combination logic judgment processing to generate a runway intrusion warning or alarm.

[0064] Furthermore, in step 4), for the ground protection zone intrusion scenario, several ground protection zone intrusion sub-scenarios are established, and the target monitoring data, confirmed control intention data, and runway status data formed in steps 1), 2), and 3) are used to perform data combination logic judgment processing, specifically including the following:

[0065] 411) When the runway status is closed, a runway incursion warning is generated when the incoming aircraft is detected to be about to intrude into the ground protection zone based on the target surveillance data in step 1), or when the aircraft's confirmed control intention data in step 2) indicates landing;

[0066] 412) When the runway status is closed, a runway incursion warning or alarm is generated when the departing aircraft is about to or has intruded into the ground protection area based on the target surveillance data in step 1), or when the aircraft's confirmed control intention data in step 2) indicates entering the runway;

[0067] 413) The runway status is check. When the incoming aircraft is detected to be about to intrude into the ground protection zone based on the target surveillance data in step 1), or when the aircraft's confirmed control intention data in step 2) indicates that it can land blindly, a runway intrusion warning is generated;

[0068] 414) the runway status is check, and based on the target monitoring data in step 1) and the confirmed control intention data in step 2), when any target other than the vehicle that can enter the runway in the confirmed control intention data intrudes into the ground protection zone, a runway intrusion alarm is generated;

[0069] 415) the runway status is open, and when a vehicle, a person, or an animal is about to or has intruded into the ground protection zone based on the target monitoring data in step 1), a runway intrusion warning or alarm is generated;

[0070] 416) The runway status is open. When it is detected based on the target monitoring data in step 1) and the confirmed control intention data in step 2) that the confirmed control intention data is not to enter the runway or cross the runway, the aircraft intrudes into the ground protection area and a runway intrusion alarm is generated.

[0071] Furthermore, the runway occupancy conflict scenario in step 4) is specifically as follows: when the runway status is open, based on the target monitoring data in step 1) and the confirmed control intention data in step 2), when it is detected that two targets are about to or have used the same runway at the same time, it is judged as a runway occupancy conflict.

[0072] Furthermore, for the runway occupancy conflict scenario, several runway occupancy conflict sub-scenarios are established, and the target monitoring data, confirmed control intention data, and runway status data formed in step 1), step 2), and step 3) are used to perform data combination logic judgment processing, specifically including:

[0073] 421) Based on the target monitoring data in step 1), when an incoming aircraft is about to land on a runway, if there are departing aircraft, aircraft crossing the runway, vehicles, personnel or animals within the ground protection area to which the runway belongs, a runway occupancy conflict alarm is generated;

[0074] 422) Based on the target monitoring data in step 1) and the confirmed control intention data in step 2), when the arriving aircraft is about to land on the runway or the confirmed control intention data is landing, the departing aircraft using the runway confirms the control intention data is entering the runway, or the control intention data of the runway-crossing aircraft using the runway is crossing the runway, a runway occupancy conflict alarm is generated;

[0075] 423) Based on the target monitoring data in step 1), when it is detected that the departing aircraft is in the ground protection zone, and there are vehicles, personnel, animals or aircraft crossing the runway on which the arriving aircraft is about to land or within the ground protection zone, a runway occupancy conflict alarm is generated;

[0076] 424) Based on the target monitoring data in step 1) and the confirmed control intention data in step 2), it is detected that when the departing aircraft is in the ground protection area or the confirmed control intention data is to enter the runway and take off, the confirmed control intention data of the arriving aircraft using the runway is to land, or the confirmed control intention data of the runway crossing aircraft using the runway is to cross the runway, a runway occupancy conflict alarm is generated.

[0077] Furthermore, in step 4), for the runway error scenario, several runway error sub-scenarios are established, and the target monitoring data, confirmed control intention data, and runway status data formed in steps 1), 2), and 3) are used to perform data combination logic judgment processing, specifically including:

[0078] When the runway number used by the aircraft detected based on the target monitoring data in step 1) is inconsistent with the runway number that the aircraft plans to use in the confirmed control intention data in step 2), a runway occupancy conflict alarm is generated;

[0079] When it is detected based on the target monitoring data in step 1) that the runway used by the aircraft is inconsistent with the runway number assigned to the aircraft in the flight plan data, a runway occupancy conflict alarm is generated;

[0080] The flight plan data is the flight plan data formed by the tower automation system receiving the aviation flight dynamic telegram (AFTN) and processing;

[0081] When the runway number that the aircraft plans to use in the control intention data is confirmed to be inconsistent with the runway number assigned to the aircraft in the flight plan data in step 2), a runway occupancy conflict alarm is generated.

[0082] Furthermore, the step 5) specifically includes: sending the target monitoring data of step 1) and the runway intrusion warning and alarm of step 4) in real time to the relevant parties of runway intrusion prevention, realizing situation monitoring and runway intrusion warning sharing, and using terminal display, alarm sound, alarm light or vibration to remind pilots, vehicle drivers, field personnel, controllers, airport managers to deal with the runway intrusion incidents that are about to or have occurred.

[0083] Furthermore, the parties involved in preventing runway intrusion in step 5) include: aircraft cockpit, vehicle side, personnel side, control tower, and airport flight area management department.

[0084] The present invention targets various runway intrusion scenarios, uses comprehensive target monitoring data formed by the fusion processing of panoramic infrared video recognition and traditional monitoring data, confirmed control intention data formed by the comparison and verification of control voice recognition and electronic process single flight status data, and collected runway status data, and performs fusion logic judgment processing to generate runway intrusion warnings and alarms, and shares the monitoring data and warnings and alarms in real time to the aircraft cockpit, vehicle end, personnel end, tower, airport flight area management department, and airport operation control department, so that all parties related to runway safety can grasp the comprehensive operation status "one picture" and comprehensive, timely, reliable and accurate warning information, establish a unified situational awareness, support the joint and timely prevention and control and disposal of runway intrusion incidents caused by aircraft, vehicles, personnel, animals (including cooperative and non-cooperative targets) by all parties, reduce the probability and severity of runway intrusions, and improve airport operation safety.

[0085] Beneficial effects of the present invention:

[0086] The target data formed by the fusion processing of target data identified by panoramic infrared video and traditional surveillance data includes cooperative and non-cooperative aircraft, vehicles, personnel, and animal targets. Aircraft and vehicles also have target contour data. It is superior to the current traditional surveillance method, solves the problems of incomplete, inaccurate and unreliable scene target monitoring, and provides a high-quality surveillance data basis for subsequent alarm calculations.

[0087] In the present invention, the control intention data of the control voice recognition is compared with the control intention data of the electronic process sheet to form confirmed control intention data. Compared with the previous alarm calculation based solely on control voice data or flight status data, which is prone to missed alarms and false alarms, the method of the present invention is more credible and reliable, providing a highly credible data basis for subsequent alarm calculations.

[0088] The present invention combines target monitoring data, control intent data, and runway status data to carry out integrated alarm logic judgment processing for various runway intrusion risks. Compared with the previous alarm processing relying on only one type of data, the data is more comprehensive and cross-fused and authenticated, and the alarm is more comprehensive, timely, reliable, and accurate.

[0089] The present invention realizes comprehensive, timely, reliable and accurate early warning / alarm of runway intrusion by aircraft, vehicles, personnel and animals (including cooperative and non-cooperative targets) in various runway intrusion scenarios, and shares situation and early warning / alarm information with pilots, vehicle drivers, field personnel, controllers, airport managers and other parties related to runway safety in real time, thereby reducing the probability and severity of runway intrusion and improving airport operation safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 The schematic diagram of the method of the present invention is shown in FIG.

[0091] Figure 2 The figure is a flow chart of the monocular 3D video detection method based on depth guidance.

[0092] Figure 3 Schematic diagram of using a depth-guided decoding network to calculate the coordinates of the geometric center of the target in the output image. DETAILED DESCRIPTION

[0093] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0094] Reference Figure 1 As shown, a runway intrusion prevention method based on multi-source heterogeneous data fusion processing of the present invention comprises the following steps:

[0095] 1) Obtaining video stream data collected by a panoramic infrared camera in the ground protection area, performing video recognition to generate target recognition data; and fusing the received traditional surveillance data with the target recognition data to form cooperative target surveillance data and non-cooperative target surveillance data; specifically comprising the following steps:

[0096] 11) Obtain the video stream data collected by the panoramic infrared camera in the ground protection area, use the depth-guided monocular 3D video detection method to perform target recognition on the video stream data, and obtain the target identification, target type, target geometric center position coordinates, and target contour coordinates;

[0097] 12) Receive the traditional monitoring data output by the monitoring device, and fuse it with the video target recognition data in step 11) to form cooperative target monitoring data and non-cooperative target monitoring data.

[0098] The panoramic infrared camera is composed of a plurality of infrared cameras arranged and combined into one, and can cover the ground protection area of ​​the runway.

[0099] The ground protection area includes the airport runway, the part of the taxiway between the applicable runway holding position and the actual runway, the soil surface area within 75m on both sides of the runway centerline, the ILS sensitive area, the ILS critical area and the runway end safety area.

[0100] Among them, the targets include aircraft, vehicles, personnel, and animals; cooperative targets refer to targets that require the cooperation of the target in sending information when the monitoring equipment detects the target; non-cooperative targets refer to targets that do not require the cooperation of the target in sending information when the monitoring equipment detects the target.

[0101] The target contour coordinates are the head position coordinates, tail position coordinates, left wingtip position coordinates, and right wingtip position coordinates of the aircraft or vehicle target.

[0102] Specifically, the traditional surveillance data is the surveillance data of aircraft and cooperative vehicles detected using the surveillance equipment MLAT, ADS-B or SMR, where the cooperative vehicles are vehicles equipped with ADS-B transmitters; the traditional surveillance data does not include non-cooperative vehicles, personnel, and animals.

[0103] Specifically, the cooperative target and non-cooperative target monitoring data include airborne aircraft monitoring data, as well as aircraft, vehicles, personnel, and animals on the airport surface.

[0104] Specifically, the aerial aircraft monitoring data includes target identification, type, location coordinates, altitude, heading, and speed.

[0105] Specifically, the airport scene aircraft and vehicle monitoring data include target identification, type, target geometric center position coordinates, target contour coordinate data, heading, and speed.

[0106] Specifically, the airport field personnel and animal monitoring data includes target identification, type, target coordinates, heading, and speed.

[0107] Reference Figure 2 As shown, the monocular 3D video detection method based on depth guidance is used to perform target recognition on video stream data, which specifically includes the following steps:

[0108] 111) Inputting the video stream data collected from the ground protection area by the panoramic infrared camera into the feature extraction network to extract the multi-scale feature map of the image;

[0109] 112) inputting the multi-scale feature map extracted in step 111) into a visual coding network to complete visual coding of the multi-scale feature map;

[0110] 113) inputting the multi-scale feature map extracted in step 111) into a deep feature extraction network to obtain a deep feature map of the image;

[0111] 114) inputting the depth feature map in step 113) into the depth coding network to complete the depth coding of the depth feature map;

[0112] 115) The visual encoding of the multi-scale feature map in step 112) and the depth encoding of the depth feature map in step 114) are input into a depth-guided decoding network to calculate the coordinates of the geometric center position of the target in the output image.

[0113] In the example, the visual coding network in step 112) is a DETR image coding network trained with image data.

[0114] In the example, the deep feature extraction network in step 113) is two 3*3 convolutional layers trained with deep data.

[0115] In the example, the deep coding network in step 114) is a DETR image coding network trained with deep data.

[0116] Reference Figure 3 As shown, the step 115) specifically includes the following steps:

[0117] 1151) Initialize multiple sequences queries denoted as q;

[0118] 1152) The initialized sequence q and the deep encoding of the deep feature map in step 114) are input into the deep decoding network, and the output result is the feature vector q 1 ;

[0119] 1153) The output result q of step 1152) is 1 The visual encoding of the multi-scale feature map in step 112) is input into the visual decoding network together, and the output result is the feature vector q 2 ;

[0120] 1154) The output result q of step 1153) is 2 The input is a labeled and trained feedforward neural network, and the output feature vector q′, q′ contains the coordinates of the target geometric center position.

[0121] Among them, the deep decoding network in step 1152) is a DETR image decoding network trained with deep data.

[0122] Wherein, the visual decoding network in step 1152) is a DETR image decoding network trained with image data.

[0123] Specifically, the step 12) includes:

[0124] 121) clustering the target geometric center position coordinates and the position coordinates in the traditional track data obtained by the video recognition detection in step 11) using the nearest neighbor matching algorithm according to the target type to form an association graph of each target data containing multiple clusters;

[0125] 122) analyzing the data in each target data association graph; when the target data association graph contains the target geometric center position coordinates obtained by video recognition detection and the position coordinates in the traditional track data, the target geometric center position coordinates obtained by video recognition detection are selected as the target position coordinate data of the fused track; when the target data association graph contains only the position coordinates in the traditional track data, the position coordinates are selected as the target position coordinate data of the fused track;

[0126] 123) According to the target position coordinate data selected in step 122), the monitoring data of the corresponding target is associated to form cooperative target monitoring data and non-cooperative target monitoring data.

[0127] 2) Collect the voice command instructions of the tower controller to perform control voice recognition and repetition consistency check, form control intention data based on voice recognition, and compare it with the collected control intention data based on the electronic process sheet for consistency to form confirmed control intention data; specifically, the following steps are included:

[0128] 21) Collect voice commands from tower controllers, pilots and vehicle drivers, perform control voice recognition and repetition consistency check, and generate six key structured control command text data: runway entry, runway crossing, takeoff, landing, blind landing, and vehicle entry, which are control intention data based on voice recognition;

[0129] 22) Collect flight status data modified by tower controllers in the electronic progress sheet system, including four types of flight status data, namely, runway entry, runway crossing, take-off, and landing, which are control intention data based on the electronic progress sheet;

[0130] 23) Compare the control intention data based on voice recognition in step 21) with the control intention data based on the electronic process sheet in step 22) for consistency to form confirmed control intention data.

[0131] Wherein, the step 21) specifically includes the following steps:

[0132] 211) Collect audio data from the control voice recorder of the tower control unit and convert analog voice signals into digital voice signals;

[0133] 212) Preprocessing the digital voice signal by noise reduction and audio segmentation, and using a deep learning speech recognition method based on a neural network to recognize the preprocessed voice signal as an unstructured control instruction text;

[0134] 213) By establishing a control command voice corpus and a digital voice rule library, the control command intention is structured extracted to form structured control command text data, including five key structured control command text data for aircraft command: entering the runway, crossing the runway, taking off, landing, and blind landing, as well as key structured control command text data for vehicles that plan to enter the runway for operations;

[0135] Examples of key structured control instruction text data are shown in Table 1 below:

[0136] Table 1

[0137]

[0138] 214) Compare the tower controller's instruction text with the pilot / vehicle driver's readback text to determine whether the two are consistent and give a readback consistency mark;

[0139] 215) Output the recognized voice text data, including: seat name, voice channel, up / down instruction mark, voice start time, voice end time, voice duration, Chinese / English mark, original voice text, structured control instruction text, target (flight, vehicle) mark, and repetition consistency mark.

[0140] Among them, the repeat consistency mark includes a repeat inconsistent mark and a repeat consistent mark; when it is a repeat inconsistent mark, an alarm prompt is issued to remind the tower controller to find the cause and deal with it, and the method ends; when it is a repeat consistent mark, step 23 is executed.

[0141] Wherein, the step 23) specifically includes the following steps:

[0142] 231) The control intention data based on speech recognition obtained in step 21) is compared with the control intention data based on the single process list in step 22) for consistency, and the comparison check data includes entering the runway, crossing the runway, taking off, and landing; when there is inconsistency, an alarm prompt is issued to remind the tower controller to find the cause and deal with it, and the method ends; when there is consistency, step 232) is executed;

[0143] 232) The four types of control intention data of entering the runway, crossing the runway, taking off and landing, and the two types of control intention data of allowing blind landing and vehicles being allowed to enter the runway that have been compared and found to be consistent are used as confirmed control intention data and output.

[0144] 3) Obtain runway status data;

[0145] The runway status data in step 3) includes the runway status data input by the tower controller and the runway status data input by the electronic process sheet system.

[0146] The runway status data in step 3) specifically includes: runway open status, runway closed status, runway inspection status, and third-party occupied status;

[0147] The runway open state means that the current runway is in normal operation, allowing aircraft to take off and land on the runway;

[0148] Runway closure status means that the runway is not suitable for aircraft takeoff and landing due to bad weather, navigation equipment failure or other events, and the runway is announced to be closed;

[0149] The runway inspection status refers to the status where the airport runway temporarily requires airport maintenance vehicles / personnel to enter the runway for runway inspection.

[0150] 4) Using the target monitoring data, confirmed control intention data, and runway status data generated in step 1), step 2), and step 3), a data combination logic judgment process is performed on various runway incursion scenarios to generate a runway incursion warning or alarm;

[0151] Among them, the runway intrusion scenarios in step 4) include: for ground protection zone intrusion, runway occupancy conflict, and runway use error, the target monitoring data, confirmed control intention data, and runway status data formed in steps 1), 2), and 3) are used to perform data combination logic judgment processing to generate a runway intrusion warning or alarm.

[0152] Specifically, in step 4), for the ground protection zone intrusion scenario, several ground protection zone intrusion sub-scenarios are established, and the target monitoring data, confirmed control intention data, and runway status data formed in steps 1), 2), and 3) are used to perform data combination logic judgment processing, specifically including the following:

[0153] 411) When the runway status is closed, based on the target surveillance data in step 1), when it is detected that the incoming aircraft is about to intrude into the ground protection zone (at a certain distance or time from the runway head, which can be configured), or based on the confirmed control intention data of the aircraft in step 2), when the aircraft is landing, a runway intrusion warning is generated;

[0154] 412) When the runway status is closed, based on the target surveillance data in step 1), when it is detected that the departing aircraft is about to (at a certain distance or time from the ground protection zone, which can be configured) or has already intruded into the ground protection zone, or based on the aircraft's confirmed control intention data in step 2), when it is entering the runway, a runway intrusion warning or alarm is generated;

[0155] 413) The runway status is check. When the incoming aircraft is detected to be about to intrude into the ground protection zone (at a certain distance or time from the runway head, which can be configured) based on the target surveillance data in step 1), or when the aircraft's confirmed control intention data in step 2) indicates that it can land blindly, a runway intrusion warning is generated;

[0156] 414) the runway status is check, and based on the target monitoring data in step 1) and the confirmed control intention data in step 2), when any target other than the vehicle that can enter the runway in the confirmed control intention data intrudes into the ground protection zone, a runway intrusion alarm is generated;

[0157] 415) the runway status is open, and when a vehicle, a person, or an animal is about to or has intruded into the ground protection zone based on the target monitoring data in step 1), a runway intrusion warning or alarm is generated;

[0158] 416) The runway status is open. When it is detected based on the target monitoring data in step 1) and the confirmed control intention data in step 2) that the confirmed control intention data is not to enter the runway or cross the runway, the aircraft intrudes into the ground protection area and a runway intrusion alarm is generated.

[0159] Among them, the runway occupancy conflict scenario in step 4) is specifically: when the runway status is open, based on the target monitoring data in step 1) and the confirmed control intention data in step 2), when it is detected that two targets are about to or have used the same runway at the same time, it is judged as a runway occupancy conflict.

[0160] For the runway occupancy conflict scenario, several runway occupancy conflict sub-scenarios are established, and the target monitoring data, confirmed control intention data, and runway status data formed in step 1), step 2), and step 3) are used to perform data combination logic judgment processing, specifically including:

[0161] 421) Based on the target monitoring data in step 1), when an incoming aircraft is about to land on a runway (a certain distance or time from the runway head, which can be configured), and the ground protection area to which the runway belongs has a departing aircraft, a runway-crossing aircraft, a vehicle, a person or an animal, a runway occupancy conflict alarm is generated;

[0162] 422) Based on the target monitoring data in step 1) and the confirmed control intention data in step 2), when the arriving aircraft is about to land on the runway (a certain distance or a certain time from the runway head, which can be configured) or the confirmed control intention data is landing, the departing aircraft using the runway confirms the control intention data is entering the runway, or the control intention data of the runway-crossing aircraft using the runway is crossing the runway, a runway occupancy conflict alarm is generated;

[0163] 423) Based on the target monitoring data in step 1), when the departing aircraft is detected in the ground protection zone, the arriving aircraft is about to land on the runway (a certain distance or a certain time from the runway head, which can be configured) or there are vehicles, personnel, animals or aircraft crossing the runway within the ground protection zone, a runway occupancy conflict alarm is generated;

[0164] 424) Based on the target monitoring data in step 1) and the confirmed control intention data in step 2), it is detected that when the departing aircraft is in the ground protection area or the confirmed control intention data is to enter the runway and take off, the confirmed control intention data of the arriving aircraft using the runway is to land, or the confirmed control intention data of the runway crossing aircraft using the runway is to cross the runway, a runway occupancy conflict alarm is generated.

[0165] For the runway use error scenario, several runway use error sub-scenarios are established, and the target monitoring data, confirmed control intention data, and runway status data formed in step 1), step 2), and step 3) are used to perform data combination logic judgment processing, including:

[0166] When the runway number used by the aircraft detected based on the target monitoring data in step 1) is inconsistent with the runway number that the aircraft plans to use in the confirmed control intention data in step 2), a runway occupancy conflict alarm is generated;

[0167] When it is detected based on the target monitoring data in step 1) that the runway used by the aircraft is inconsistent with the runway number assigned to the aircraft in the flight plan data, a runway occupancy conflict alarm is generated;

[0168] The flight plan data is the flight plan data formed by the tower automation system receiving the aviation flight dynamic telegram (AFTN) telegram and processing it;

[0169] When the runway number that the aircraft plans to use in the control intention data is confirmed to be inconsistent with the runway number assigned to the aircraft in the flight plan data in step 2), a runway occupancy conflict alarm is generated.

[0170] 5) sending the target monitoring data in step 1) and the runway intrusion warning or alarm in step 4) to relevant parties in runway intrusion prevention in real time, realizing situation monitoring and runway intrusion warning; specifically including: sending the target monitoring data in step 1) and the runway intrusion warning and alarm in step 4) to relevant parties in runway intrusion prevention in real time, realizing situation monitoring and runway intrusion warning sharing, and using terminal display, alarm sound, alarm light or vibration to remind pilots, vehicle drivers, field personnel, controllers, and airport management personnel to deal with the runway intrusion incidents that are about to or have occurred.

[0171] Among them, the relevant parties for preventing runway intrusion in step 5) include: aircraft cockpit, vehicle end, personnel end, control tower, and airport flight area management department.

[0172] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A runway intrusion prevention method based on multi-source heterogeneous data fusion processing, characterized in that: Here are the steps: 1) Obtaining video stream data collected by a panoramic infrared camera over the ground protected area, performing video recognition to generate target recognition data; and fusing the received traditional surveillance data with the target recognition data to form cooperative target surveillance data and non-cooperative target surveillance data; 2) Collect the voice commands of the tower controller to conduct control voice recognition and repetition consistency check, form control intention data based on voice recognition, and compare it with the control intention data collected based on the electronic process sheet for consistency, to form confirmed control intention data; 3) Obtain runway status data; 4) Using the target monitoring data, confirmed control intention data, and runway status data generated in step 1), step 2), and step 3), a data combination logic judgment process is performed on various runway incursion scenarios to generate a runway incursion warning or alarm; 5) The target monitoring data in step 1) and the runway intrusion warning or alarm in step 4) are sent in real time to the relevant parties of runway intrusion prevention to realize situation monitoring and runway intrusion warning.

2. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 1 is characterized in that: The step 1) specifically comprises the following steps: 11) Obtain the video stream data collected by the panoramic infrared camera in the ground protection area, use the depth-guided monocular 3D video detection method to perform target recognition on the video stream data, and obtain the target identification, target type, target geometric center position coordinates, and target contour coordinates; 12) Receive the traditional monitoring data output by the monitoring device, and fuse it with the video target recognition data in step 11) to form cooperative target monitoring data and non-cooperative target monitoring data.

3. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 2 is characterized in that: The method of using a monocular 3D video detection method based on depth guidance to perform target recognition on video stream data specifically includes the following steps: 111) Inputting the video stream data collected from the ground protection area by the panoramic infrared camera into the feature extraction network to extract the multi-scale feature map of the image; 112) inputting the multi-scale feature map extracted in step 111) into a visual coding network to complete visual coding of the multi-scale feature map; 113) inputting the multi-scale feature map extracted in step 111) into a deep feature extraction network to obtain a deep feature map of the image; 114) inputting the depth feature map in step 113) into the depth coding network to complete the depth coding of the depth feature map; 115) The visual encoding of the multi-scale feature map in step 112) and the depth encoding of the depth feature map in step 114) are input into a depth-guided decoding network to calculate the coordinates of the geometric center position of the target in the output image.

4. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 3 is characterized in that: The step 115) specifically includes the following steps: 1151) Initialize multiple sequences queries denoted as q; 1152) inputting the initialized sequence q and the deep encoding of the deep feature map in step 114) into the deep decoding network, and the output result is the feature vector q1; 1153) inputting the output result q1 of step 1152) and the visual encoding of the multi-scale feature map in step 112) into the visual decoding network, and the output result is the feature vector q2; 1154) The output result q2 of step 1153) is input into the trained feedforward neural network, and the output feature vector q′, q′ contains the coordinates of the target geometric center position.

5. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 2 is characterized in that: The step 12) specifically includes: 121) clustering the target geometric center position coordinates and the position coordinates in the traditional track data obtained by the video recognition detection in step 11) using the nearest neighbor matching algorithm according to the target type to form an association graph of each target data containing multiple clusters; 122) analyzing the data in each target data association graph; when the target data association graph contains the target geometric center position coordinates obtained by video recognition detection and the position coordinates in the traditional track data, the target geometric center position coordinates obtained by video recognition detection are selected as the target position coordinate data of the fused track; when the target data association graph contains only the position coordinates in the traditional track data, the position coordinates are selected as the target position coordinate data of the fused track; 123) According to the target position coordinate data selected in step 122), the monitoring data of the corresponding target is associated to form cooperative target monitoring data and non-cooperative target monitoring data.

6. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 1 is characterized in that: The step 2) specifically comprises the following steps: 21) Collect voice commands from tower controllers, pilots and vehicle drivers, perform control voice recognition and repetition consistency check, and form 6 key structured control command text data, namely, enter the runway, cross the runway, take off, land, blind landing is allowed, and vehicles can enter the runway, which are control intention data based on voice recognition; 22) Collect flight status data modified by tower controllers in the electronic progress sheet system, including four types of flight status data, namely, runway entry, runway crossing, take-off, and landing, which are control intention data based on the electronic progress sheet; 23) Compare the control intention data based on voice recognition in step 21) with the control intention data based on the electronic process sheet in step 22) for consistency to form confirmed control intention data.

7. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 6 is characterized in that: The step 21) specifically comprises the following steps: 211) Collect audio data from the control voice recorder of the tower control unit and convert analog voice signals into digital voice signals; 212) Preprocessing the digital voice signal by noise reduction and audio segmentation, and using a deep learning speech recognition method based on a neural network to recognize the preprocessed voice signal as an unstructured control instruction text; 213) By establishing a control command voice corpus and a digital voice rule library, the control command intention is structured extracted to form structured control command text data, including five key structured control command text data for aircraft command: entering the runway, crossing the runway, taking off, landing, and blind landing, as well as key structured control command text data for vehicles that plan to enter the runway for operations; 214) Compare the tower controller's instruction text with the pilot / vehicle driver's readback text to determine whether the two are consistent and give a readback consistency mark; 215) Output the recognized voice text data, including: seat name, voice channel, up / down instruction mark, voice start time, voice end time, voice duration, Chinese / English mark, original voice text, structured control instruction text, target mark, and repetition consistency mark.

8. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 6 is characterized in that: The step 23) specifically includes the following steps: 231) The control intention data based on speech recognition obtained in step 21) is compared with the control intention data based on the single process list in step 22) for consistency, and the comparison check data includes entering the runway, crossing the runway, taking off, and landing; when there is inconsistency, an alarm prompt is issued to remind the tower controller to find the cause and deal with it, and the method ends; when there is consistency, step 232) is executed; 232) The four types of control intention data of entering the runway, crossing the runway, taking off and landing, and the two types of control intention data of allowing blind landing and vehicles being allowed to enter the runway that have been compared and found to be consistent are used as confirmed control intention data and output.

9. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 1 is characterized in that: The runway intrusion scenarios in step 4) include: for ground protection zone intrusion, runway occupancy conflict, and runway use error, the target monitoring data, confirmed control intention data, and runway status data formed in steps 1), 2), and 3) are used to perform data combination logic judgment processing to generate a runway intrusion warning or alarm.

10. The method for preventing runway intrusion based on multi-source heterogeneous data fusion processing according to claim 1, characterized in that: The runway status data in step 3) specifically includes: runway open status, runway closed status, runway inspection status, and third-party occupied status; The runway open state means that the current runway is in normal operation, allowing aircraft to take off and land on the runway; Runway closure status means that the runway is not suitable for aircraft takeoff and landing due to bad weather, navigation equipment failure or other events, and the runway is announced to be closed; The runway inspection status refers to the status where the airport runway temporarily requires airport maintenance vehicles / personnel to enter the runway for runway inspection.

Images