An airport runway foreign matter detection method and system based on multi-sensor fusion

Abstract

The application provides an airport runway foreign matter detection method and system based on multi-sensor fusion, which comprises the following steps: step 1, obtaining radar information of FOD, and calculating the azimuth and distance of the FOD; step 2, obtaining image information according to the azimuth and distance information of the FOD calculated in step 1; and step 3, performing feature fusion on the radar information in step 1 and the image information in step 2 to obtain a detection result. The multi-sensor fusion method makes the advantages and disadvantages of the radar equipment and the optical equipment complementary, has the advantages of the radar equipment, such as positioning ability, strong anti-interference ability and no influence of light, and also has the advantage of the optical equipment, that is, rich semantic information can be collected, thereby effectively preventing missed detection and false detection and improving the accuracy of FOD detection.

Description

Technical Field

[0001] This invention relates to the technical field of radar detection and image target detection, specifically to a method and system for detecting foreign objects on airport runways based on multi-sensor fusion. Background Technology

[0002] Foreign object debris (FOD) on airport runways has always been a bottleneck in maintaining airport safety. Common FOD items include aircraft and engine fasteners, passenger belongings, and animals. FOD, especially on the runway, can cause significant damage to aircraft, resulting in substantial economic losses and even personal injury or death.

[0003] With economic development and technological advancements, configuring foreign object detection (FOD) systems for airport runways has gradually become a crucial indicator of airport safety. Currently, the main technologies for FOD detection at airports both domestically and internationally fall into two categories: optical equipment detection and radar equipment detection. In visual equipment detection methods, early FOD detection algorithms used manual feature extraction followed by SVM classification, a method similar to early pedestrian detection. Later, with the rise of deep learning, researchers used deep learning methods for FOD detection. The most common are improved Yolov3 and improved Faster R-CNN algorithms, which made improvements to FOD detection based on its characteristics. However, since FOD targets are typically small with low resolution, and there are often low-visibility application scenarios at night and in special weather conditions, the reliability of using optical detection equipment alone is relatively low. Radar equipment detection has more practical applications than visual equipment detection. Currently, the radar used for airport runway FOD detection is millimeter-wave radar, such as the Tarsier 1100 and FOD Finder abroad, and domestic companies like Saiying Technology and Suzhou Leike. Millimeter-wave radar is widely used in FOD (Fear of Disasters) detection due to its strong anti-interference capabilities and long detection range. However, it also suffers from low accuracy, frequent false alarms, and a lack of semantic information, making it unable to distinguish the types of detected objects. Therefore, its performance as a standalone detection device is unsatisfactory. This invention addresses these problems and, considering practical application scenarios, uses an airport edge-light detection system as the detection platform. It plans to design a multi-sensor fusion neural network detection algorithm that complements the strengths and weaknesses of optical and radar equipment. Summary of the Invention

[0004] This invention provides a multi-sensor fusion detection method and system that primarily uses visual detection and secondarily uses radar detection, in order to improve the performance of foreign object detection on airport runways.

[0005] A method for detecting foreign objects on airport runways based on multi-sensor fusion includes the following steps:

[0006] Step 1: Obtain radar information of FOD and calculate FOD's azimuth and distance;

[0007] Step 2: Obtain image information based on the FOD orientation and distance information calculated in Step 1;

[0008] Step 3: Apply the radar information from Step 1 and the image information from Step 2 to perform feature fusion and obtain the detection results.

[0009] A further improvement of the present invention is that step 3 specifically includes: feeding image information into a neural network to obtain multiple candidate regions, filtering each candidate region according to radar information, and obtaining the final detection result.

[0010] A further improvement of the present invention is that: the neural network is a target detector, the target detector includes a backbone feature extraction network to obtain multi-layer image features, a feature pyramid to fuse the multi-layer image features, and a detection head to generate candidate boxes; the candidate boxes are the candidate regions.

[0011] A further improvement of the present invention is that: after the image information is processed by the backbone feature extraction network, feature layer 3, feature layer 2, and feature layer 1 are obtained;

[0012] The feature pyramid fusion process includes: feature layer 1 is processed through convolutional layers and upsampling layers to obtain feature layer P1; feature layer P1 is weighted and fused with feature layer 2 to obtain fused feature P2; fused feature P2 is processed through convolutional layers and upsampling layers and then weighted and fused with feature layer 3 to obtain fused feature P3; fused feature P3 is used as fused feature N1; fused feature N1 is processed through convolutional layers and downsampling layers and then weighted and fused with fused feature P2 and feature layer 2 to obtain fused feature N2; fused feature N2 is processed through convolutional layers and downsampling layers and then weighted and fused with feature layer 1 to obtain fused feature N3; fused features N1, fused feature N2, and fused feature N3 are input into the detection head to generate candidate boxes.

[0013] A further improvement of the present invention is that: the process of filtering each candidate region based on radar information to obtain the final detection result specifically includes the following steps:

[0014] First, the image information is divided into a certain number of equal-width vertical slices. Then, the point cloud information of FOD acquired by the radar is projected onto the coordinate system of the image information through a transformation matrix to obtain the mapping points of the radar point cloud in the image information. The slices in the image information containing the mapping points of the radar point cloud are radar interest slices. The IOU values ​​of the interest slices and each candidate box are used to filter candidate boxes.

[0015] A further improvement of the present invention is that the radar information is obtained by transmitting a frequency-modulated continuous wave from a millimeter-wave radar sensor, which then contacts the foreign object (FOD) and returns the information. The position and angle of the foreign object relative to the radar are calculated based on the difference between the transmitted and received radar signals.

[0016] A further improvement of the present invention is that the neural network is trained using existing FOD data before it is used to detect FOD.

[0017] A further improvement of the present invention is that training a neural network using existing FOD data specifically includes: before the historical data is input into the neural network, data augmentation is performed on the historical data to improve the diversity of the dataset and the detection capability of the neural network for small sample targets and low-brightness scenes.

[0018] This invention provides an airport runway foreign object detection system based on multi-sensor fusion, comprising:

[0019] The radar monitoring module is used to transmit frequency-modulated continuous wave signals to acquire radar information about FOD;

[0020] The camera module is used to acquire image information of FOD;

[0021] The detector module is used to deploy FOD target detectors;

[0022] The controller includes a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the computer program to implement the steps of the airport runway foreign object detection method based on multi-sensor fusion as described in any one of claims 1-8.

[0023] The beneficial effects of the method of the present invention are as follows: the multi-sensor fusion method makes the advantages and disadvantages of radar equipment and optical equipment complement each other. It has the advantages of radar equipment in terms of strong positioning ability and anti-interference ability and is not affected by light, and it also has the advantages of optical equipment in terms of collecting rich semantic information. It effectively prevents missed detection and false detection and improves the accuracy of FOD detection. Attached Figure Description

[0024] Figure 1 This is a schematic flowchart of an embodiment provided by the present invention;

[0025] Figure 2 This is a schematic diagram of the neural network process provided in the embodiments of the present invention;

[0026] Figure 3 This is a schematic diagram of the feature pyramid process in an embodiment of the present invention;

[0027] Figure 4 This is a demonstration of intermediate results for the neural network portion of an embodiment provided by the present invention;

[0028] Figure 5 A schematic diagram of a radar information filtering candidate box for an embodiment provided by the present invention;

[0029] Figure 6 This is a schematic diagram of the training neural network process provided in an embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the overall system structure provided in the embodiment of the present invention. Detailed Implementation

[0031] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0032] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0033] Some exemplary embodiments of the invention have been described for illustrative purposes. It should be understood that the invention may be implemented in other ways not specifically shown in the accompanying drawings.

[0034] like Figures 1-4 As shown, this invention provides a method for detecting foreign objects on airport pavement based on multi-sensor fusion. It can identify FOD targets by fusing real-time radar information and image information, solving the shortcomings of traditional single-sensor detection and improving runway safety.

[0035] Specifically, this method includes the following steps:

[0036] Step 1: Obtain radar information of the FOD and calculate its azimuth and distance. Specifically, the radar sensor emits an FMCW signal, which returns after contacting the FOD. The azimuth and distance of the FOD are obtained by calculating the difference between the transmitted and received signals.

[0037] Step 2: Obtain image information based on the FOD location and distance information calculated in Step 1. Specifically, adjust the camera module's shooting angle and focal length based on the FOD's location and distance information to obtain clear FOD image information.

[0038] Step 3: Fuse the radar information from Step 1 with the image information from Step 2 to obtain the detection result. Specifically, feed the image information into a neural network to obtain a preliminary result, filter the preliminary result of the image information based on the radar information, and obtain the final detection result.

[0039] Step 4: Store the fusion detection results from Step 3 in the memory and display them to the front-end interface. Specifically, the samples with FOD detected in Step 3 and the detection results are transferred to the memory for storage, and the detection results along with information from the detection site are transmitted to the front-end to prompt staff for processing.

[0040] Combination Figure 2 In step 3 above, the neural network structure includes a backbone feature extraction network, a feature pyramid, and a detection head. The backbone feature extraction network uses CSPDarknet53 to extract image features at different scales; the feature pyramid is used to fuse the image features at different scales to obtain a fused feature layer, which enhances the information richness of the fused feature layer; the detection head is used to detect the fused feature layer and generate candidate boxes on the image information.

[0041] Combination Figure 3 The aforementioned feature pyramid employs a bidirectional weighted path aggregation network. This network performs only four fusion operations: one top-down and one bottom-up. The bidirectional nature of the fusion operation involves introducing a backbone feature of the same scale during the three fusion operations to extract the original features from the network.

[0042] like Figure 3 As shown, in this embodiment, the backbone feature extraction network CSPDarknet53 processes the image through 2... 3 ,2 4 ,2 5 The feature layers 3, 2, and 1 are obtained by downsampling by a factor of 1. The dimension of feature layer 3 is 80*80*256; the dimension of feature layer 2 is 40*40*512; and the dimension of feature layer 1 is 20*20*1024.

[0043] The Feature Pyramid (Bi-PA Net) employs a bidirectional weighted path aggregation network. Feature layer 1 passes through convolutional and upsampling layers to obtain feature layer P1. Feature layer P1 is then weighted and fused with feature layer 2 to obtain fused feature P2. Fusion feature P2 passes through convolutional and upsampling layers and is then weighted and fused with feature layer 3 to obtain fused feature P3; this completes the "top-down" fusion operation. Subsequently, fused feature P3, as fused feature N1, participates in the "bottom-up" fusion operation. In this process, fused feature N1 passes through convolutional and downsampling layers and is weighted and fused with fused feature P2 and feature layer 2 to obtain fused feature N2. Fusion feature N2 passes through convolutional and downsampling layers and is weighted and fused with feature layer 1 to obtain fused feature N3. Fusion features N1, N2, and N3 are input into the detection head to generate candidate boxes.

[0044] The weighted fusion process described above involves element-wise weighting of the two or three input matrices, all of which have the same dimension. Each input matrix has a corresponding weight, which is a learnable parameter and is updated during training. The expression for the weighted fusion is as follows:

[0045]

[0046] Where ω i The input matrix I i The corresponding learnable weights, ε = 0.0001, are a bias to improve numerical stability, I i O is the input matrix, and O is the output feature matrix. Weighted fusion allows the network to learn the weights of each feature during aggregation.

[0047] In this embodiment, the process of obtaining fusion features P2, P3, and N3 involves two input matrices; the weighted process of obtaining fusion feature N2 involves three input matrices. (Combined) Figure 4 After the feature fusion of the bidirectional weighted path aggregation network, the effects of deep features and shallow features are complementary. Figure 4 Feature 1, Feature 2, and Feature 3 represent the image after 2... 3 ,2 4 ,2 5 The sampling is multiplied by a factor of 1; fused feature 1, fused feature 2, and fused feature 3 are fused features of the same size generated after feature fusion of the three features. Specifically, the rich positional information of the shallow features and the rich semantic information of the deep features are fused; simultaneously, the shortcomings of shallow features (more noise) and deep features (low resolution and poor detail perception) are complemented.

[0048] Combination Figure 5In step 3 above, "the candidate regions are screened based on radar information to obtain the final detection result." Specifically, firstly, the image information is divided into a certain number of strip slices by performing a vertical slicing operation with equal width; then, the point cloud information of FOD acquired by the radar is projected onto the coordinate system of the image through a transformation matrix to obtain the mapping of the radar point cloud. The radar projection onto the image involves three steps:

[0049] a. The coordinate transformation from the millimeter-wave coordinate system to the camera-centric world coordinate system has the following mapping matrix:

[0050]

[0051] b. Transform the coordinates from the world coordinate system to the camera coordinate system. The mapping matrix is ​​as follows:

[0052]

[0053] c. Transform the coordinates from the camera coordinate system to the image coordinate system. The mapping matrix is:

[0054]

[0055] Where X, Y, and Z are three-dimensional spatial coordinates, r is the radar coordinate, and c is the camera coordinate. These three matrices complete the transformation from the radar coordinate system to the image coordinate system, obtaining the mapping points of the radar point cloud on the image. The slice containing the mapping points of the radar point cloud in the image information is the radar slice of interest. The Intersection over Union (IOU) value between the slice of interest and each candidate box is used to filter candidate boxes, resulting in the final candidate boxes. IOU is the area of ​​the intersection between the slice and each candidate box. In this application, the candidate box with the largest IOU is used as the final detection result. Through the above fusion, a more accurate location of the Foreground Object Dependence (FOD) can be obtained.

[0056] In step 1 above, radar information is acquired to calculate the azimuth and range of the FOD (Focus on Displacement). Specifically, the millimeter-wave radar continuously transmits frequency-modulated continuous waves to the airport runway surface, and then uses sensors to receive the millimeter-wave signals returned from the FOD. A mixer combines the transmitted and received signals into a single intermediate frequency (IF) signal, which is then used to calculate the distance to the object. Since the millimeter-wave radar has multiple receiving antennas, the distance between two receiving antennas will cause a change in the peak phase of the range FFT or Doppler FFT. This change can be used to calculate the phase angle between the FOD and the detection equipment. To reduce the impact of dust in the airport air, paint protrusions on the runway surface, and roughened ground, this invention uses a Kalman filter to denoise the point cloud generated by the millimeter-wave radar, removing noise from the air and the ground to obtain more accurate radar information.

[0057] Combination Figure 7Before using the neural network to detect FOD, it should be trained using existing FOD data. Specifically, historical FOD images of Daxing Airport and simulated images of some school pavements are used as real-world data. The images are labeled and organized according to the COCO dataset format, and the network parameters are trained using this dataset.

[0058] like Figure 6 As shown, during the training of the aforementioned neural network, to enrich the number of small samples and low-brightness scene samples in the dataset, improved Mosaic and MixUp data augmentation are used before the dataset images enter the neural network. Mosaic data augmentation randomly flips, scales, and processes the brightness of four images before stitching them together. During the random brightness processing, there is a 40% chance that the image brightness will be reduced. When the brightness is reduced, the image brightness value V is reduced to 1-0.3 times the original value with equal probability. MixUp uses the copypaste method to paste one image onto another image, and both images use a constant opacity of 0.5.

[0059] Embodiments of the present invention also provide an airport runway foreign object detection system based on multi-sensor fusion, comprising:

[0060] The radar monitoring module is used to transmit and receive frequency-modulated continuous wave signals to obtain radar information about FOD;

[0061] The camera module is used to acquire image information of FOD;

[0062] The detector module is used to deploy FOD target detectors;

[0063] The controller includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the control method for the FOD detection structure described above.

[0064] The radar monitoring module described above uses Texas Instruments IWR millimeter-wave radar, which includes two transmit channels and four receive channels, and is capable of transmitting and receiving FMCW signals to obtain radar information.

[0065] The aforementioned camera module uses an industrial camera, which can work in conjunction with radar information to acquire clear FOD image information.

[0066] The aforementioned detector module uses an NVIDIA development board to deploy the target detector, which uses image and radar information to detect the type and location of foreign objects on the airport pavement.

[0067] The aforementioned controller includes a memory and a processor. The processor coordinates with the radar module, camera module, and detector module to transmit radar information detected by the radar module to the camera module, adjust the camera's orientation and focal length, and acquire image information; transmit the images acquired by the camera module to the detector to generate FOD candidate boxes, then project the radar information onto the image to filter the candidate boxes and obtain the final detection result; save the detection result to the memory, and transmit the detection result and the information from the detection site to the front end to prompt the staff for processing; historical data in the memory can be used to update the neural network model.

[0068] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for detecting foreign objects on airport runways based on multi-sensor fusion, characterized in that, Includes the following steps: Step 1: Obtain radar information of the FOD and calculate its azimuth and distance. The radar information is obtained by transmitting a frequency-modulated continuous wave from a millimeter-wave radar sensor, which then contacts the FOD and returns the signal. The azimuth and angle of the foreign object relative to the radar are calculated based on the difference between the transmitted and received radar signals. Step 2: Obtain image information based on the FOD orientation and distance information calculated in Step 1; Step 3: Apply the radar information from Step 1 and the image information from Step 2 to perform feature fusion and obtain the detection results; Step 3 specifically includes: feeding image information into a neural network to obtain multiple candidate regions, filtering each candidate region according to radar information, and obtaining the final detection result; The process of filtering candidate regions based on radar information to obtain the final detection results includes the following steps: First, the image information is divided into a certain number of slices by performing a vertical slicing operation with equal width. Then, the point cloud information of FOD acquired by the radar is projected onto the coordinate system of the image information through a transformation matrix to obtain the mapping points of the radar point cloud in the image information. The slices in the image information containing the mapping points of the radar point cloud are radar interest slices. The IOU values ​​of the radar interest slices and each candidate box are used to filter candidate boxes. The IOU value is the intersection area of ​​the radar slice of interest and each candidate box.

2. The method for detecting foreign objects on airport runways based on multi-sensor fusion according to claim 1, characterized in that, The neural network is a target detector, which includes a backbone feature extraction network to obtain multi-layer image features, a feature pyramid to fuse the multi-layer image features, and a detection head to generate candidate boxes; the candidate boxes are the candidate regions.

3. The method for detecting foreign objects on airport runways based on multi-sensor fusion according to claim 2, characterized in that, After the image information is processed by the backbone feature extraction network, feature layer 3, feature layer 2, and feature layer 1 are obtained; The feature pyramid fusion process includes: feature layer 1 is processed through convolutional layers and upsampling layers to obtain feature layer P1; feature layer P1 is weighted and fused with feature layer 2 to obtain fused feature P2; fused feature P2 is processed through convolutional layers and upsampling layers and then weighted and fused with feature layer 3 to obtain fused feature P3; fused feature P3 is used as fused feature N1; fused feature N1 is processed through convolutional layers and downsampling layers and then weighted and fused with fused feature P2 and feature layer 2 to obtain fused feature N2; fused feature N2 is processed through convolutional layers and downsampling layers and then weighted and fused with feature layer 1 to obtain fused feature N3; fused features N1, fused feature N2, and fused feature N3 are input into the detection head to generate candidate boxes.

4. The method for detecting foreign objects on airport runways based on multi-sensor fusion according to claim 2, characterized in that, The neural network is trained using existing FOD data before it is used to detect FOD.

5. The method for detecting foreign objects on airport runways based on multi-sensor fusion according to claim 4, characterized in that, Training a neural network using existing FOD data specifically includes: before the historical data is input into the neural network, data augmentation is performed on the historical data to improve the diversity of the dataset and the neural network's ability to detect small-sample targets and low-light scenes.

6. A foreign object detection system for airport runways based on multi-sensor fusion, characterized in that, include: The radar monitoring module is used to transmit frequency-modulated continuous wave signals to acquire radar information about FOD; The camera module is used to acquire image information of FOD; The detector module is used to deploy FOD target detectors; The controller includes a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the computer program to implement the steps of the airport runway foreign object detection method based on multi-sensor fusion as described in any one of claims 1-5.

Images