An airport FOD detection and cleaning system and method based on multi-source perception fusion

Abstract

This invention discloses an airport FOD (Foreign Object Debris) detection and removal system and method based on multi-source sensing fusion. The system includes a mobile transport chassis, a multi-source heterogeneous sensing unit, a six-degree-of-freedom collaborative robotic arm, a composite cleaning end effector, and an edge computing platform. During operation, environmental data is first collected using LiDAR, millimeter-wave radar, and a binocular camera. Spatiotemporal fusion and state estimation algorithms are used to achieve all-weather, high-precision positioning and attribute calculation of FOD. The composite cleaning end effector integrates a high-negative-pressure suction port, a ring-shaped electromagnetic adsorption array, and a flexible mechanical claw. Subsequently, the system adaptively switches between magnetic suction, suction pickup, or gripping modes based on the material, volume, and shape characteristics of the FOD, and combines a visual closed-loop feedback mechanism to automatically upgrade and verify the cleaning strategy. This invention effectively solves the problems of poor perception robustness under complex weather conditions and insufficient adaptability of single cleaning methods, significantly improving the intelligence level and operational efficiency of airport pavement foreign object removal.

Description

Technical Field

[0001] This invention relates to the field of foreign object detection and removal technology for airport pavements, specifically to an airport FOD detection and removal system and method based on multi-source sensing fusion. Background Technology

[0002] Foreign Object Debris (FOD) refers to any object located within the airport's operational area that has no operational function, such as aircraft parts, pavement debris, lost tools, and plastic products. The presence of FOD poses a significant threat to the operational safety of aircraft. If it is sucked into an engine or punctured a tire, it can easily lead to serious safety accidents and cause huge economic losses. Therefore, timely and efficient removal of FOD from airport pavements is a key aspect of ensuring the safe operation of the airport.

[0003] Currently, airport FOD (Fouling Object) cleanup mainly relies on manual inspections or large cleaning vehicles. Manual inspections are inefficient and prone to omissions due to human eye fatigue and weather conditions. While large cleaning vehicles cover a large area, they often employ indiscriminate mechanical cleaning methods, lacking specificity, consuming high energy, and struggling to handle complex shapes of debris. Although some automated cleaning solutions have been proposed, existing automated FOD cleanup technologies still face numerous bottlenecks in practical applications: First, the environmental robustness of the perception system is insufficient. Existing equipment mostly uses a single vision sensor for detection. However, the airport environment is complex and variable. Under conditions such as direct sunlight, low light at night, rain and fog, and reflections from water on the runway, the recognition rate of the vision system will drop sharply, making it prone to missed detections or false alarms. In addition, relying solely on vision is not enough to obtain accurate depth information of FOD, which limits the positioning accuracy of the robotic arm and affects the success rate of grasping.

[0004] Secondly, the cleaning execution mechanism has poor adaptability. Existing equipment mostly uses a single vacuum cleaner-style negative pressure suction mechanism or a simple rigid mechanical gripper. Negative pressure suction works well for sheet-like lightweight objects, but when faced with irregular heavy objects, ferromagnetic objects, or objects partially embedded in the pavement gaps, it often fails to clean due to insufficient suction or poor sealing. On the other hand, traditional rigid mechanical grippers are prone to damaging pavement lights or scratching the runway surface due to improper control during the gripping process, posing safety hazards.

[0005] Finally, there is a lack of refined operational strategies and closed-loop feedback mechanisms. The existing equipment operates in a fixed mode, performing the same cleaning actions regardless of the type of FOD, lacking the intelligent decision-making ability to automatically adjust operational parameters based on target characteristics. This results in low operational efficiency and high energy consumption. Furthermore, after each cleaning operation, there is a lack of effective verification methods to confirm whether foreign objects have been completely removed, which can easily lead to FOD residue and prevent the formation of a complete operational closed loop.

[0006] In summary, developing a cleaning system with all-weather, high-precision perception capabilities, adaptability to various FOD (Foreign Object Debris) patterns, and intelligent decision-making and closed-loop verification functions is a pressing technical challenge in the field of airport pavement foreign object management. Summary of the Invention

[0007] To address the problems existing in the aforementioned background technology, this invention proposes an airport FOD detection and removal system and method based on multi-source perception fusion. By constructing a spatiotemporal fusion perception network based on lidar, millimeter-wave radar, and visual sensors, integrating a multimodal composite removal terminal with aerodynamic-magnetic-flexible grasping capabilities, and an adaptive hierarchical decision-making mechanism based on visual closed-loop feedback, this invention solves core technical problems such as poor robustness of single visual perception under complex weather conditions, insufficient adaptability of single removal methods to irregularly shaped or ferromagnetic heavy objects, and lack of intelligent error correction in traditional operation modes. This achieves joint optimization of target detection accuracy, removal operation adaptability, and overall closed-loop safety in airport pavement foreign object removal.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: An airport FOD detection and cleanup system based on multi-source sensing fusion includes: The mobile transport chassis serves as the main load-bearing component of the system. A multi-source heterogeneous sensing unit is installed at the front end of the mobile transport chassis to collect multi-dimensional environmental data of the airport pavement. A multi-degree-of-freedom collaborative robotic arm is positioned in the middle of the mobile transport chassis; A composite cleaning end effector is disposed at the end of the multi-degree-of-freedom collaborative robotic arm. The composite cleaning end effector includes three functional modules: a high negative pressure pneumatic suction port, a ring electromagnetic adsorption array, and a flexible mechanical gripper. The edge computing platform is communicatively connected to the mobile transport chassis, the multi-source heterogeneous sensing unit, the multi-degree-of-freedom collaborative robotic arm, and the composite cleaning end effector, and is used to perform data fusion calculation and issue operation control commands.

[0009] Specifically, the multi-source heterogeneous sensing unit is fixedly mounted on the front end of the mobile transport chassis via a rigid bracket; the multi-source heterogeneous sensing unit includes: A lidar unit, installed on the top layer of the rigid support, is used to acquire three-dimensional point cloud data of the environment. A millimeter-wave radar is installed at the center of the bottom layer of the rigid support for long-range detection and motion compensation. A binocular industrial camera, symmetrically distributed on both sides of the millimeter-wave radar, is used to acquire high-resolution texture images; The optical center positions of the lidar, millimeter-wave radar, and binocular industrial camera were jointly calibrated.

[0010] Specifically, the three functional modules of the composite cleaning end effector are independently controlled, adapting to the switching cleaning of different FODs. The high negative pressure pneumatic suction port is located at the center of the composite cleaning end effector and is connected to the turbofan vacuum pump inside the mobile transport chassis through a corrugated hose. The bottom of the high negative pressure pneumatic suction port is provided with a flexible silicone skirt. The annular electromagnetic adsorption array is arranged around the outside of the high negative pressure pneumatic suction port and encapsulates several independently controlled excitation coils inside. The flexible mechanical claw is located on the front side of the composite cleaning end effector. The finger body of the flexible mechanical claw is made of pneumatic variable stiffness material, and the surface of the fingertip is provided with a high friction coefficient coating.

[0011] Specifically, the edge computing platform is equipped with a fusion tracking module for performing FOD target location calculation, and the calculation logic includes: (1) Establish the state and observation model of the linear discrete system. The state equation is expressed as: ; in, For FOD target The state vector at time step includes planar coordinates Horizontal speed , This is the state transition matrix, which describes the motion relationship of the FOD target between two adjacent frames. For process noise, describe the random disturbances in the motion of the FOD target; The observation equation is expressed as: ; in, The observed coordinates of the FOD target in the image. This is the observation matrix, used to map the three-dimensional state space of the FOD target to the two-dimensional image observation space. To observe noise; (2) Predict the current location of the FOD target based on its historical trajectory: ; ; in, for The prior state prediction value of the FOD target at time step. This is the prior covariance matrix, used to describe the uncertainty of the predicted values. For process noise covariance; (3) Calculate the Kalman gain : ; in, To observe the noise covariance; (4) Using the current observation coordinates and Kalman gain After correcting the prediction error, the final optimal state estimate is obtained: ; ; That is, characterization The optimal coordinates of the FOD target at any given time.

[0012] An airport FOD detection and cleanup method based on multi-source sensing fusion is applied to an airport FOD detection and cleanup system based on multi-source sensing fusion as described above. The method includes the following steps: S1. Multi-dimensional environmental data of the work area is collected using a multi-source heterogeneous sensing unit. After spatiotemporal alignment and data fusion processing by an edge computing platform, the attribute features of the FOD target are calculated. The attribute features include spatial location, material type, and geometric parameters. S2. Based on the attribute characteristics of the FOD target, the edge computing platform adopts an adaptive hierarchical strategy to match the corresponding cleanup mode; S3. The multi-degree-of-freedom collaborative robotic arm adjusts its posture to drive the composite cleaning end effector to perform the operation according to the matching cleaning mode; S4. After a single cleaning operation is completed, the multi-source heterogeneous sensing unit is invoked to perform a second scan of the work area to determine whether there is FOD residue in the work area. If residue is determined to exist, the strategy upgrade mechanism is triggered to perform a second cleaning.

[0013] Specifically, the spatiotemporal alignment method in step S1 is as follows: Construct a joint calibration model based on rigid transformation, and use the pre-calibrated extrinsic matrix and camera intrinsic matrix to project the 3D point cloud data collected by the lidar and the target data collected by the millimeter-wave radar onto the pixel coordinate system of the binocular industrial camera. The specific formula is as follows: ; in, This is a scale factor used to correct the scaling ratio of three-dimensional coordinates projected onto a two-dimensional plane. Here are the pixel coordinates of FOD in the camera image. For the camera intrinsic parameter matrix, These are the rigid body rotation matrix and translation matrix of the lidar to the camera, respectively. The coordinates of FOD in the lidar coordinate system are given.

[0014] Specifically, the data fusion processing steps in step S1 include: S11. Use a deep learning object detection network to extract the observation coordinates of FOD targets in the visual image. ; S12. The spatial position of the FOD target is smoothed using a state estimation algorithm to obtain the optimal coordinates of the FOD target. The calculation logic of the state estimation algorithm includes: (1) Establish the state and observation model of the linear discrete system. The state equation is expressed as: ; in, For FOD target The state vector at time step includes planar coordinates Horizontal speed , This is the state transition matrix, which describes the motion relationship of the FOD target between two adjacent frames. For process noise, describe the random disturbances in the motion of the FOD target; The observation equation is expressed as: ; in, The observed coordinates of the FOD target in the image. This is the observation matrix, used to map the three-dimensional state space of the FOD target to the two-dimensional image observation space. To observe noise; (2) Predict the current location of the FOD target based on its historical trajectory: ; ; in, for The prior state prediction value of the FOD target at time step. This is the prior covariance matrix, used to describe the uncertainty of the predicted values. For process noise covariance; (3) Calculate the Kalman gain : ; in, To observe the noise covariance; (4) Using the current observation coordinates and Kalman gain After correcting the prediction error, the final optimal state estimate is obtained: ; ; That is, characterization The optimal coordinates of the FOD target at any given time; S13. Based on the optimal coordinates of the FOD target, extract the point cloud data of the corresponding spatial region from the lidar point cloud. After removing the road background point cloud through background filtering, obtain the exclusive point cloud cluster of the FOD target. Solve the geometric parameters of the FOD target, including volume, bottom area, height and length and width parameters. S14. Extract the radar reflectivity features of the FOD target location detected by millimeter-wave radar and the texture and spectral features of the FOD target location corresponding to the binocular industrial camera. Input the multi-source features into the pre-trained material classification model to determine the material type features of the FOD target. The material type features include two categories: ferromagnetic metals and non-metals. S15. Encapsulate the calculated spatial location, geometric parameters, and material type features to generate the attribute feature vector of the FOD target. .

[0015] Specifically, the process of the adaptive hierarchical strategy in step S2 is as follows: S21. Obtain the attribute feature vector of the FOD target output in step S15. ; S22, Determine the attribute feature vector Material type of FOD target Is it a ferromagnetic metal? Then a magnetic attraction mode command is generated; S23. If the material type of the FOD target is determined to be non-metallic, then calculate the shape factor of the FOD target. : ; in, For the volume of the FOD target, The base area of ​​the FOD target. The height of the FOD target; S24. If the volume of the FOD target Less than the preset volume threshold and shape factor If the value is less than the preset value and the FOD target indicator is flat, then the FOD target is determined to be a lightweight floating object, a suction mode command is generated, and based on... Calculating the power of a turbofan vacuum pump : ; in, This is a proportionality coefficient, obtained through prior calibration. This is the base power of the turbofan vacuum pump; S25. If the volume of the FOD target Greater than the preset volume threshold or shape factor If the value is greater than the preset value and the FOD target indication is blocky, then the FOD target is determined to be a heavy foreign object, and a grab mode instruction is generated.

[0016] Specifically, the action logic for performing the magnetic attraction mode for cleaning in step S3 is as follows: the multi-degree-of-freedom collaborative robotic arm adjusts its posture to align the ring electromagnetic adsorption array with the FOD target, descends to a predetermined distance from the FOD target, the excitation coil is energized to generate a strong magnetic field to attract foreign objects, and then moves to the top of the recycling box, and a reverse pulse current is introduced to demagnetize it, causing the FOD target to fall off. The action logic for cleaning in suction mode is as follows: the multi-degree-of-freedom collaborative robotic arm adjusts its posture so that the high negative pressure pneumatic suction port is vertically aligned with the pavement surface, descends to a predetermined height above the ground, the flexible silicone skirt fits the pavement surface to reduce leakage, the turbofan vacuum pump is started to generate negative pressure to suck up the FOD target, and after the differential pressure sensor confirms that the suction is successful, it moves to the top of the recovery box and stops the pump to release. The action logic for performing the gripping mode for cleaning is as follows: the flexible mechanical claw adjusts its opening range according to the FOD target contour, extends downward to envelop the FOD target, and the flexible fingers use the damping characteristics and deformation ability of their material to adhere to the surface of the foreign object, perform a closing gripping action, and then perform lifting and flipping actions to prevent slippage, and move to the top of the recycling bin to release.

[0017] Specifically, the process of step S4 is as follows: S41. Calculate the difference norm of the images of the work area before and after cleaning. : ; in, To clean up the image data of the work area, Image data for the pre-cleaning work area; the image data is limited to the local ROI region image where the FOD target is located and has undergone pixel-level registration, with a difference norm. The calculations are performed only within this local ROI region; S42. Judge the cleaning effect; if... Greater than the preset success threshold If so, it is determined that the FOD has been cleared; if Less than the preset success threshold If FOD remains, a policy upgrade mechanism is triggered; the success threshold is then determined. The value range is set to be greater than the maximum difference caused by background environmental fluctuations. And less than the minimum difference value generated when the FOD target is completely removed. The strategy upgrade mechanism is as follows: (1) If the cleaning mode used in the last cleaning operation was suction mode, and If the power is less than the maximum power of the turbofan vacuum pump, the control command of the turbofan vacuum pump will be adjusted to full power output, and the suction mode will be executed again. (2) If the cleaning mode used in the last cleaning operation was the suction mode and the output power of the turbofan vacuum pump reached the maximum value, then upgrade to the grab mode to clean the FOD target. (3) If the cleaning mode used in the last cleaning operation was the capture mode, then stop the cleaning operation, record the current location coordinates and image, and send a manual takeover alarm to the remote control center through the edge computing platform.

[0018] In summary, the beneficial technical effects of the present invention are as follows: 1. Possesses all-weather robust perception capability: This invention breaks through the limitations of traditional airport FOD detection, which relies excessively on a single visual sensor. By integrating lidar, millimeter-wave radar, and industrial cameras to construct a multi-source heterogeneous perception unit, and using a spatiotemporal fusion algorithm to unify multi-dimensional data into the same coordinate system, even under complex weather conditions such as strong direct sunlight, low illumination at night, rain, fog, dust, and reflective water on the runway, the system can still utilize the penetration of millimeter-wave radar and the three-dimensional modeling capability of lidar, combined with state estimation algorithms, to achieve stable detection and high-precision positioning of FOD targets, significantly reducing the missed detection rate and false alarm rate.

[0019] 2. Adaptive Multimodal Cleaning Capability: Addressing the limitations of single negative pressure suction mechanisms in handling irregularly shaped heavy objects and the potential damage to pavement lighting caused by rigid mechanical claws, this invention innovatively designs a composite cleaning end effector integrating a high negative pressure pneumatic suction port, a ring-shaped electromagnetic adsorption array, and a two-finger flexible mechanical claw. The system can intelligently switch operating modes based on the material, volume, and geometric characteristics of the FOD target. It employs non-contact magnetic attraction for ferromagnetic metals, completely avoiding the risk of scratching the pavement; uses pneumatic suction for lightweight objects, achieving high efficiency and energy saving; and employs flexible gripping for heavy or irregular objects, ensuring cleaning reliability. This multimodal integrated design greatly expands the coverage range of cleaning targets.

[0020] 3. Intelligent decision-making and closed-loop error correction mechanism: This invention establishes a complete "perception-decision-execution-review" closed-loop control system. The edge computing platform can not only automatically match the optimal cleaning strategy based on multi-dimensional features, but also review the cleaning effect in real time through a visual closed-loop feedback mechanism. Once a cleaning failure is detected, the system automatically triggers the strategy upgrade mechanism, realizing adaptive error correction and secondary cleaning in the operation process, effectively improving operation efficiency and safety. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall structure of the system of the present invention; Figure 2 This is a schematic diagram of the composite cleaning end effector structure in this invention; Figure 3 This is a flowchart of the method of the present invention; Reference numerals: 1. Mobile transport chassis; 2. Multi-source heterogeneous sensing unit; 21. LiDAR; 22. Millimeter-wave radar; 23. Binocular industrial camera; 3. Multi-degree-of-freedom collaborative robotic arm; 4. Composite cleaning end effector; 41. High negative pressure pneumatic suction port; 42. Annular electromagnetic adsorption array; 43. Flexible robotic gripper; 44. Corrugated hose interface; 5. Edge computing platform. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention clearer and easier to understand, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0023] Example like Figure 1 , Figure 2 As shown, the airport FOD detection and cleaning system based on multi-source sensing fusion provided by the present invention consists of a mobile transport chassis 1, a multi-source heterogeneous sensing unit 2, a multi-degree-of-freedom collaborative robotic arm 3, a composite cleaning end effector 4, and an edge computing platform 5.

[0024] Among them, the mobile transport chassis 1 serves as the main carrier of the system, responsible for driving the entire system to move and operate in areas such as airport runways, taxiways, and aprons. It is equipped with four independently driven explosion-proof off-road tires at the bottom, giving the system all-terrain off-road capability and enabling it to adapt to various complex road surface environments. The mobile transport chassis 1 integrates a high-energy-density lithium battery pack and a power management module to provide continuous operating power for the entire machine.

[0025] The multi-source heterogeneous sensing unit 2 is fixedly mounted on the front top of the mobile transport chassis 1 via a shock-absorbing rigid bracket. It is used to collect multi-dimensional data of the airport pavement environment from all directions. This unit integrates three different types of sensors in its physical structure and achieves spatiotemporal synchronization through joint calibration. Specifically: The lidar 21 is installed on the top layer of the sensing unit. In this embodiment, a 16-line mechanical lidar is preferred, with a ranging range of 0.5-150m, a point cloud density of 108,000 points / second, and a repeatability accuracy of ±2cm. It is used to acquire high-precision three-dimensional point cloud data of the environment to calculate the volume, size, and attitude of the FOD target.

[0026] The millimeter-wave radar 22 is installed at the bottom center of the sensing unit. In this embodiment, a 77GHz long-range millimeter-wave radar is preferred, with a detection range of 5-200m, an angular resolution of ±0.5°, and an anti-attenuation capability of ≥20dB in rain and fog. Utilizing its good penetration, it is responsible for detecting and motion compensation of distant targets in low visibility environments such as rain, fog, and dust.

[0027] The binocular industrial camera 23 is symmetrically distributed on both sides of the sensing unit to acquire high-resolution texture images in order to capture semantic information such as color and texture of the FOD target.

[0028] The multi-degree-of-freedom collaborative robotic arm 3 is fixedly installed on the load-bearing beam in the middle of the mobile transport chassis 1. In this embodiment, a six-degree-of-freedom robotic arm is used, with a repeatability accuracy of ±0.1mm and a maximum working radius of 1.2m. It can flexibly adjust the end posture to meet the cleaning needs of FOD in different locations.

[0029] The composite cleaning end effector 4 is rigidly connected to the end of the multi-degree-of-freedom collaborative robotic arm 3. To address the poor adaptability of a single cleaning method, it adopts an integrated layout, integrating three operating modules: The high negative pressure pneumatic suction port 41 is located at the center of the composite cleaning end effector 4. It is connected to the high-power turbofan vacuum pump inside the mobile transport chassis 1 through the corrugated hose interface 44. The bottom of the suction port is equipped with a flexible silicone skirt, which can fit the surface to form a sealed cavity for picking up light sheet objects such as paper scraps and plastics.

[0030] The annular electromagnetic adsorption array 42 is arranged around the outside of the high negative pressure pneumatic suction port 41. It is encapsulated with multiple independently controlled excitation coils. When energized, it generates a strong magnetic field, which can non-contactly adsorb ferromagnetic metal foreign objects such as nuts and iron sheets. When the power is cut off and a reverse pulse current is used, it can be demagnetized and released.

[0031] The flexible mechanical claw 43 is installed on the front side of the composite cleaning end effector 4. In this embodiment, it is a two-finger flexible mechanical claw. The finger body is made of a pneumatic variable stiffness material and the finger pad surface is covered with a high friction coefficient coating. This mechanical claw can envelop and grab irregular heavy objects such as stones and rubber blocks, effectively avoiding damage to pavement lights caused by rigid gripping.

[0032] The edge computing platform 5 is housed in the protective housing at the end of the multi-degree-of-freedom collaborative robotic arm 3. It communicates with the mobile transport chassis 1, the multi-source heterogeneous sensing unit 2, the multi-degree-of-freedom collaborative robotic arm 3, and the composite cleaning end effector 4 via a CAN bus. The platform has a built-in high-performance processor that is responsible for performing spatiotemporal fusion calculation of multi-source data, FOD target identification and attribute judgment, and generating corresponding operation control commands based on the judgment results and sending them to each actuator to achieve full-process automated control.

[0033] like Figure 3 As shown, specifically, for the above-mentioned system, the detection and cleaning method provided by the present invention includes the following steps: S1. During the movement of the mobile transport chassis 1, the system continuously collects multi-dimensional environmental data of the work area using the multi-source heterogeneous sensing unit 2. The edge computing platform 5 first constructs a joint calibration model based on rigid transformation. Using the pre-calibrated extrinsic matrix and camera intrinsic matrix, the three-dimensional point cloud data collected by the lidar 21 and the target data collected by the millimeter-wave radar 22 are projected onto the pixel coordinate system of the binocular industrial camera 23 to form RGB-D enhanced data. The specific formula is as follows: ; in, This is a scale factor used to correct the scaling ratio of three-dimensional coordinates projected onto a two-dimensional plane. Here are the pixel coordinates of FOD in the camera image. For the camera intrinsic parameter matrix, These are the rigid body rotation matrix and translation matrix of the lidar to the camera, respectively. The coordinates of FOD in the lidar coordinate system are given.

[0034] Furthermore, in order to achieve high-precision fusion of heterogeneous sensor data under a unified spatial benchmark and to calculate the attribute features of the FOD target, including spatial location, material type, and geometric parameters, the edge computing platform 5 performs data fusion processing. The specific data fusion processing steps include: S11. Use a deep learning object detection network to extract the observation coordinates of FOD targets in the visual image. In this embodiment of the invention, the deep learning object detection network preferably adopts the YOLOv5s model structure; S12. The spatial position of the FOD target is smoothed using a state estimation algorithm to obtain the optimal coordinates of the FOD target. The calculation logic of the state estimation algorithm includes: (1) Establish the state and observation model of the linear discrete system. The state equation is expressed as: ; in, For FOD target The state vector at time step includes planar coordinates Horizontal speed , This is the state transition matrix, which describes the motion relationship of the FOD target between two adjacent frames. For process noise, describe the random disturbances in the motion of the FOD target; The observation equation is expressed as: ; in, The observed coordinates of the FOD target in the image. This is the observation matrix, used to map the three-dimensional state space of the FOD target to the two-dimensional image observation space. To observe noise, it follows a Gaussian distribution; this value will increase significantly when visual interference occurs. (2) Predict the current location of the FOD target based on its historical trajectory: ; ; in, for The prior state prediction value of the FOD target at time step. This is the prior covariance matrix, used to describe the uncertainty of the predicted values. For process noise covariance; (3) Calculate the Kalman gain : ; in, To observe the noise covariance, when vision is disturbed by strong light / rain / fog It will increase, at which point the calculated value will be... The value will automatically decrease, indicating that the system has less confidence in the observed values ​​and relies more on historical predictions to avoid sudden changes in positioning. (4) Using the current observation coordinates and Kalman gain After correcting the prediction error, the final optimal state estimate is obtained: ; ; That is, characterization The optimal coordinates of the FOD target at any given time are smooth and continuous, and will not change due to brief visual omissions. S13. Based on the optimal coordinates of the FOD target, extract the point cloud data of the corresponding spatial region from the lidar point cloud. After removing the road background point cloud through background filtering, obtain the exclusive point cloud cluster of the FOD target. Solve the geometric parameters of the FOD target, including volume, bottom area, height and length and width parameters. S14. Extract the radar reflectivity features of the FOD target location detected by millimeter-wave radar and the texture and spectral features of the FOD target location corresponding to the binocular industrial camera. Input the multi-source features into the pre-trained material classification model to determine the material type features of the FOD target. The material type features include two categories: ferromagnetic metals and non-metals. S15. Encapsulate the calculated spatial location, geometric parameters, and material type features to generate the attribute feature vector of the FOD target. .

[0035] S2. Based on the attribute characteristics of FOD targets, the edge computing platform adopts an adaptive hierarchical strategy to match the corresponding cleanup mode. The first level of judgment prioritizes metals, the second level prioritizes efficiency, and the third level provides a safety net. The specific process of this adaptive hierarchical strategy is as follows: S21. Obtain the attribute feature vector of the FOD target output in step S15. ; S22, Determine the attribute feature vector Material type of FOD target Is it a ferromagnetic metal? Then, a magnetic attraction mode command is generated, which can avoid the rigid gripping from scratching the pavement lights; S23. If the material type of the FOD target is determined to be non-metallic, then calculate the shape factor of the FOD target. : ; in, For the volume of the FOD target, The base area of ​​the FOD target. The height of the FOD target; The smaller the value, the flatter the FOD. The larger the value, the more three-dimensional the FOD (Form of Dimensional Object). S24. If the volume of the FOD target Less than the preset volume threshold and shape factor If the value is less than the preset value and the FOD target indicator is flat, then the FOD target is determined to be a lightweight floating object, a suction mode command is generated, and based on... Calculating the power of a turbofan vacuum pump : ; in, This is a proportionality coefficient, obtained through prior calibration. This is the base power of the turbofan vacuum pump; S25. If the volume of the FOD target Greater than the preset volume threshold or shape factor If the value is greater than the preset value and the FOD target indication is blocky, then the FOD target is determined to be a heavy foreign object, and a grab mode instruction is generated.

[0036] S3. The multi-degree-of-freedom collaborative robotic arm adjusts its posture to drive the composite cleaning end effector to perform the operation according to the matching cleaning mode; In the default cruise mode, the end of the robotic arm maintains a retracted posture with an angle of 0° to the horizontal direction and a retraction stroke of 50mm to prevent accidental collision with the pavement lights; When performing cleaning in magnetic attraction mode, the action logic is as follows: The multi-degree-of-freedom collaborative robotic arm adjusts its posture to align the ring-shaped electromagnetic adsorption array with the FOD target, descends to a distance of 1-2 cm from the FOD target, and the excitation coil is energized to generate a strong magnetic field to attract foreign objects. It then moves above the recovery box, where a reverse pulse current is applied to demagnetize the object, causing the FOD target to detach. The magnetic attraction force follows Maxwell's law of electromagnetic attraction. ,in The air gap magnetic induction intensity The cross-sectional area of ​​the magnetic poles. Given the free permeability, the system modulates the excitation current using PWM. Thus controlling the magnetic induction intensity ( This allows the adsorption force to be dynamically adjusted, enabling it to firmly adsorb heavy bolts while also using weak magnetic force to attract small iron filings for easy removal.

[0037] When performing cleaning in suction mode, the action logic is as follows: the multi-degree-of-freedom collaborative robotic arm adjusts its posture to vertically align the high-negative-pressure pneumatic suction port with the pavement surface, descends to a height of 3-5cm above the ground, the flexible silicone skirt adheres to the pavement surface to reduce leakage, the turbofan vacuum pump is activated to generate negative pressure to suck up the FOD target, and after the differential pressure sensor confirms successful suction, it moves above the recovery tank and stops pumping to release the FOD. The pneumatic suction capability is based on Bernoulli's principle and differential pressure flow dynamics formulas. ; in Standard atmospheric pressure The internal absolute pressure generated by the turbofan vacuum pump The effective projected contact area between the suction port and the foreign object. In this embodiment, the leakage loss coefficient due to pavement roughness is... Controlled by adjusting the fan speed It can achieve precise pickup of objects with different surface densities.

[0038] When performing the gripping mode for cleaning, the action logic is as follows: the flexible mechanical gripper adjusts its opening width according to the FOD target contour, descends to envelop the FOD target, and the flexible fingers utilize the damping properties and deformation capabilities of their material to adhere to the surface of the foreign object, performing a closing gripping action, followed by lifting and flipping actions to prevent slippage, and finally releasing it above the recycling bin; the fingers adopt a segmented air chamber structure with a bending angle With driving air pressure Exhibiting a nonlinear positive correlation, upon contact with a foreign object, the flexible finger surface utilizes the microscopic deformation of the high-damping material to increase the static friction coefficient. To ensure gripping force when grasping irregular objects in the envelope. It generates static friction force sufficient to overcome gravity, thereby enabling reliable removal of stubborn foreign objects without damaging the pavement lights.

[0039] S4. After a single cleaning operation is completed, the multi-source heterogeneous sensing unit is invoked to perform a second scan of the work area to determine whether there is any FOD residue. If residue is found, the policy upgrade mechanism is triggered to perform a second cleaning. The specific process is as follows: S41. Calculate the difference norm of the images of the work area before and after cleaning. : ; in, To clean up the image data of the work area, Image data for the pre-cleaning work area; preferably, the image data is limited to the local ROI region image where the FOD target is located and has undergone pixel-level registration, with a difference norm. The calculations are performed only within the local ROI region; calculations using the entire work area image are prohibited. S42. Judge the cleaning effect; if... Greater than the preset success threshold If so, it is determined that the FOD has been cleared; if Less than the preset success threshold If FOD remains, the policy upgrade mechanism is triggered, and the success threshold is set. The value range is set to be greater than the maximum difference caused by background environmental fluctuations. And less than the minimum difference value generated when the FOD target is completely removed. The strategy upgrade mechanism is as follows: (1) If the cleaning mode used in the last cleaning operation was suction mode, and If the power is less than the maximum power of the turbofan vacuum pump, the control command of the turbofan vacuum pump will be adjusted to full power output, and the suction mode will be executed again. (2) If the cleaning mode used in the last cleaning operation was the suction mode and the output power of the turbofan vacuum pump reached the maximum value, then upgrade to the grab mode to clean the FOD target. (3) If the cleaning mode used in the last cleaning operation was the capture mode, then stop the cleaning operation, record the current location coordinates and image, and send a manual takeover alarm to the remote control center through the edge computing platform.

[0040] Therefore, this invention provides an airport FOD detection and removal system and method based on multi-source perception fusion. It establishes an all-weather, high-precision foreign object detection defense line by employing a joint spatiotemporal fusion mechanism of lidar, millimeter-wave radar, and machine vision; utilizes an integrated pneumatic-magnetic-flexible gripping composite end effector for adaptive removal of irregularly shaped and ferromagnetic FODs through hierarchical classification; and ultimately constructs a core process based on a closed-loop operation review and dynamic strategy upgrade architecture using visual difference norm and decision tree logic. This achieves an automated control process of multi-source environmental deep perception—multi-modal adaptive operation—visual feedback closed-loop error correction, solving the problems of existing... Airport cleaning equipment suffers from technical pain points such as high missed detection rates under complex weather conditions, low success rates in cleaning irregularly shaped heavy objects, and rigid operation modes due to reliance on a single visual sensor or a single suction mechanism. This solution improves the accuracy of target identification, the coverage of cleaning operations, and the reliability of task execution in complex pavement environments. It significantly suppresses the risk of secondary FOD caused by incomplete cleaning, enhances the equipment's adaptability to irregular foreign objects and dynamic weather changes, and provides an all-weather, intelligent, and self-correcting solution for foreign object management operations that require high efficiency and high safety in the field of smart airport and flight area pavement maintenance.

[0041] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An airport FOD detection and cleanup system based on multi-source sensing fusion, characterized in that, include: The mobile transport chassis serves as the main load-bearing component of the system. A multi-source heterogeneous sensing unit is installed at the front end of the mobile transport chassis to collect multi-dimensional environmental data of the airport pavement. A multi-degree-of-freedom collaborative robotic arm is positioned in the middle of the mobile transport chassis; A composite cleaning end effector is disposed at the end of the multi-degree-of-freedom collaborative robotic arm. The composite cleaning end effector includes three functional modules: a high negative pressure pneumatic suction port, a ring electromagnetic adsorption array, and a flexible mechanical gripper. The edge computing platform is communicatively connected to the mobile transport chassis, the multi-source heterogeneous sensing unit, the multi-degree-of-freedom collaborative robotic arm, and the composite cleaning end effector, and is used to perform data fusion calculation and issue operation control commands.

2. The airport FOD detection and cleanup system based on multi-source sensing fusion according to claim 1, characterized in that: The multi-source heterogeneous sensing unit is fixedly mounted on the front end of the mobile transport chassis by a rigid bracket; The multi-source heterogeneous sensing unit includes: A lidar unit, installed on the top layer of the rigid support, is used to acquire three-dimensional point cloud data of the environment. A millimeter-wave radar is installed at the center of the bottom layer of the rigid support for long-range detection and motion compensation. A binocular industrial camera, symmetrically distributed on both sides of the millimeter-wave radar, is used to acquire high-resolution texture images; The optical center positions of the lidar, millimeter-wave radar, and binocular industrial camera were jointly calibrated.

3. The airport FOD detection and cleanup system based on multi-source sensing fusion according to claim 1, characterized in that: The three functional modules of the composite cleaning end effector are independently controlled and adaptable to the switching cleaning of different FODs. The high negative pressure pneumatic suction port is located at the center of the composite cleaning end effector and is connected to the turbofan vacuum pump inside the mobile transport chassis through a corrugated hose. The bottom of the high negative pressure pneumatic suction port is provided with a flexible silicone skirt. The annular electromagnetic adsorption array is arranged around the outside of the high negative pressure pneumatic suction port and encapsulates several sets of independently controlled excitation coils inside. The flexible mechanical claw is located on the front side of the composite cleaning end effector. The finger body of the flexible mechanical claw is made of pneumatic variable stiffness material and the fingertip surface is provided with a high friction coefficient coating.

4. The airport FOD detection and cleanup system based on multi-source sensing fusion according to claim 1, characterized in that, The edge computing platform is equipped with a fusion tracking module for calculating the target location of FOD (Foreign Object Demand). The calculation logic includes: (1) Establish the state and observation model of the linear discrete system. The state equation is expressed as: ； in, For FOD target The state vector at time step includes planar coordinates Horizontal speed , This is the state transition matrix, which describes the motion relationship of the FOD target between two adjacent frames. For process noise, describe the random disturbances in the motion of the FOD target; The observation equation is expressed as: ； in, The observed coordinates of the FOD target in the image. This is the observation matrix, used to map the three-dimensional state space of the FOD target to the two-dimensional image observation space. To observe noise; (2) Predict the current location of the FOD target based on its historical trajectory: ； ； in, for The prior state prediction value of the FOD target at time step. This is the prior covariance matrix, used to describe the uncertainty of the predicted values. For process noise covariance; (3) Calculate the Kalman gain : ； in, To observe the noise covariance; (4) Using the current observation coordinates and Kalman gain After correcting the prediction error, the final optimal state estimate is obtained: ； ； That is, characterization The optimal coordinates of the FOD target at any given time.

5. A method for detecting and removing FOD (Foreign Object Destruction) at airports based on multi-source sensing fusion, characterized in that, The method applied to the airport FOD detection and cleanup system based on multi-source sensing fusion as described in any one of claims 1-4 includes the following steps: S1. Multi-dimensional environmental data of the work area is collected using a multi-source heterogeneous sensing unit. After spatiotemporal alignment and data fusion processing by an edge computing platform, the attribute features of the FOD target are calculated. The attribute features include spatial location, material type, and geometric parameters. S2. Based on the attribute characteristics of the FOD target, the edge computing platform adopts an adaptive hierarchical strategy to match the corresponding cleanup mode; S3. The multi-degree-of-freedom collaborative robotic arm adjusts its posture to drive the composite cleaning end effector to perform the operation according to the matching cleaning mode; S4. After a single cleaning operation is completed, the multi-source heterogeneous sensing unit is invoked to perform a second scan of the work area to determine whether there is FOD residue in the work area. If residue is determined to exist, the strategy upgrade mechanism is triggered to perform a second cleaning.

6. The airport FOD detection and cleanup method based on multi-source sensing fusion according to claim 5, characterized in that, The specific method for spatiotemporal alignment in step S1 is as follows: Construct a joint calibration model based on rigid transformation, and use the pre-calibrated extrinsic matrix and camera intrinsic matrix to project the 3D point cloud data collected by the lidar and the target data collected by the millimeter-wave radar onto the pixel coordinate system of the binocular industrial camera. The specific formula is as follows: ； in, This is a scale factor used to correct the scaling ratio of three-dimensional coordinates projected onto a two-dimensional plane. Here are the pixel coordinates of FOD in the camera image. For the camera intrinsic parameter matrix, These are the rigid body rotation matrix and translation matrix of the lidar to the camera, respectively. The coordinates of FOD in the lidar coordinate system are given.

7. The airport FOD detection and cleanup method based on multi-source sensing fusion according to claim 6, characterized in that, The data fusion processing steps in step S1 include: S11. Use a deep learning object detection network to extract the observation coordinates of FOD targets in the visual image. ; S12. The spatial position of the FOD target is smoothed using a state estimation algorithm to obtain the optimal coordinates of the FOD target. The calculation logic of the state estimation algorithm includes: (1) Establish the state and observation model of the linear discrete system. The state equation is expressed as: ； in, For FOD target The state vector at time step includes planar coordinates Horizontal speed , This is the state transition matrix, which describes the motion relationship of the FOD target between two adjacent frames. For process noise, describe the random disturbances in the motion of the FOD target; The observation equation is expressed as: ； in, The observed coordinates of the FOD target in the image. This is the observation matrix, used to map the three-dimensional state space of the FOD target to the two-dimensional image observation space. To observe noise; (2) Predict the current location of the FOD target based on its historical trajectory: ； ； in, for The prior state prediction value of the FOD target at time step. This is the prior covariance matrix, used to describe the uncertainty of the predicted values. For process noise covariance; (3) Calculate the Kalman gain : ； in, To observe the noise covariance; (4) Using the current observation coordinates and Kalman gain After correcting the prediction error, the final optimal state estimate is obtained: ； ； That is, characterization The optimal coordinates of the FOD target at any given time; S13. Based on the optimal coordinates of the FOD target, extract the point cloud data of the corresponding spatial region from the lidar point cloud. After removing the road background point cloud through background filtering, obtain the exclusive point cloud cluster of the FOD target. Solve the geometric parameters of the FOD target, including volume, bottom area, height and length and width parameters. S14. Extract the radar reflectivity features of the FOD target location detected by millimeter-wave radar and the texture and spectral features of the FOD target location corresponding to the binocular industrial camera. Input the multi-source features into the pre-trained material classification model to determine the material type features of the FOD target. The material type features include two categories: ferromagnetic metals and non-metals. S15. Encapsulate the calculated spatial location, geometric parameters, and material type features to generate the attribute feature vector of the FOD target. .

8. The airport FOD detection and cleanup method based on multi-source sensing fusion according to claim 7, characterized in that, The specific process of the adaptive hierarchical strategy in step S2 is as follows: S21. Obtain the attribute feature vector of the FOD target output in step S15. ; S22, Determine the attribute feature vector Material type of FOD target Is it a ferromagnetic metal? Then a magnetic attraction mode command is generated; S23. If the material type of the FOD target is determined to be non-metallic, then calculate the shape factor of the FOD target. : ； in, For the volume of the FOD target, The base area of ​​the FOD target. The height of the FOD target; S24. If the volume of the FOD target Less than the preset volume threshold and shape factor If the value is less than the preset value and the FOD target indicator is flat, then the FOD target is determined to be a lightweight floating object, a suction mode command is generated, and based on... Calculating the power of a turbofan vacuum pump : ； in, This is a proportionality coefficient, obtained through prior calibration. This is the base power of the turbofan vacuum pump; S25. If the volume of the FOD target Greater than the preset volume threshold or shape factor If the value is greater than the preset value and the FOD target indication is blocky, then the FOD target is determined to be a heavy foreign object, and a grab mode instruction is generated.

9. The airport FOD detection and cleanup method based on multi-source sensing fusion according to claim 8, characterized in that, The action logic for performing the magnetic attraction mode for cleaning in step S3 is as follows: the multi-degree-of-freedom collaborative robotic arm adjusts its posture to align the ring electromagnetic adsorption array with the FOD target, descends to a predetermined distance from the FOD target, the excitation coil is energized to generate a strong magnetic field to attract foreign objects, and then moves to the top of the recycling box, and a reverse pulse current is introduced to demagnetize it, causing the FOD target to fall off. The action logic for cleaning in suction mode is as follows: the multi-degree-of-freedom collaborative robotic arm adjusts its posture so that the high negative pressure pneumatic suction port is vertically aligned with the pavement surface, descends to a predetermined height above the ground, the flexible silicone skirt fits the pavement surface to reduce leakage, the turbofan vacuum pump is started to generate negative pressure to suck up the FOD target, and after the differential pressure sensor confirms that the suction is successful, it moves to the top of the recovery box and stops the pump to release. The action logic for performing the gripping mode for cleaning is as follows: the flexible mechanical claw adjusts its opening range according to the FOD target contour, extends downward to envelop the FOD target, and the flexible fingers use the damping characteristics and deformation ability of their material to adhere to the surface of the foreign object, perform a closing gripping action, and then perform lifting and flipping actions to prevent slippage, and move to the top of the recycling bin to release.

10. The airport FOD detection and cleanup method based on multi-source sensing fusion according to claim 9, characterized in that, The specific process of step S4 is as follows: S41. Calculate the difference norm of the images of the work area before and after cleaning. : ； in, To clean up the image data of the work area, Image data for the pre-cleaning work area; the image data is limited to the local ROI region image where the FOD target is located and has undergone pixel-level registration, with a difference norm. The calculations are performed only within this local ROI region; S42. Judge the cleaning effect; if... Greater than the preset success threshold If so, it is determined that the FOD has been cleared; if Less than the preset success threshold If FOD remains, a policy upgrade mechanism is triggered; the success threshold is then determined. The value range is set to be greater than the maximum difference caused by background environmental fluctuations. And less than the minimum difference value generated when the FOD target is completely removed. The strategy upgrade mechanism is as follows: (1) If the cleaning mode used in the last cleaning operation was suction mode, and If the power is less than the maximum power of the turbofan vacuum pump, the control command of the turbofan vacuum pump will be adjusted to full power output, and the suction mode will be executed again. (2) If the cleaning mode used in the last cleaning operation was the suction mode and the output power of the turbofan vacuum pump reached the maximum value, then upgrade to the grab mode to clean the FOD target. (3) If the cleaning mode used in the last cleaning operation was the capture mode, then stop the cleaning operation, record the current location coordinates and image, and send a manual takeover alarm to the remote control center through the edge computing platform.

Images