An airport runway foreign matter detection method and device based on wide-line structured light three-dimensional reconstruction

Abstract

The application discloses an airport runway foreign matter detection method and device based on wide-line structured light three-dimensional reconstruction. The method comprises the following steps: controlling a mobile carrier to move along an airport runway, acquiring wide-line structured light images collected by a line structured light measurement system carried on the mobile carrier during the movement of the mobile carrier along the airport runway and displacement data of the mobile carrier; extracting a line structured light center line; performing three-dimensional reconstruction, combining the displacement data, and generating three-dimensional point cloud data of a surface of the airport runway; fitting a reference surface of the airport runway and extracting a foreign matter candidate area; performing multi-modal feature analysis and three-dimensional geometric size calculation on the foreign matter candidate area, determining foreign matter detection and recognition results, and outputting position and size information of the foreign matter. The application can ensure the three-dimensional reconstruction accuracy, realize a large single detection coverage width, effectively reduce the scanning times, improve the airport runway foreign matter detection efficiency, and has the advantages of fast detection speed, large coverage range, strong adaptability and the like.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional measurement technology, specifically a method and device for detecting foreign objects on airport runways based on wide-width line structured light three-dimensional reconstruction. Background Technology

[0002] Foreign objects (FOOs) on airport runways refer to any foreign material present on airport runways, taxiways, or aprons that may cause damage to aircraft. Timely detection and removal of FEOs on airport runways are closely related to aviation operational safety. Even small FEOs on the runway surface can reflect potential safety hazards in the flight environment; therefore, high-precision FEO detection research has always been a key research direction in this field. Using non-contact optical scanning methods to measure the height and location of unknown FEOs can improve detection accuracy from a fundamental perspective. It also avoids the shortcomings of traditional manual inspection methods, such as low efficiency, high false negative rates, and disruption to normal runway operation. This method is currently one of the most effective detection techniques used in airport safety assurance.

[0003] Currently, runway foreign object detection mainly employs technologies such as machine vision, lidar, millimeter-wave radar, infrared imaging, and line structured light. Among these, two-dimensional machine vision relies primarily on the target's color, texture, and shape features, making it susceptible to interference from ambient lighting, runway texture, and background. Its ability to detect small objects or those with colors similar to the runway surface is limited. Lidar and millimeter-wave radar can acquire spatial information about targets, but they suffer from high equipment costs, large size, complex installation and deployment, and limited spatial resolution, hindering the widespread application of mobile detection equipment. Currently, line structured light-based airport runway foreign object detection methods primarily detect foreign objects by analyzing the deformation, bending, or grayscale changes of laser stripes. They determine the location and shape of the foreign object based on the local deformation of the laser stripes at the object. While these methods are simple to implement and computationally efficient, they lack three-dimensional height information due to their reliance on two-dimensional image information, resulting in the following shortcomings in practical applications:

[0004] (1) The detection accuracy is greatly affected by the features of two-dimensional images. Existing methods mainly rely on the bending degree of laser stripes or changes in image grayscale for detection, which makes it difficult to accurately reflect the true spatial size and height information of foreign objects, and the detection accuracy is limited for foreign objects that are small in size or have complex shapes.

[0005] (2) The false alarm rate is relatively high. The center line, marking lines, cracks, joints and local wear on the runway surface may cause changes in the shape of the laser stripes, which can be easily misjudged as foreign objects, affecting the reliability of the detection results.

[0006] (3) Limited detection coverage. Existing line structured light inspection devices mostly use narrow-band line laser irradiation, which results in a small coverage width for a single inspection. To achieve the inspection of the entire runway, multiple scans are required, leading to low detection efficiency and making it difficult to meet the needs of rapid inspection of airport runways.

[0007] (4) Insufficient system integration. Some detection systems use a combination of multiple sensors or large detection equipment, which have problems such as large equipment size, complex installation and debugging, and inconvenient mobile deployment, which are not conducive to engineering applications on mobile detection platforms.

[0008] Furthermore, existing line structured light 3D reconstruction algorithms are typically designed for small to medium field of view or local measurement scenarios, employing conventional camera intrinsic parameter calibration and distortion correction methods. When a line structured light detection system needs to achieve large-area, wide-swath imaging by expanding the lens field of view and laser projection range, traditional calibration methods struggle to simultaneously ensure both calibration accuracy and 3D reconstruction accuracy across the entire imaging field of view, specifically exhibiting the following shortcomings:

[0009] (1) Traditional line structured light systems typically perform camera calibration for a relatively limited imaging area, with calibration feature points mainly concentrated in the image center or local areas. When the camera needs to cover a large lateral detection range, there are differences in the degree of imaging distortion between the image center and edge areas. If local calibration data is used or correction is performed based solely on a unified theoretical distortion model, it is difficult to fully describe the pixel position error within the entire wide imaging area, leading to a decrease in calibration accuracy in the edge areas.

[0010] (2) Traditional calibration results usually lack error compensation mechanisms for edge regions. Existing line structured light 3D reconstruction methods typically perform distortion correction on the laser centerline directly after obtaining camera distortion parameters, and then perform triangulation based on the corrected pixel coordinates. They lack a processing mechanism to establish local distortion residuals for different lateral field-of-view positions and perform further compensation. Therefore, when the detection range is expanded to a wide field of view, it is difficult to effectively reduce the 3D measurement errors caused by lens distortion, pixel projection errors, and edge region imaging errors. Summary of the Invention

[0011] The purpose of this invention is to address the defects or deficiencies of the existing technology by providing a method and device for detecting foreign objects on airport runways based on wide-width line structured light three-dimensional reconstruction.

[0012] The technical solution to achieve the objective of this invention is as follows: On the one hand, a method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction is provided, the method comprising the following steps:

[0013] Step 1: Control the mobile carrier to move along the airport runway, and acquire the wide-band structured light image and displacement data of the mobile carrier collected by the structured light measurement system carried on the mobile carrier during the movement along the airport runway.

[0014] Step 2: Preprocess the line structured light image and extract the center line of the line structured light;

[0015] Step 3: Perform three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the center line of the line structured light, and combine the displacement data to generate three-dimensional point cloud data of the airport runway surface;

[0016] Step 4: Fit the reference surface of the airport runway based on the three-dimensional point cloud data, and extract the abnormal protrusion areas above the reference surface to obtain the foreign object candidate areas.

[0017] Step 5: Perform multimodal feature analysis and three-dimensional geometric dimension calculation on the candidate foreign object region to determine the foreign object detection and identification results, and output the location and size information of the foreign object.

[0018] Further, step 2 involves preprocessing the line structured light image and extracting the center line of the line structured light, specifically including:

[0019] Step 2-1: The line structured light image is pre-processed by binarization using a dynamic threshold, wherein the dynamic threshold is adaptively determined based on the grayscale distribution of each column of the image to extract the line structured light region of the corresponding column.

[0020] Step 2-2: Calculate the sub-pixel coordinates of the center of each column of the extracted structured light region using the gray-scale centroid method, and connect the sub-pixel coordinates of each column to form a continuous structured light center line.

[0021] Furthermore, the dynamic threshold is determined based on the statistical distribution of gray values ​​in each column, and the gray value corresponding to the 90% to 95% quantile of the gray values ​​in the corresponding column is selected as the dynamic threshold of that column.

[0022] Furthermore, the line structured light measurement system includes: a projection unit for projecting wide-band line structured light onto the runway surface; and an image acquisition unit for synchronously acquiring line structured light images of the runway surface.

[0023] Furthermore, the system calibration parameters mentioned in step 3 include the intrinsic parameter matrix, distortion parameters, extrinsic parameter matrix of the image acquisition unit, and the laser plane spatial equation in the coordinate system of the image acquisition unit;

[0024] Step 3, which involves performing three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the centerline of the structured light to generate three-dimensional point cloud data of the airport runway surface, specifically includes:

[0025] Step 3-1: Extract the sub-pixel coordinates of the center line of the line structured light, and perform distortion correction processing on the sub-pixel coordinates based on the distortion parameters;

[0026] Step 3-2: Establish spatial rays from the distorted pixel coordinates to the coordinate system of the image acquisition unit based on the intrinsic parameter matrix;

[0027] Step 3-3: Calculate the coordinates of the intersection point of the spatial ray and the spatial equation of the laser plane to obtain the three-dimensional coordinate point in the coordinate system of the image acquisition unit;

[0028] Steps 3-4: Transform the three-dimensional coordinate points to the world coordinate system based on the extrinsic parameter matrix;

[0029] Steps 3-5: Based on the displacement data, perform displacement compensation, coordinate stitching, and filtering on the three-dimensional coordinate points in the world coordinate system to generate a three-dimensional topographic model of the airport runway surface, i.e., three-dimensional point cloud data.

[0030] Furthermore, the calibration of the image acquisition unit adopts a multi-position full-field-of-view calibration method, specifically including: setting multiple calibration positions within the effective imaging field of view of the image acquisition unit, moving the calibration target sequentially to the image center region, the intermediate transition region, and the left and right edge regions, and acquiring calibration images at the corresponding positions respectively, so that the calibration feature points on the calibration target cover the entire effective imaging area of ​​the image acquisition unit; and uniformly solving the intrinsic parameters and distortion parameters of the image acquisition unit based on the calibration feature points acquired at different calibration positions.

[0031] Furthermore, the plurality of calibration positions are set along the lateral direction of the imaging field of view of the image acquisition unit, including at least the left edge region, the left transition region, the center region, the right transition region, and the right edge region, so that the calibration feature points cover different lateral positions of the wide imaging area.

[0032] Furthermore, step 3-1 specifically includes:

[0033] The residual between the actual pixel coordinates and the ideal projection coordinates is calculated based on the calibration feature points collected at each calibration location, and a distortion residual mapping relationship covering the entire effective imaging area is established.

[0034] For the extracted center line of the line structured light, the corresponding distortion compensation amount is obtained from the distortion residual mapping relationship according to its image position, and the sub-pixel coordinates of the center line of the line structured light are corrected.

[0035] Specifically, for the center point of the line structured light located within a preset area at the edge of the image, a local interpolation method is used to determine the distortion compensation amount of the edge region based on the distortion residuals corresponding to adjacent calibration positions. The distortion compensation amount is then used to perform sub-pixel-level position correction on the center point of the line structured light to reduce the three-dimensional reconstruction error of the image edge region under wide-span imaging conditions.

[0036] Further, in step 4, a reference surface of the airport runway is fitted based on the three-dimensional point cloud data, and abnormal protrusion regions above the reference surface are extracted to obtain candidate foreign object regions, specifically including:

[0037] Step 4-1: Perform plane fitting on the three-dimensional point cloud data to determine the reference height of the airport runway surface;

[0038] Step 4-2: Extract points in the 3D point cloud data that are higher than the reference height as spatial outliers;

[0039] Step 4-3: Based on the three-dimensional spatial connectivity between adjacent spatial anomalies, perform regional aggregation on the spatial anomalies to construct the foreign object candidate region.

[0040] Further, step 5 involves performing multimodal feature analysis and three-dimensional geometric dimension calculation on the foreign object candidate region to determine the foreign object detection and identification results, and outputting the location and size information of the foreign object, specifically including:

[0041] Step 5-1: Extract the three-dimensional geometric features, spatial distribution features, and surface reflected light intensity features of the foreign object candidate region;

[0042] Step 5-2: Calculate the three-dimensional geometric dimensions of the foreign object candidate region based on the three-dimensional geometric features, including length, width, height, and spatial three-dimensional coordinates;

[0043] Step 5-3: Perform feature analysis by combining the three-dimensional morphological features, spatial distribution features and reflected light intensity features of the foreign object candidate region, identify and determine the foreign object category, and output the location and size information of the foreign object in the airport runway.

[0044] On the other hand, an airport runway foreign object detection device based on wide-band structured light 3D reconstruction is provided. The device includes a mobile carrier, a mounting bracket, a projection unit, an image acquisition unit, and a processing and control unit.

[0045] The mobile carrier is used to carry the line structured light measurement system and move along the runway of the airport to be measured.

[0046] The mounting bracket is disposed on the mobile carrier;

[0047] The projection unit is mounted on the mounting bracket and is used to project a wide-band structured light onto the surface of the airport runway to form a laser light strip covering the runway detection area.

[0048] The image acquisition unit is mounted on the mounting bracket and forms a line structured light triangulation structure with the projection unit, which is used to synchronously acquire line structured light images of the airport runway surface.

[0049] The processing control unit is communicatively connected to the projection unit and the image acquisition unit, respectively, and is used to control the projection unit and the image acquisition unit to work together, receive the line structured light image and the displacement data of the moving carrier, and execute the airport runway foreign object detection method.

[0050] Furthermore, the projection unit includes a laser emission source and a beam shaping and expanding element:

[0051] The beam shaping and expanding element is disposed in the output optical path of the laser emission source and is used to expand and homogenize the beam emitted by the laser emission source to generate a wide-band linear laser beam.

[0052] Furthermore, the beam shaping and expanding element is a Powell prism, and the laser emission source is a semiconductor laser; the divergence angle of the Powell prism is 80° to 120°, and the line structured light projected onto the runway surface covers an area with a width of 5m to 7m in a single pass.

[0053] Furthermore, the mounting bracket includes a projection unit mounting surface and an image acquisition unit mounting surface;

[0054] The projection unit is fixedly installed on the projection unit mounting surface. The normal direction of the projection unit mounting surface has a first preset angle with the vertical direction, and the projection unit has a first preset installation height relative to the runway surface.

[0055] The image acquisition unit is fixedly installed on the mounting surface of the acquisition unit. The normal direction of the mounting surface of the acquisition unit has a second preset angle with the vertical direction, and the image acquisition unit has a second preset installation height relative to the runway surface.

[0056] Furthermore, the first preset included angle is 55° to 65°, and the first preset installation height is 0.9m to 1.1m;

[0057] The second preset included angle is 45° to 55°, and the second preset installation height is 1.7m to 2.0m.

[0058] Furthermore, the projection plane of the projection unit intersects with the optical axis of the image acquisition unit to form a spatial three-dimensional triangulation layout.

[0059] Furthermore, the image acquisition unit has a resolution of 5120×300 pixels, a maximum acquisition frame rate of 3300fps, and is equipped with an imaging lens with a focal length of 8mm.

[0060] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the airport runway foreign object detection method based on wide-width line structured light three-dimensional reconstruction.

[0061] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the airport runway foreign object detection method based on wide-width line structured light three-dimensional reconstruction.

[0062] Compared with the prior art, the significant advantages of this invention are:

[0063] (1) Improve the accuracy of foreign object detection. This invention adopts wide-span line structured light three-dimensional reconstruction technology, establishes a three-dimensional topographic model of the airport runway surface through system calibration parameters, and uses the spatial height information of foreign objects relative to the runway reference surface for detection. Compared with existing detection methods based on laser stripe bending or two-dimensional image grayscale changes, it can more accurately obtain the spatial size and height information of foreign objects, thus improving the detection accuracy of small-sized foreign objects and complex-shaped foreign objects.

[0064] (2) Reduce the false alarm rate of foreign objects. The present invention detects foreign objects based on the three-dimensional point cloud or height information of the runway surface. It uses the spatial geometric features and reflection features of foreign objects as multi-modal features as the discrimination criteria, which effectively reduces the interference of two-dimensional texture features such as runway centerline, marking lines, cracks, joints and local wear on the detection results, thereby reducing the false alarm probability and improving the reliability of the detection results.

[0065] (3) Expand the detection coverage and improve detection efficiency. This invention adopts a wide-span line structured light illumination method and combines it with a large field of view high-speed industrial camera to achieve one-time coverage detection of a wide area of ​​the airport runway. While ensuring the accuracy of three-dimensional reconstruction, it effectively expands the coverage of a single detection, reduces the number of scans during the detection process, improves the detection efficiency of foreign objects on the airport runway, and meets the needs of rapid inspection of the airport runway.

[0066] (4) Improve system integration and engineering application capabilities. This invention integrates a wide-band laser, a high-speed industrial camera, and a host computer into a mobile carrier (mobile load vehicle), and constructs a fixed triangulation structure through a mounting bracket to achieve an integrated design of the detection system. Compared with large radar equipment or multi-sensor detection systems, this invention has a simple structure, high equipment integration, and convenient deployment, and is applicable to foreign object detection scenarios in small and medium-sized airports and mobile airport runways.

[0067] (5) Improve the accuracy of 3D reconstruction of edge regions under wide-field imaging conditions. The present invention adopts a multi-position full-field calibration method, which moves the calibration target to the center region, the intermediate transition region and the left and right edge regions of the image in sequence, so that the calibration feature points cover the entire wide-field imaging view. Compared with the traditional local calibration or single calibration region calibration method, it can more fully characterize the imaging distortion at different field positions under wide-field imaging conditions. Furthermore, by establishing the distortion correction and residual compensation relationship, the center line of the line structure light is corrected at the sub-pixel level, which reduces the distortion error in the image edge region and the triangulation measurement error caused therefrom. Thus, while expanding the detection coverage, the consistency of 3D reconstruction accuracy in the entire wide field of view is improved.

[0068] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the structure of an airport runway foreign object detection device in one embodiment.

[0070] Figure 2 This is a flowchart of an airport runway foreign object detection method in one embodiment.

[0071] Figure 3 This is a laser projection image of an airport runway without foreign objects in one embodiment.

[0072] Figure 4 This is a laser projection image of an airport runway with foreign objects in one embodiment.

[0073] Figure 5 This is a diagram of a runway foreign object scenario in one embodiment.

[0074] Figure 6 This is a three-dimensional point cloud image of an airport runway in one embodiment. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0076] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0077] In one embodiment, combined Figure 2 This paper provides a method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction. The method includes the following steps:

[0078] Step 1: Control the mobile carrier to move along the airport runway, and acquire the wide-band structured light image and displacement data of the mobile carrier collected by the structured light measurement system carried on the mobile carrier during the movement along the airport runway.

[0079] Here, the mobile carrier refers to, but is not limited to, mobile load vehicles, etc.

[0080] Preferably, in some embodiments, the line structured light measurement system includes: a projection unit for projecting wide-band line structured light onto the runway surface; and an image acquisition unit for synchronously acquiring line structured light images of the runway surface.

[0081] Preferably, in some embodiments, the projection unit uses, but is not limited to, a wide-band laser, and the image acquisition unit uses, but is not limited to, a high-speed industrial camera.

[0082] Step 2: Preprocess the line structured light image and extract the center line of the line structured light;

[0083] like Figure 3 As shown, when there are no foreign objects on the airport runway surface, wide-band structured light forms continuous and straight laser stripes on the runway surface; as Figure 4 As shown, when there are foreign objects on the surface of the airport runway, the laser stripes are locally shifted and deformed at the foreign object. By extracting the center line of the structured light and combining it with the system calibration parameters for triangulation, the three-dimensional spatial coordinates of the foreign object area can be recovered.

[0084] Step 3: Perform three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the center line of the line structured light, and combine the displacement data to generate three-dimensional point cloud data of the airport runway surface;

[0085] Step 4: Fit the reference surface of the airport runway based on the three-dimensional point cloud data, and extract the abnormal protrusion areas above the reference surface to obtain the foreign object candidate areas.

[0086] Step 5: Perform multimodal feature analysis and three-dimensional geometric dimension calculation on the candidate foreign object region to determine the foreign object detection and identification results, and output the location and size information of the foreign object.

[0087] Furthermore, in one embodiment, step 2 involves preprocessing the line structured light image to reduce the impact of ambient lighting changes and image noise on the accuracy of laser stripe extraction, and extracting the center line of the line structured light, specifically including:

[0088] Step 2-1: The line structured light image is pre-processed by binarization using a dynamic threshold, wherein the dynamic threshold is adaptively determined based on the grayscale distribution of each column of the image to extract the line structured light region of the corresponding column.

[0089] Step 2-2: Calculate the sub-pixel coordinates of the center of each column of the extracted structured light region using the gray-scale centroid method, and connect the sub-pixel coordinates of each column to form a continuous structured light center line.

[0090] Here, compared to directly using pixel-level center localization methods, the grayscale centroid method can effectively improve the center extraction accuracy of line structured light, providing accurate feature point data for subsequent 3D reconstruction.

[0091] More preferably, in some embodiments, the dynamic threshold is determined based on the statistical distribution of gray values ​​in each column to adapt to brightness changes in different regions and improve the extraction accuracy of line structured light regions. The gray value corresponding to the 90% to 95% quantile of the gray value in the corresponding column is selected as the dynamic threshold of that column.

[0092] Furthermore, in one embodiment, step 3 employs a three-dimensional reconstruction method based on the principle of line structured light triangulation. The system calibration parameters in step 3 include the intrinsic parameter matrix, distortion parameters, extrinsic parameter matrix of the image acquisition unit, and the laser plane spatial equation in the coordinate system of the image acquisition unit.

[0093] Step 3, which involves performing three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the centerline of the structured light to generate three-dimensional point cloud data of the airport runway surface, specifically includes:

[0094] Step 3-1: Extract the sub-pixel coordinates of the center line of the line structured light, and perform distortion correction processing on the sub-pixel coordinates based on the distortion parameters;

[0095] Step 3-2: Establish spatial rays from the distorted pixel coordinates to the coordinate system of the image acquisition unit based on the intrinsic parameter matrix;

[0096] Step 3-3: Calculate the coordinates of the intersection point of the spatial ray and the spatial equation of the laser plane to obtain the three-dimensional coordinate point in the coordinate system of the image acquisition unit;

[0097] Steps 3-4: Transform the three-dimensional coordinate points to the world coordinate system based on the extrinsic parameter matrix;

[0098] Steps 3-5: Based on the displacement data, perform displacement compensation, coordinate stitching, and filtering on the three-dimensional coordinate points in the world coordinate system to generate a three-dimensional topographic model of the airport runway surface, i.e., three-dimensional point cloud data.

[0099] More preferably, in some embodiments, the calibration of the image acquisition unit adopts a multi-position full-field-of-view calibration method, specifically including: setting multiple calibration positions within the effective imaging field of view of the image acquisition unit, moving the calibration target sequentially to the image center region, the intermediate transition region, and the left and right edge regions, and acquiring calibration images at the corresponding positions respectively, so that the calibration feature points on the calibration target cover the entire effective imaging area of ​​the image acquisition unit; and uniformly solving the intrinsic parameters and distortion parameters of the image acquisition unit based on the calibration feature points acquired at different calibration positions.

[0100] More preferably, in some embodiments, the plurality of calibration positions are arranged along the lateral direction of the imaging field of view of the image acquisition unit, including at least the left edge region, the left transition region, the center region, the right transition region and the right edge region, so that the calibration feature points cover different lateral positions of the wide imaging area.

[0101] More preferably, in some embodiments, step 3-1 specifically includes:

[0102] The residual between the actual pixel coordinates and the ideal projection coordinates is calculated based on the calibration feature points collected at each calibration location, and a distortion residual mapping relationship covering the entire effective imaging area is established for distortion removal processing in 3D reconstruction.

[0103] For the extracted center line of the line structured light, the corresponding distortion compensation amount is obtained from the distortion residual mapping relationship according to its image position, and the sub-pixel coordinates of the center line of the line structured light are corrected.

[0104] Specifically, for the center point of the line structured light located within a preset area at the edge of the image, a local interpolation method is used to determine the distortion compensation amount of the edge region based on the distortion residuals corresponding to adjacent calibration positions. The distortion compensation amount is then used to perform sub-pixel-level position correction on the center point of the line structured light to reduce the three-dimensional reconstruction error of the image edge region under wide-span imaging conditions.

[0105] Furthermore, after extracting the centerline of the structured light, the image coordinates corresponding to the centerline are back-projected into spatial rays based on camera intrinsic parameters. The intersection coordinates of these spatial rays and the laser plane are then calculated to obtain the three-dimensional coordinates of the airport runway surface based on the camera coordinate system. These coordinates are then transformed by translation and rotation using an extrinsic parameter matrix to obtain the three-dimensional point coordinates based on the airport runway coordinate system. The three-dimensional coordinates of each frame are stitched together using continuous scanning data obtained by the load vehicle 1 moving along the runway to form a three-dimensional point cloud model of the airport runway surface.

[0106] Furthermore, in one embodiment, step 4, which involves fitting a reference surface of the airport runway based on the three-dimensional point cloud data and extracting abnormally protruding regions above the reference surface to obtain foreign object candidate regions, specifically includes:

[0107] Step 4-1: Perform plane fitting on the three-dimensional point cloud data to determine the reference height of the airport runway surface;

[0108] Step 4-2: Extract points in the 3D point cloud data that are higher than the reference height as spatial outliers;

[0109] Step 4-3: Based on the three-dimensional spatial connectivity between adjacent spatial anomalies, perform regional aggregation on the spatial anomalies to construct the foreign object candidate region.

[0110] Furthermore, in one embodiment, step 5 involves performing multimodal feature analysis and three-dimensional geometric dimension calculation on the foreign object candidate region to determine the foreign object detection and identification results, and outputting the location and size information of the foreign object, specifically including:

[0111] Step 5-1: Extract the three-dimensional geometric features, spatial distribution features, and surface reflected light intensity features of the foreign object candidate region;

[0112] Step 5-2: Calculate the three-dimensional geometric dimensions of the foreign object candidate region based on the three-dimensional geometric features, including length, width, height, and spatial three-dimensional coordinates;

[0113] Step 5-3: Perform feature analysis by combining the three-dimensional morphological features, spatial distribution features and reflected light intensity features of the foreign object candidate region, identify and determine the foreign object category, and output the location and size information of the foreign object in the airport runway.

[0114] In one embodiment, combined Figure 1 A foreign object detection device for airport runways based on wide-band structured light 3D reconstruction is provided. The device includes a mobile carrier (load vehicle 1), a mounting bracket 2, a projection unit 4, an image acquisition unit 3, and a processing and control unit.

[0115] The mobile carrier is used to carry the line structured light measurement system and move along the runway of the airport to be measured.

[0116] The mounting bracket is disposed on the mobile carrier;

[0117] The projection unit is mounted on the mounting bracket and is used to project a wide-band structured light onto the surface of the airport runway to form a laser light strip covering the runway detection area.

[0118] The image acquisition unit is mounted on the mounting bracket and forms a line structured light triangulation structure with the projection unit, which is used to synchronously acquire line structured light images of the airport runway surface.

[0119] The processing control unit is communicatively connected to the projection unit and the image acquisition unit, respectively, and is used to control the projection unit and the image acquisition unit to work together, receive the line structured light image and the displacement data of the moving carrier, and execute the airport runway foreign object detection method.

[0120] Furthermore, in one embodiment, the projection unit includes a laser emission source and a beam shaping and expanding element:

[0121] The beam shaping and expanding element is disposed in the output optical path of the laser emission source and is used to expand and homogenize the beam emitted by the laser emission source to generate a wide-band linear laser beam and project it onto the surface of the airport runway.

[0122] Preferably, in some embodiments, the projection unit is a wide-band line laser 4, the beam shaping and expanding element is a Powell prism, and the laser emission source is a semiconductor laser; the divergence angle of the Powell prism is 80° to 120°, and the line structured light projected onto the runway surface covers an area with a width of 5m to 7m in a single pass.

[0123] Furthermore, in one embodiment, the image acquisition unit employs, but is not limited to, a high-speed industrial camera 3. The high-speed industrial camera has an image resolution of 5120×300 pixels, a maximum image acquisition frame rate of 3300fps, and is equipped with an industrial lens with a focal length of 8mm to achieve line structured light image acquisition of a 6m wide area of ​​the airport runway and meet the 3D reconstruction accuracy requirements of 5mm foreign objects.

[0124] Furthermore, in one embodiment, the mounting bracket includes a projection unit mounting surface and an image acquisition unit mounting surface;

[0125] The projection unit is fixedly installed on the projection unit mounting surface. The normal direction of the projection unit mounting surface has a first preset angle with the vertical direction, and the projection unit has a first preset installation height relative to the runway surface.

[0126] The image acquisition unit is fixedly installed on the mounting surface of the acquisition unit. The normal direction of the mounting surface of the acquisition unit has a second preset angle with the vertical direction, and the image acquisition unit has a second preset installation height relative to the runway surface.

[0127] Preferably, in some embodiments, the first preset included angle is 55° to 65°, and the first preset installation height is 0.9m to 1.1m; the second preset included angle is 45° to 55°, and the second preset installation height is 1.7m to 2.0m.

[0128] Preferably, in some embodiments, the projection plane of the projection unit intersects with the optical axis of the image acquisition unit to form a spatial three-dimensional triangulation layout.

[0129] Combination Figure 3 As shown, wide-band structured light can cover a 6-meter wide area of ​​an airport runway.

[0130] Furthermore, the image acquisition unit has a resolution of 5120×300 pixels, a maximum acquisition frame rate of 3300fps, and is equipped with an imaging lens with a focal length of 8mm.

[0131] In one embodiment, an airport runway foreign object detection system based on wide-band structured light 3D reconstruction is provided, the system comprising:

[0132] The first module is used to: control the mobile carrier to move along the airport runway, and acquire wide-band structured light images and displacement data of the mobile carrier collected by the line structured light measurement system carried on the mobile carrier during the movement along the airport runway.

[0133] The second module is used to: preprocess the line structured light image and extract the center line of the line structured light;

[0134] The third module is used to: perform three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the center line of the line structured light, and combine the displacement data to generate three-dimensional point cloud data of the airport runway surface;

[0135] The fourth module is used to: fit the reference surface of the airport runway based on the three-dimensional point cloud data, and extract abnormal protrusion areas above the reference surface to obtain foreign object candidate areas;

[0136] The fifth module is used to perform multimodal feature analysis and three-dimensional geometric dimension calculation on the foreign object candidate region, determine the foreign object detection and identification results, and output the position and size information of the foreign object.

[0137] Specific limitations regarding the airport runway foreign object detection system based on wide-swath structured light 3D reconstruction can be found in the limitations of the airport runway foreign object detection method based on wide-swath structured light 3D reconstruction described above, and will not be repeated here. Each module in the aforementioned airport runway foreign object detection system based on wide-swath structured light 3D reconstruction can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0138] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the airport runway foreign object detection method based on wide-band structured light 3D reconstruction.

[0139] Step 1: Control the mobile carrier to move along the airport runway, and acquire the wide-band structured light image and displacement data of the mobile carrier collected by the structured light measurement system carried on the mobile carrier during the movement along the airport runway.

[0140] Step 2: Preprocess the line structured light image and extract the center line of the line structured light;

[0141] Step 3: Perform three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the center line of the line structured light, and combine the displacement data to generate three-dimensional point cloud data of the airport runway surface;

[0142] Step 4: Fit the reference surface of the airport runway based on the three-dimensional point cloud data, and extract the abnormal protrusion areas above the reference surface to obtain the foreign object candidate areas.

[0143] Step 5: Perform multimodal feature analysis and three-dimensional geometric dimension calculation on the candidate foreign object region to determine the foreign object detection and identification results, and output the location and size information of the foreign object.

[0144] For specific limitations on each step, please refer to the limitations on the airport runway foreign object detection method based on wide-width line structured light 3D reconstruction mentioned above, which will not be repeated here.

[0145] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the airport runway foreign object detection method based on wide-band structured light 3D reconstruction.

[0146] Step 1: Control the mobile carrier to move along the airport runway, and acquire the wide-band structured light image and displacement data of the mobile carrier collected by the structured light measurement system carried on the mobile carrier during the movement along the airport runway.

[0147] Step 2: Preprocess the line structured light image and extract the center line of the line structured light;

[0148] Step 3: Perform three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the center line of the line structured light, and combine the displacement data to generate three-dimensional point cloud data of the airport runway surface;

[0149] Step 4: Fit the reference surface of the airport runway based on the three-dimensional point cloud data, and extract the abnormal protrusion areas above the reference surface to obtain the foreign object candidate areas.

[0150] Step 5: Perform multimodal feature analysis and three-dimensional geometric dimension calculation on the candidate foreign object region to determine the foreign object detection and identification results, and output the location and size information of the foreign object.

[0151] For specific limitations on each step, please refer to the limitations on the airport runway foreign object detection method based on wide-width line structured light 3D reconstruction mentioned above, which will not be repeated here.

[0152] Combination Figure 5 The measurement example involves measuring multiple gauge blocks of different sizes and heights using the apparatus and method of this embodiment. From left to right, the heights of the three gauge blocks are 20mm, 10mm, and 5mm, respectively.

[0153] Combination Figure 6 Based on the extracted centerline of the structured light beam and the system calibration parameters, the intersection point of the spatial ray and the laser plane is calculated to obtain... Figure 5 The measurement example shown is a 3D point cloud of the airport runway surface. Further planar fitting is performed on the 3D point cloud to extract abnormal protrusions above the runway reference surface. The length, width, height, and spatial location of the foreign object are calculated, and foreign object detection and localization are completed. The point cloud corresponding to the foreign object is selected and its height is displayed.

[0154] In summary, this invention employs wide-band structured light 3D reconstruction technology, which enables continuous detection over a width of approximately 6 m while simultaneously completing 3D reconstruction and detection of foreign objects on airport runways of 5 mm or larger. It boasts advantages such as wide detection coverage, high detection accuracy, low false alarm rate, and high system integration.

[0155] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction, characterized in that, The method includes the following steps: Step 1: Control the mobile carrier to move along the airport runway, and acquire the wide-band structured light image and displacement data of the mobile carrier collected by the structured light measurement system carried on the mobile carrier during the movement along the airport runway. Step 2: Preprocess the line structured light image and extract the center line of the line structured light; Step 3: Perform three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the center line of the line structured light, and combine the displacement data to generate three-dimensional point cloud data of the airport runway surface; Step 4: Fit the reference surface of the airport runway based on the three-dimensional point cloud data, and extract the abnormal protrusion areas above the reference surface to obtain the foreign object candidate areas. Step 5: Perform multimodal feature analysis and three-dimensional geometric dimension calculation on the candidate foreign object region to determine the foreign object detection and identification results, and output the location and size information of the foreign object.

2. The method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction according to claim 1, characterized in that, Step 2 involves preprocessing the line structured light image and extracting the center line of the line structured light, specifically including: Step 2-1: The line structured light image is pre-processed by binarization using a dynamic threshold, wherein the dynamic threshold is adaptively determined based on the grayscale distribution of each column of the image to extract the line structured light region of the corresponding column. Step 2-2: Calculate the sub-pixel coordinates of the center of each column of the extracted line structured light region using the gray-scale centroid method, and connect the sub-pixel coordinates of each column to form a continuous line structured light centerline. The dynamic threshold is determined based on the statistical distribution of gray values ​​in each column, and the gray value corresponding to the 90% to 95% quantile of the gray values ​​in the corresponding column is selected as the dynamic threshold of that column.

3. The method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction according to claim 1, characterized in that, The line structured light measurement system includes: a projection unit for projecting wide-band line structured light onto the runway surface; an image acquisition unit for synchronously acquiring line structured light images of the runway surface; the system calibration parameters in step 3 include the intrinsic parameter matrix, distortion parameters, extrinsic parameter matrix of the image acquisition unit, and the laser plane space equation in the coordinate system of the image acquisition unit. The step of performing three-dimensional reconstruction based on the pre-calibrated system calibration parameters and the centerline of the line structured light to generate three-dimensional point cloud data of the airport runway surface specifically includes: Step 3-1: Extract the sub-pixel coordinates of the center line of the line structured light, and perform distortion correction processing on the sub-pixel coordinates based on the distortion parameters; Step 3-2: Establish spatial rays from the distorted pixel coordinates to the coordinate system of the image acquisition unit based on the intrinsic parameter matrix; Step 3-3: Calculate the coordinates of the intersection point of the spatial ray and the spatial equation of the laser plane to obtain the three-dimensional coordinate point in the coordinate system of the image acquisition unit; Steps 3-4: Transform the three-dimensional coordinate points to the world coordinate system based on the extrinsic parameter matrix; Steps 3-5: Based on the displacement data, perform displacement compensation, coordinate stitching, and filtering on the three-dimensional coordinate points in the world coordinate system to generate a three-dimensional topographic model of the airport runway surface, i.e., three-dimensional point cloud data.

4. The method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction according to claim 3, characterized in that, The calibration of the image acquisition unit adopts a multi-position full-field-of-view calibration method, which specifically includes: setting multiple calibration positions within the effective imaging field of view of the image acquisition unit, moving the calibration target sequentially to the image center region, the intermediate transition region, and the left and right edge regions, and acquiring calibration images at the corresponding positions respectively, so that the calibration feature points on the calibration target cover the entire effective imaging area of ​​the image acquisition unit; and uniformly solving the intrinsic parameters and distortion parameters of the image acquisition unit based on the calibration feature points acquired at different calibration positions. The multiple calibration positions are set along the lateral direction of the imaging field of view of the image acquisition unit, including at least the left edge region, the left transition region, the center region, the right transition region, and the right edge region, so that the calibration feature points cover different lateral positions of the wide imaging area.

5. The method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction according to claim 4, characterized in that, Step 3-1 specifically includes: The residual between the actual pixel coordinates and the ideal projection coordinates is calculated based on the calibration feature points collected at each calibration location, and a distortion residual mapping relationship covering the entire effective imaging area is established. For the extracted center line of the line structured light, the corresponding distortion compensation amount is obtained from the distortion residual mapping relationship according to its image position, and the sub-pixel coordinates of the center line of the line structured light are corrected. Specifically, for the center point of the line structured light located within a preset area at the edge of the image, a local interpolation method is used to determine the distortion compensation amount of the edge region based on the distortion residuals corresponding to adjacent calibration positions. The distortion compensation amount is then used to perform sub-pixel-level position correction on the center point of the line structured light to reduce the three-dimensional reconstruction error of the image edge region under wide-span imaging conditions.

6. The method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction according to claim 1, characterized in that, Step 4 involves fitting a reference surface of the airport runway based on the 3D point cloud data and extracting abnormally protruding regions above the reference surface to obtain candidate foreign object regions. Specifically, this includes: Step 4-1: Perform plane fitting on the three-dimensional point cloud data to determine the reference height of the airport runway surface; Step 4-2: Extract points in the 3D point cloud data that are higher than the reference height as spatial outliers; Step 4-3: Based on the three-dimensional spatial connectivity between adjacent spatial anomalies, perform regional aggregation on the spatial anomalies to construct the foreign object candidate region.

7. The method for detecting foreign objects on airport runways based on wide-band structured light 3D reconstruction according to claim 1, characterized in that, Step 5 involves performing multimodal feature analysis and three-dimensional geometric dimension calculation on the candidate foreign object region to determine the foreign object detection and identification results, and outputting the location and size information of the foreign object. Specifically, this includes: Step 5-1: Extract the three-dimensional geometric features, spatial distribution features, and surface reflected light intensity features of the foreign object candidate region; Step 5-2: Calculate the three-dimensional geometric dimensions of the foreign object candidate region based on the three-dimensional geometric features, including length, width, height, and spatial three-dimensional coordinates; Step 5-3: Perform feature analysis by combining the three-dimensional morphological features, spatial distribution features and reflected light intensity features of the foreign object candidate region, identify and determine the foreign object category, and output the location and size information of the foreign object in the airport runway.

8. A foreign object detection device for airport runways based on wide-band structured light 3D reconstruction, characterized in that, The device includes a mobile carrier, a mounting bracket, a projection unit, an image acquisition unit, and a processing and control unit. The mobile carrier is used to carry the line structured light measurement system and move along the runway of the airport to be measured. The mounting bracket is disposed on the mobile carrier; The projection unit is mounted on the mounting bracket and is used to project a wide-band structured light onto the surface of the airport runway to form a laser light strip covering the runway detection area. The image acquisition unit is mounted on the mounting bracket and forms a line structured light triangulation structure with the projection unit, which is used to synchronously acquire line structured light images of the airport runway surface. The processing control unit is communicatively connected to the projection unit and the image acquisition unit, respectively, and is used to control the projection unit and the image acquisition unit to work together, receive the line structured light image and the displacement data of the moving carrier, and execute the airport runway foreign object detection method as described in any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images