A system and method for low-cost foreign object detection and classification on airport runways

Abstract

The present application relates to the field of visual inspection technology, and specifically to a system and method for low-cost detection and classification of foreign objects on airport runways. The system includes a two-dimensional turntable, an integrated detection machine and a server. The integrated detection machine is fixed above the pitch mechanism of the two-dimensional turntable, and a laser radar, a camera, a Mini industrial computer and a wiper are integrated inside the integrated detection machine. The present application combines laser radar with an optical camera, and adopts a ground-fixed two-dimensional turntable equipped with radar and a camera to cruise left and right to complete the large-field-of-view monitoring of the airport runway; the self-learning of the model is carried out in combination with the re-inspection results of manual intervention, and the detection and classification of point clouds and two-dimensional images can be realized by combining traditional algorithms with deep learning methods. At the same time, in the specific outdoor environment of the airport, the distribution of dispersed sites and the laser indication of foreign objects in a large field of view environment are completed, and the water droplets and stains on the lens can be cleaned at the same time.

Description

Technical Field

[0001] The present application relates to the field of visual inspection technology, and in particular to a system and method for low-cost foreign object detection and classification on airport runways. Background Art

[0002] In recent decades, with the occurrence of safety accidents during takeoff and landing at airports, foreign object debris (Foreign Object Debris, runway foreign objects) on airport runways has gradually attracted the attention of airport staff. In order to reduce and eliminate the danger caused by foreign objects on the runway during takeoff and landing, the importance of installing a runway foreign object monitoring system on the runway has become increasingly obvious. There are many types of typical runway foreign objects on the runway, including metal parts such as screws and nuts, metal tools, flying objects such as personal documents, pens, etc., as well as mechanical tools, animals and plants, wood blocks, plastic products, etc. Usually, the size of runway foreign objects is small, and its detection and classification is a relatively difficult task for a large field of view such as the runway.

[0003] At present, the international method for dealing with foreign object detection on airport runways is mostly hybrid detection of millimeter-wave radar combined with optical cameras. Most of the structures are tower-type and the cost is relatively high. Although the millimeter-wave has the advantage of good object-circumvention ability and is less affected by weather and environment, the low detection accuracy is its unavoidable shortcoming. With the popularization of LiDAR in autonomous driving, the price of LiDAR has gradually decreased, and its performance and accuracy have also been greatly improved. At present, some existing LiDAR combined with optical camera detection methods only have the function of foreign object detection. It is impossible to classify foreign objects. At the same time, the existing technology does not have the direction indication of foreign objects, nor does it have a solution to the problem of lens cleaning in the outdoor environment of the airport. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide a system and method for low-cost foreign object detection and classification on airport runways. The system and method combine lidar and optical cameras, and use a ground-fixed two-axis gimbal equipped with radar and cameras to perform left and right cruise scanning to complete large-field-of-view monitoring of airport runways. At the same time, the system and method are combined with manual initial re-inspection to carry out self-learning of the model, so that point clouds and two-dimensional images can be classified using deep learning methods.

[0005] To achieve the above objectives, the present application provides the following technical solutions:

[0006] In a first aspect, in an embodiment provided in the present application, a system for low-cost foreign object detection and classification on an airport runway is provided, the system comprising a two-dimensional turntable, an integrated detection machine and a server, the integrated detection machine being fixed above the pitch mechanism of the two-dimensional turntable, and the integrated detection machine being internally integrated with a laser radar, a camera, a Mini industrial computer and a wiper;

[0007] Two viewing areas corresponding to the camera and the laser radar are reserved at the front end of the integrated detection machine, and the viewing areas are used for the camera and the laser radar to monitor through the areas; a naturally drooping wiper is provided in the middle of the integrated detection machine, and the wiper is used to rotate left and right to clean the glass;

[0008] The two-dimensional turntable is connected to the Mini industrial computer inside the integrated detection machine via the serial port RS485, and the Mini industrial computer is connected to the server via a switch.

[0009] As a further solution of the present invention, the system for low-cost foreign object detection and classification on airport runways also includes a power supply, the two-dimensional turntable and the integrated detection machine are connected to the power supply, the power supply voltage of the two-dimensional turntable and the laser radar is 24V, the camera is powered by a POE network cable, and the power supply voltage of the Mini industrial computer is 220V.

[0010] As a further solution of the present invention, a laser indication module is provided above the camera, and the laser indication module and the camera together constitute an optical camera module. The front end of the integrated detection machine adopts K9 glass combined with glass hydrophobic coating.

[0011] As a further solution of the present invention, the integrated detection machine is used for collecting and processing point cloud data and two-dimensional images, and judges whether there are foreign objects on the road surface through point cloud recognition algorithm and deep learning model, saves the judgment results and log records locally, and uploads them to the server for communication.

[0012] As a further solution of the present invention, the integrated detection machine integrates a wiper cleaning module, a laser radar module, an optical camera module and an industrial computer. The optical camera module integrates an infrared light, a zoom lens, a starlight-level CMOS and a laser indicator.

[0013] As a further solution of the present invention, the integrated detection machine and the pan / tilt platform of the two-dimensional turntable are fixedly installed on both sides of the runway and connected to the switch of the server via a network cable.

[0014] In a second aspect, in another embodiment provided in the present application, a method for low-cost foreign object detection and classification on an airport runway is provided, comprising:

[0015] Step 1: Install the integrated detection machine and the PTZ of the two-dimensional turntable on both sides of the runway and connect them to the switch of the server via a network cable;

[0016] Step 2: The two-dimensional turntable carries the integrated detection machine to perform a scan, and the scanning field of view is divided into a number of preset points. The fields of view collected at each preset point have an intersection, and a basic point cloud collection and real-time video recording are performed on the cleaned runway;

[0017] Step 3: The two-dimensional turntable carries the integrated detection machine to perform real-time back and forth scanning. After the two-dimensional turntable reaches each preset point, it pauses for a set time, and then the laser radar is used to collect static point clouds;

[0018] Step 4: The industrial computer processes the data collected from the point cloud and detects foreign objects on the airport runway. If it is determined that there is no foreign object, it moves to the next point for collection. If it is determined that there is a foreign object, it will alarm in real time. At the same time, the lens and pan / tilt are used to enlarge and collect the local image of the foreign object. According to the point cloud model, the foreign object image model, and manual re-inspection, the foreign objects are classified. Among them, the manual re-judgment result is used as the input data for the next self-training of the model to update and self-learn the model;

[0019] Step 5: Manually remove foreign matter, reset the device, and return to step 3 to start retesting.

[0020] As a further solution of the present invention, in the method for low-cost foreign object detection and classification on airport runways, when performing foreign object detection and classification, the industrial computer collects real-time data or position information from the optical camera module, the laser radar, the two-dimensional turntable and the wiper, and controls the optical camera module, the laser radar, the two-dimensional turntable and the wiper respectively;

[0021] The industrial computer uploads the identified foreign object information, images and alarms to the server, and the server sends the identification model and control commands to the industrial computer. The server displays the identified foreign object information, images and alarms to the operator, who then determines the type of foreign object and makes corresponding controls.

[0022] As a further solution of the present invention, when the integrated detection machine and the pan / tilt of the two-dimensional turntable are fixedly installed on both sides of the runway, the method of arranging the stations of each detection end and setting the preset points includes:

[0023] Arrange the distance between each detection client according to the runway width;

[0024] Each testing station, centered on itself, covers a fan-shaped area with a radius of the runway width on the side facing the runway; the coverage areas of adjacent testing stations on the same side of the runway form adjacent overlapping areas, and adjacent testing stations on different sides of the runway are staggered to form overlapping areas for repeated testing and confirmation by multiple stations.

[0025] As a further solution of the present invention, the method for low-cost foreign object detection and classification on an airport runway further includes detecting foreign objects by point cloud acquisition. The method for detecting foreign objects by point cloud includes:

[0026] Step 11), the road surface to be detected is divided into 18 sector areas with an interval of 10 degrees as preset points, and the sector area within the range of 2.5m-50m under each preset point is collected as basic point cloud data to provide differential comparison data for the real-time detection process;

[0027] Step 12), when the laser radar collects the real-time road point cloud of the two-dimensional turntable at the preset position, the corresponding basic point cloud data is used to align with the two-dimensional turntable, and after the alignment, the real-time point cloud coincides or approximately coincides with the basic point cloud;

[0028] Step 13) construct voxels of real-time point cloud data and basic point cloud. When there is a point cloud in a voxel, the center point of the voxel is used to represent all point clouds in the voxel. All point clouds are downsampled once. For two point cloud sets that are not completely overlapped and have position deviations, position correction is performed. The corresponding points are transformed to completely overlap through the center.

[0029] Step 14), using the two pairs of point cloud sets after the center transformation to perform difference, the point cloud clusters of the airport runway foreign objects are displayed after the real-time road surface point cloud difference, and the airport runway foreign object point cloud clusters that meet the conditions are clustered using the regional growing algorithm, and the location and distance of the cluster are output;

[0030] Step 15), when it is detected in step 14 that the number of foreign objects exceeds 5, a bird's-eye view is generated from the point cloud image in the target area, and the reflectivity of different objects in the point cloud image and the characteristics of partial shadows of raised foreign objects on the road surface are utilized to use a self-learning trained model to identify whether there are foreign objects in the bird's-eye view; if the model determines that there are still foreign objects, it is determined that there are foreign objects on the real-time road surface, and the size and position of the foreign objects are output.

[0031] As a further solution of the present invention, the distribution method of the preset points includes:

[0032] Driven by the two-dimensional turntable, the integrated machine has a coverage angle from 0° to 180°, and the laser radar field of view angle is 14°. Starting from 0°, an object 2.5m away on one side of the point cloud collected by the laser radar appears in the point cloud set collected by the next preset point. Starting from the first preset point, each adjacent preset point is within an angle of α=10.65° and covers a distance range within a working distance L=2.5m.

[0033] The technical solution provided by this application has the following beneficial effects:

[0034] The system and method for low-cost detection and classification of foreign objects on airport runways provided by this application combines laser radar and optical cameras, and uses a ground-fixed two-dimensional turntable (the rotating mechanism adopts a fragile design) equipped with radar and cameras to cruise left and right to complete the large-field monitoring of the airport runway; at the same time, combined with the results of manual intervention re-inspection, the model is self-learned, and the traditional algorithm combined with deep learning methods can be used to detect and classify point clouds and two-dimensional images. At the same time, in the specific outdoor environment of the airport, the distribution of dispersed sites and the laser indication of foreign objects in a large-field environment are completed, and corresponding cleaning designs are made for water droplets and stains on the lens.

[0035] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments of the present application. In the drawings:

[0037] Figure 1 This is a structural block diagram of a system for low-cost foreign object detection and classification on airport runways according to an embodiment of the present application.

[0038] Figure 2 This is a flow chart of a method for low-cost foreign object detection and classification on an airport runway according to an embodiment of the present application.

[0039] Figure 3 A schematic diagram of data flow in a method for low-cost foreign object detection and classification on an airport runway according to an embodiment of the present application.

[0040] Figure 4 This is a schematic diagram of the arrangement of monitoring sites in a method for low-cost foreign object detection and classification on an airport runway according to an embodiment of the present application.

[0041] Figure 5 This is a flow chart of point cloud detection of foreign objects in a method for low-cost foreign object detection and classification on an airport runway according to an embodiment of the present application.

[0042] Figure 6 This is a schematic diagram of the distribution of preset points in a method for low-cost foreign object detection and classification on an airport runway according to an embodiment of the present application.

[0043] Figure 7 A schematic diagram of laser indication of a detection site in a method for low-cost foreign object detection and classification on an airport runway according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0046] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0047] It should be understood that in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with substantially the same functions and effects. For example, the first callback function and the second callback function are only used to distinguish different callback functions, and their order is not limited. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit them to be different.

[0048] In the context, "computer equipment", also called "computer", refers to an intelligent electronic device that can perform predetermined processing processes such as numerical calculations and / or logical calculations by running predetermined programs or instructions. It may include a processor and a memory, and the processor executes the pre-stored instructions in the memory to perform the predetermined processing process, or the hardware such as ASIC, FPGA, DSP, etc. performs the predetermined processing process, or a combination of the above two. Computer equipment includes but is not limited to servers, personal computers, laptops, tablets, smart phones, etc.

[0049] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0050] Since some existing laser radar combined with optical camera detection methods only have the function of foreign object detection, it is impossible to classify foreign objects. At the same time, the existing technology does not indicate the location of foreign objects, and does not provide a solution to the lens cleaning problem in the outdoor environment of the airport.

[0051] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide a system and method for low-cost foreign object detection and classification on airport runways. The system and method combine lidar and optical cameras, and use a ground-fixed two-axis gimbal equipped with radar and cameras to perform left and right cruise scanning to complete large-field-of-view monitoring of airport runways. At the same time, the system and method are combined with manual initial re-inspection to carry out self-learning of the model, so that point clouds and two-dimensional images can be classified using deep learning methods.

[0052] Specifically, the embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0053] See also Figure 1 As shown, Figure 1 This is a structural block diagram of a system for low-cost foreign object detection and classification on airport runways provided in an embodiment of the present application. The system for low-cost foreign object detection and classification on airport runways includes a two-dimensional turntable, an integrated detection machine and a server. The integrated detection machine is fixed above the pitch mechanism of the two-dimensional turntable. The integrated detection machine integrates a laser radar, a camera, a Mini industrial computer and a wiper.

[0054] Two viewing areas corresponding to the camera and the laser radar are reserved at the front end of the integrated detection machine, and the viewing areas are used for the camera and the laser radar to monitor through the areas; in the middle of the integrated detection machine is a naturally drooping wiper, and the wiper is used to rotate left and right to clean the glass.

[0055] The two-dimensional turntable is connected to the Mini industrial computer inside the integrated detection machine via the serial port RS485, and the Mini industrial computer is connected to the server via a switch.

[0056] In an embodiment of the present invention, the two-dimensional turntable is used to realize left and right rotation and up and down pitch of the integrated detection machine. An easy breaking point is arranged in the rotating structure of the two-dimensional turntable. The easy breaking point is 30 mm above the ground. The easy breaking point can withstand a bending moment of 204 N·m without being damaged, and it will break before the bending moment reaches 680 N·m. The purpose of this design is to meet the requirements that the airport infrastructure equipment is stable and wind-resistant and can break under a certain impact force.

[0057] The integrated detection machine is fixed above the pitch mechanism of the two-dimensional turntable, and can realize left and right rotation and up and down pitch under the movement of the turntable. The integrated detection machine is internally integrated with components such as laser radar, monitoring camera, Mini industrial computer, wiper, etc.

[0058] The system for low-cost foreign object detection and classification on airport runways also includes a power supply. The two-dimensional turntable and the integrated detection machine are connected to the power supply. The power supply voltage of the two-dimensional turntable and the laser radar is 24V. The camera is powered by a POE (active Ethernet) network cable, and the power supply voltage of the Mini industrial computer is 220V.

[0059] In the embodiment of the present invention, a laser indicator module is provided above the camera, and the laser indicator module and the camera together constitute an optical camera module, and the front end of the integrated detection machine uses K9 glass combined with glass hydrophobic coating. That is, two viewing areas are left at the front end of the integrated detection machine, and the camera and the laser radar monitor through the area. The laser indicator module together constitutes an optical camera module above the camera, and in the middle is a naturally drooping wiper that can rotate left and right to clean the glass.

[0060] In an embodiment of the present invention, the integrated detection machine is used for collecting and processing point cloud data and two-dimensional images, and judges whether there are foreign objects on the road surface through point cloud recognition algorithm and deep learning model, saves the judgment results and log records locally, and uploads them to the server for communication.

[0061] In an embodiment of the present invention, the integrated detection machine integrates a wiper cleaning module, a laser radar module, an optical camera module and an industrial computer. The optical camera module integrates an infrared light, a zoom lens, a starlight-level CMOS and a laser indicator.

[0062] Among them, the integrated detection module is responsible for the collection and data processing of point cloud data and two-dimensional images. It judges whether there are foreign objects on the road surface through point cloud recognition algorithm and deep learning model, saves the judgment results and log records locally, and uploads them to the server for communication. This module integrates wiper cleaning module, lidar module, optical camera module, industrial computer, etc. The optical camera module integrates infrared light (850nm), zoom lens (8-32mm), starlight-level CMOS (minimum illumination 0.0005lux), and laser pointer (1000mW, green horizontal line).

[0063] In an embodiment of the present invention, the integrated detection machine and the pan / tilt platform of the two-dimensional turntable are fixedly installed on both sides of the runway and connected to a switch of the server via a network cable.

[0064] See also Figure 2 , Figure 2 : is a flow chart of a method for low-cost foreign object detection and classification on an airport runway provided in an embodiment of the present application. Figure 2 As shown, the method for low-cost foreign object detection and classification on airport runways includes the following steps:

[0065] Step 1. Fix the integrated detection machine and the pan-tilt table of the two-dimensional turntable on both sides of the runway, and connect the switch of the server through a network cable; that is, install and arrange the integrated detection machine, the two-dimensional turntable, and the server: fix the integrated detection machine and the pan-tilt table on both sides of the runway, and connect the switch of the server through a network cable.

[0066] Step 2: The two-dimensional turntable is equipped with the integrated detection machine to perform a scan, and the scanning field of view is divided into a number of preset points. The fields of view collected by each preset point have an intersection, and basic point cloud collection and real-time video recording are performed on the cleaned runway; wherein, the two-dimensional turntable is equipped with the integrated detection machine to perform a scan, and the scanning field of view is divided into 60 preset points. The fields of view collected by each preset point have a certain intersection, ensuring that the working distance of 2.5m-50m can be included, and basic point cloud collection and real-time video recording are performed on the runway after daily cleaning.

[0067] Step three, the two-dimensional turntable is equipped with the integrated detection machine to perform real-time back and forth scanning. After the two-dimensional turntable reaches each preset point, it pauses according to the set time, and the laser radar is used to collect static point cloud; in the embodiment of the present application, after the turntable reaches each preset point, it pauses for 3s, and the laser radar is used to collect static point cloud. The industrial computer processes the point cloud data and detects foreign objects on the airport runway (foreign objects outside the airport). If it is determined that there are no foreign objects, it moves to the next point for collection.

[0068] Step 4: The industrial computer processes the data collected from the point cloud and detects foreign objects on the airport runway. If it is determined that there is no foreign object, it moves to the next point for collection. If it is determined that there is a foreign object, it will alarm in real time. At the same time, the lens and the pan / tilt are used to enlarge and collect the local image of the foreign object. According to the point cloud model, the foreign object image model, and manual re-inspection, the three are combined to re-judge and classify the foreign objects. Among them, the manual re-judgment result is used as the input data for the next self-training of the model to update and self-learn the model;

[0069] Step 5: Manually remove foreign matter, reset the device, and return to step 3 to start retesting.

[0070] In an embodiment of the present invention, when the method for low-cost foreign object detection and classification of an airport runway is performing foreign object detection and classification, the industrial computer collects real-time data or position information from the optical camera module, the laser radar, the two-dimensional turntable and the wiper, and controls the optical camera module, the laser radar, the two-dimensional turntable and the wiper respectively;

[0071] The industrial computer uploads the identified foreign object information, images and alarms to the server, and the server sends the identification model and control commands to the industrial computer. The server displays the identified foreign object information, images and alarms to the operator, who then determines the type of foreign object and makes corresponding controls.

[0072] In this application, the data flow is as follows Figure 3 As shown, the industrial computer collects real-time data or location information from the optical camera module, laser radar, two-dimensional turntable, and wiper, and controls them respectively. The industrial computer uploads the identified foreign object information, images, alarms, etc. to the server, and the server sends the recognition model and control commands to the industrial computer. The server displays the identified foreign object information, images, alarms, etc. to the operator, who identifies the foreign object category and makes corresponding controls.

[0073] In an embodiment of the present invention, when the integrated detection machine and the pan / tilt of the two-dimensional turntable are fixedly installed on both sides of the runway, the method of arranging the stations of each detection end and setting the preset points includes:

[0074] Arrange the distance between each detection client according to the runway width;

[0075] Each testing station, centered on itself, covers a fan-shaped area with a radius of the runway width on the side facing the runway; the coverage areas of adjacent testing stations on the same side of the runway form adjacent overlapping areas, and adjacent testing stations on different sides of the runway are staggered to form overlapping areas for repeated testing and confirmation by multiple stations.

[0076] Therefore, see Figure 4 As shown, the method of arranging each detection end and setting the preset points in this application is:

[0077] ① Based on the runway width of 50m, each detection client is arranged at an interval of about 50m; Figure 5 As shown;

[0078] ② Each detection station, with itself as the center, covers a sector area with a radius of 50m on the side facing the runway. As shown in the figure, detection station 1 covers half of the sector area downward, and detection station 2 covers half of the circular area upward, and forms a certain overlap with the coverage area of ​​detection station 1 and detection station 3. The advantage of this is that on the one hand, it can form the effect of multi-station repeated detection and confirmation, and on the other hand, due to the size and orientation of different foreign objects, the reflectivity in a certain direction is low, which is easy to miss detection, so multi-directional scanning detection is required.

[0079] In the embodiments of the present invention, see Figure 5 As shown, the method for low-cost foreign object detection and classification on airport runways also includes detecting foreign objects by point cloud acquisition. The method for detecting foreign objects by point cloud includes:

[0080] Step 11), the road surface to be detected is divided into 18 sector areas with an interval of 10 degrees as preset points, and the sector area within the range of 2.5m-50m under each preset point is collected as basic point cloud data to provide differential comparison data for the real-time detection process;

[0081] Step 12), when the laser radar collects the real-time road point cloud of the two-dimensional turntable at the preset position, the corresponding basic point cloud data is used to align with the two-dimensional turntable, and after the alignment, the real-time point cloud coincides or approximately coincides with the basic point cloud;

[0082] Step 13) construct voxels of real-time point cloud data and basic point cloud. When there is a point cloud in a voxel, the center point of the voxel is used to represent all point clouds in the voxel. All point clouds are downsampled once. For two point cloud sets that are not completely overlapped and have position deviations, position correction is performed. The corresponding points are transformed to completely overlap through the center.

[0083] Step 14), using the two pairs of point cloud sets after the center transformation to perform difference, the point cloud clusters of the airport runway foreign objects are displayed after the real-time road surface point cloud difference, and the airport runway foreign object point cloud clusters that meet the conditions are clustered using the regional growing algorithm, and the location and distance of the cluster are output;

[0084] Step 15), when it is detected in step 14 that the number of foreign objects exceeds 5, a bird's-eye view is generated from the point cloud image in the target area, and the reflectivity of different objects in the point cloud image and the characteristics of partial shadows of raised foreign objects on the road surface are utilized to use a self-learning trained model to identify whether there are foreign objects in the bird's-eye view; if the model determines that there are still foreign objects, it is determined that there are foreign objects on the real-time road surface, and the size and position of the foreign objects are output.

[0085] When using point cloud to detect foreign objects, it can be seen from the system workflow that before the system detection starts, the first step is to divide the road surface to be inspected into 18 sector areas with an interval of 10 degrees as preset points, and collect the sector area within the range of 2.5m-50m under each preset point as the basic point cloud data to provide differential comparison data for the real-time detection process.

[0086] In the second step, when the laser radar collects the real-time road point cloud of the turntable at a preset position, it uses the corresponding basic point cloud data to align it with the real-time point cloud. After the alignment, the real-time point cloud coincides or approximately coincides with the basic point cloud.

[0087] The third step is to construct voxels of real-time point cloud data and basic point cloud, that is, to use continuous three-dimensional 0.01m*0.01m*0.01m voxel grids to divide the point cloud to be checked. When there is a point cloud in a voxel, the center point of the voxel is used to finally represent all the point clouds in the voxel. Finally, all point clouds can be downsampled once. At the same time, for two point cloud sets that do not completely overlap and have a certain position deviation, a certain position correction is performed. Even if the two point cloud sets have a certain position deviation after the second step of alignment, as long as the corresponding points are also in the 0.01m*0.01m*0.01m voxel grid, the corresponding points will completely overlap after the center transformation.

[0088] The fourth step is to use the two pairs of point cloud sets after the above center transformation to perform differential, that is, the real-time road surface point cloud minus the basic road surface point cloud set. If there are foreign objects on the real-time road surface, the point cloud clusters of the airport runway foreign objects will be displayed after the real-time road surface point cloud is differentiated. The airport runway foreign object point cloud clusters that meet certain conditions are clustered using the regional growing algorithm, and finally the location and distance of the cluster are output.

[0089] Step 5: If the number of foreign objects detected in the fourth step exceeds 5, there are two possibilities. One is that there are indeed many foreign objects on the road surface, but this situation is generally rare. The other possibility is that there is a large deviation between the real-time road surface point cloud and the basic road surface during the second step of registration, resulting in the point cloud set not completely overlapping after matrix transformation and central voxel filtering. At this time, the point cloud image of the target area, i.e. the 2.5m-50m fan-shaped area, is used to generate a bird's-eye view. The reflectivity of different objects in the point cloud image and the characteristics of the partial shadows of raised foreign objects on the road surface are used to use the self-learning trained model to identify whether there are foreign objects in the bird's-eye view. If the model determines that there are still foreign objects at this time, it is determined that there are foreign objects on the real-time road surface, and the size and position of the foreign objects are output.

[0090] In the embodiments of the present invention, see Figure 6 As shown, the distribution method of the preset points includes:

[0091] Driven by the two-dimensional turntable, the integrated machine has a coverage angle from 0° to 180°, and the laser radar field of view angle is 14°. Starting from 0°, an object 2.5m away on one side of the point cloud collected by the laser radar appears in the point cloud set collected by the next preset point. Starting from the first preset point, each adjacent preset point is within an angle of α=10.65° and covers a distance range within a working distance L=2.5m.

[0092] See also Figure 6 As shown, the integrated machine is driven by a two-dimensional turntable, with a coverage angle from 0° to 180°, and the laser radar field of view angle is 14°. Starting from 0°, in order to ensure that the field of view of the working distance of 2.5m can be covered, the object at a distance of 2.5m on one side of the point cloud collected by the laser radar also needs to appear in the point cloud set collected by the next preset point. Starting from the first preset point, this patent forms an angle of α=10.65° between each adjacent preset point, which can ensure that the closest distance to the working distance L=2.5m can be covered.

[0093] The embodiment of the present invention also includes a point cloud registration method. The registration method used in this application is an algorithm for solving the rigid body transformation relationship between two 3D point cloud sets. In order to find the rigid body transformation T between the real-time road surface point cloud set A and the basic road surface point cloud set B, so that point cloud A coincides with point cloud B after the change of T, this application uses the minimum sum of the distances between each point in point cloud set A and the nearest point in point cloud set B as the mathematical expression of coincidence, and its implementation steps are implemented as follows:

[0094] (1) For each point P in the point set A ai Applying the initial transformation T0, we get P ai ';

[0095] (2) Find the distance point P from point set B ai 'The nearest point P bi , forming corresponding point pairs;

[0096] (3) Solve the optimal transformation:

[0097]

[0098] in,

[0099]

[0100] (4) Determine whether the solution has converged based on the two iteration errors and the number of iterations. If it has converged, output the final result: T = ΔT*T0; otherwise, T0 = ΔT*T0 and repeat step (1).

[0101] In an embodiment of the present invention, a voxel center filtering method is also included. The voxel center grid filtering point cloud simplification process is as follows:

[0102] (1) Based on the point cloud data coordinate set, find the maximum value x on the three coordinate axes of X, Y, and Z max， y max , z max and the minimum value x min ,y min , z min .

[0103] (2) According to the maximum and minimum values ​​on the three coordinate axes of X, Y, and Z, the side length l of the minimum bounding box of the point cloud is obtained x , l y , l z .

[0104]

[0105] (3) Set the side length of the voxel grid cell = 0.01 m, and divide the three coordinate axes X, Y, and Z equally into M, N, and L parts. Then the minimum bounding box is divided into M*N*L voxel grids.

[0106] sum=M*N*L

[0107]

[0108] in, Indicates rounding down, and sum is the total number of small voxel grids.

[0109] (4) Number each voxel grid as (i, j, k); determine the voxel grid to which each data point belongs.

[0110]

[0111] (5) Perform point cloud simplification. Calculate the center of each voxel grid and use the center to replace all points in the grid. This completes the simplification process.

[0112] In the embodiment of the present invention, the principle of the region generation algorithm is as follows: the basic idea of ​​using region growing after difference is to gather points with similar properties to form a region. First, find a seed as the starting point of growth for each region to be segmented, and then merge the points with the same or similar properties as the seed in the neighborhood around the seed (determined according to a predetermined growth or similarity criterion, mostly normal vectors and curvatures) into the region where the seed is located. The new point continues to act as a seed to grow around until there are no more points that meet the conditions to be included, and a region is grown.

[0113] The result of curvature-based region growing is one or more cluster sets, each of which is considered to be part of the same smooth surface. First, the points are sorted according to their curvature, and the area where the initial seed point is located is the smoothest area, which can reduce the total number of segmented regions. The pcl::RegionGrowing class uses this method.

[0114] In an embodiment of the present invention, the process of the region growing algorithm includes:

[0115] Set an empty seed point sequence and an empty cluster array, select the seed point (the point with the smallest curvature) and add it to the seed point sequence;

[0116] Search for neighboring points of the current seed point, calculate the angle between the normal of the neighboring point and the normal of the current seed point, and add the neighboring points that are less than the smoothing threshold to the current area;

[0117] Check the curvature of each neighborhood point, add the neighborhood points that are less than the curvature threshold to the seed point sequence, delete the current seed point, and continue to grow with the new seed point.

[0118] Repeat the above growth process until the seed point sequence is cleared. At this point, a region is grown and added to the clustering array;

[0119] Repeat the above steps for the remaining points until all points are traversed.

[0120] The embodiment of the present invention also includes a method for image judgment, classification and self-cleaning:

[0121] Before the actual judgment of foreign objects, it is necessary to collect 6 types of data (machine tools, animals and plants, wood blocks, plastic products, glass beads, glass stains) in actual working scenes or simulated actual scenes. Each data set has more than 30 typical category images collected. After collecting the images, the target objects are annotated and the data set is constructed. The training is carried out through the YOLOX framework. After the training model is obtained, real-time detection and classification are performed. After the foreign object is detected in the actual point cloud, the distance and direction of the foreign object are determined. The center of the camera lens is aligned with the direction of the foreign object using a two-dimensional turntable, and the number of pixels occupied by the camera lens in the image is increased. The amount is increased, specifically more than 8*8 pixels. After taking a partial image of the foreign object, it is passed to the trained model. If the recognition result is the four types of foreign objects, namely mechanical tools, animals and plants, wood blocks, and plastic products, it will wait for manual secondary confirmation of the image category. If the manually confirmed category is consistent with the model recognition category, the training data set will not be generated, and the alarm and laser indication will be directly performed. If the manually confirmed category is inconsistent with the model recognition category, the manually corrected category annotation and the picture will be packaged into a training data set. The training data set and the local data set will be retrained and deployed every morning or during the idle time of the server. Another method is to stop the detection of integrated all-in-one machine when the recognition result is water droplets or stains, control the blowing device in front of the glass to blow for 6s, and then control the wipers to clean the external glass back and forth. In addition to cleaning the windshield after the image recognizes pollution or water stains, the wipers will also perform air blowing and cleaning work after each device initialization and after an interval of 2h.

[0122] In an embodiment of the present invention, a method for converting a point cloud image into a bird's-eye view is also included. The image coordinate axis is different from the point cloud coordinate axis. The coordinate axis in the image is always positive, the origin is located in the upper left corner, and the coordinate is an integer value. The coordinate value of the point cloud can be positive or negative, and the coordinate is a real number. The positive X axis represents the front, the positive Y axis represents the left, and the positive Z axis represents the top. The following steps are used to convert the point cloud into a bird's-eye view:

[0123] ① Create a rectangular area to completely cover the sector area of ​​2.5m-50m with an angle of 14° (the field of view of the laser radar is 14°), that is,

[0124] From far left to far right: side_range = (-7.5, 7.5)

[0125] Last to first: fwd_range = (0,60)

[0126] ② Create a filter to retain the point cloud within the specified rectangle and clear other point clouds.

[0127] ③Map point position to pixel position.

[0128] In the point cloud data, the data is real numbers, and the mapped image data is integers. It is necessary to convert the point cloud with a measurement unit of meters into image pixels with a resolution of 2 cm: Where x_img and y_img are the x and y positions of the point cloud in the bird's eye view, respectively, and x_point and y_point are the x and y values ​​of the point cloud, respectively. The x-axis and y-axis in the point cloud are swapped and the direction is reversed so that the image coordinates can be processed. To round down.

[0129] ④ Shift the origin: The x and y values ​​are still negative and cannot be projected onto the image, so the data needs to be shifted so that the data at the (0,0) position is the smallest.

[0130] x_img=x_img+(7.5 / 0.02)y_img=y_img+(60 / 0.02)

[0131] ⑤Specify the values ​​to be filled in these pixel positions. The method is to fill the height data. In order to set the pixel value to an integer and adjust the pixel value range to the interval of 0-255, get the minimum and maximum height values ​​from the data and rescale them to the range of 0-255.

[0132] The embodiment of the present invention also includes a laser indication method. Figure 7 As shown, the laser indication method is:

[0133] Before each station automatically runs, the point cloud data of objects with obvious features at two angles of 0 degrees and 180 degrees are saved. For example, a point at 0° is Point10 for detection station 1, a point at 180° is Point11, and a point at 0° is Point20 for detection station 2, and a point at 180° is Point21. After a single station detects a foreign object at detection station 1, it is first determined that the foreign object is closest to station 2. Then, the relative position from station 1 to station 2 (the height is consistent by default) is used to calculate the transformation matrix A between the point clouds collected by the two stations. The point at which the foreign object is detected by detection station 1 is point X, and the angle ∠XO1Point12 between point X and station 1 is calculated. Then, after the two-dimensional turntable is moved to this angle, the laser pointer is turned on. The point corresponding to point X in detection station 2 is X'=AX, and the angle between X' and station 2 is calculated to be ∠X'O2Point21. At the same time, the two-dimensional platform of detection station 2 is moved to this angle to turn on the laser pointer. The point where the two laser lines intersect is the position indicated by the foreign object.

[0134] In summary, the system and method for low-cost detection and classification of foreign objects on airport runways provided by this application combines laser radar and optical cameras, and uses a ground-fixed two-dimensional turntable (the rotating mechanism adopts a fragile design) equipped with radar and cameras to cruise left and right to complete the large-field monitoring of the airport runway; at the same time, combined with the results of manual intervention re-inspection, the model is self-learned, and the point cloud and two-dimensional image can be detected and classified using traditional algorithms combined with deep learning methods. At the same time, in the specific outdoor environment of the airport, the distribution of dispersed sites and the laser indication of foreign objects in a large-field environment are completed, and corresponding cleaning designs are made for water droplets and stains on the lens.

[0135] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for low-cost foreign body detection and classification on airport runways, characterized in that: The method for low-cost foreign body detection and classification on airport runways is based on a system for low-cost foreign body detection and classification on airport runways for foreign body detection and classification. The system includes a two-dimensional turntable, an integrated detection machine and a server. The integrated detection machine is fixed above the pitch mechanism of the two-dimensional turntable. The integrated detection machine is internally integrated with a laser radar, a camera, a Mini industrial computer and a wiper; two viewing areas corresponding to the camera and the laser radar are reserved at the front end of the integrated detection machine, and the viewing areas are used for the camera and the laser radar to monitor through the area; the middle of the integrated detection machine is a naturally drooping wiper, and the wiper is used to rotate left and right to clean the glass; the two-dimensional turntable is connected to the Mini industrial computer inside the integrated detection machine via the serial port RS485, and the Mini industrial computer is connected to the server via a switch; the method for low-cost foreign body detection and classification on airport runways includes the following steps: S1. Install the integrated detection machine and the PTZ of the two-dimensional turntable on both sides of the runway and connect them to the switch of the server via a network cable; S2. The two-dimensional turntable carries the integrated detection machine to perform a scan. The scanning field of view is divided into a number of preset points. The fields of view collected at each preset point have an intersection. The cleaned runway is subjected to a basic point cloud collection and real-time video recording. S3, the two-dimensional turntable carries the integrated detection machine to perform real-time back and forth scanning. After the two-dimensional turntable reaches each preset point, it pauses according to the set time, and then the laser radar is used to collect static point cloud; S4. The industrial computer processes the data collected from the point cloud and detects foreign objects on the airport runway. If it is determined that there is no foreign object, it moves to the next point for collection. If it is determined that there is a foreign object, it will alarm in real time. At the same time, the lens and pan / tilt are used to enlarge and collect the local image of the foreign object. According to the point cloud model, the foreign object image model, and manual re-inspection, the foreign objects are classified. Among them, the manual re-judgment result is used as the input data for the next self-training of the model to update and self-learn the model; S5, manually remove foreign matter, reset the device, and return to step S3 to start re-detection; The method for low-cost foreign object detection and classification on an airport runway further includes detecting foreign objects by collecting point clouds. The method for detecting foreign objects by point clouds includes: S101, dividing the road surface to be detected into 18 sector areas with an interval of 10 degrees as preset points, collecting the sector area within the range of 2.5m-50m under each preset point as basic point cloud data, and providing differential comparison data for the real-time detection process; S102, when the laser radar collects the real-time road point cloud of the two-dimensional turntable at the preset position, the corresponding basic point cloud data is used to align with the two-dimensional turntable, and after the alignment, the real-time point cloud coincides or approximately coincides with the basic point cloud; S103, constructing voxels of real-time point cloud data and basic point cloud. When there are point clouds in a voxel, all point clouds in the voxel are represented by the center point of the voxel, and all point clouds are downsampled once. Position correction is performed on two point cloud sets that are not completely overlapped and have position deviations, and the corresponding points are transformed to completely overlap through center transformation; S104, using the two pairs of point cloud sets after the center transformation to perform a difference, the point cloud clusters of the airport runway foreign objects are displayed after the real-time road surface point cloud difference, the airport runway foreign object point cloud clusters that meet the conditions are clustered using the regional growing algorithm, and the location and distance of the cluster are output; S105. When it is detected in step S104 that the number of foreign objects exceeds 5, a bird's-eye view is generated from the point cloud image in the target area. The reflectivity of different objects in the point cloud image and the presence of partial shadows of raised foreign objects on the road surface are utilized to use a self-learning trained model to identify whether there are foreign objects in the bird's-eye view. If the model determines that there are still foreign objects, it is determined that there are foreign objects on the real-time road surface, and the size and position of the foreign objects are output.

2. The method for low-cost foreign body detection and classification on airport runways as claimed in claim 1, characterized in that: When the method for low-cost foreign object detection and classification on an airport runway is used for foreign object detection and classification, the industrial computer collects real-time data or position information from the optical camera module, the laser radar, the two-dimensional turntable and the wiper, and controls the optical camera module, the laser radar, the two-dimensional turntable and the wiper respectively; The industrial computer uploads the identified foreign object information, images and alarms to the server, and the server sends the identification model and control commands to the industrial computer. The server displays the identified foreign object information, images and alarms to the operator, who then determines the type of foreign object and makes corresponding controls.

3. The method for low-cost foreign body detection and classification on airport runways as claimed in claim 2, characterized in that: It also includes the station layout and preset point setting of each detection end, wherein the method of station layout and preset point setting of each detection end includes: Arrange the detection clients at intervals according to the runway width; Each detection station, centered on itself, covers a fan-shaped area facing the runway side, so that the detection station covers half of the circular area upwards and overlaps with the two adjacent detection stations in half of the fan-shaped area downwards, forming a multi-station repeated detection confirmation effect.

4. The method for low-cost foreign body detection and classification on airport runways as claimed in claim 3, characterized in that: The distribution method of the preset points includes: Driven by the two-dimensional turntable, the integrated machine has a coverage angle from 0° to 180°, and the laser radar field of view angle is 14°. Starting from 0°, an object 2.5m away on one side of the point cloud collected by the laser radar appears in the point cloud set collected by the next preset point. Starting from the first preset point, each adjacent preset point is within an angle of α=10.65° and covers a distance range within a working distance L=2.5m.

5. The method for low-cost foreign body detection and classification on airport runways as claimed in claim 3, characterized in that: In the registration of the basic point cloud data with the two-dimensional turntable, the point cloud registration method is to solve the rigid body transformation relationship between the two 3D point cloud sets, and the mathematical expression of the coincidence is the minimum sum of the distances between each point in the point cloud set A and the nearest point in the point cloud set B, including the following steps: S201, applying an initial transformation T0 to each point Pai in the point set A to obtain Pai'; S202, searching for the point Pbi closest to the point Pai' from the point set B to form a corresponding point pair; S203, solving the optimal transformation: in, S204, judging whether convergence occurs according to the iterative errors between the previous and the next two times and the number of iterations, if converged, outputting the final result: T = ΔT*T0, otherwise T0 = ΔT*T0, repeating step S201.

6. The method for low-cost foreign body detection and classification on airport runways as claimed in claim 1, characterized in that: It also includes a voxel center grid filtering point cloud simplification process, a voxel center grid filtering method, and the following steps: S301, according to the point cloud data coordinate set, find the maximum value x on the three coordinate axes of X, Y, and Z max ,y max , z max and the minimum value x min ,y min , z min ; S302, according to the maximum and minimum values ​​on the three coordinate axes X, Y, and Z, the side length l of the minimum bounding box of the point cloud is obtained x , l y , l z ; S303, set the side length of the voxel grid cell=0.01m, divide the three coordinate axes X, Y, and Z equally into M, N, and L parts, and then divide the minimum bounding box into M*N*L voxel grids; sum=M*N*L in, Indicates rounding down, and sum is the total number of small voxel grids; S304, number each voxel grid as (i, j, k); determine the voxel grid to which each data point belongs; S305 , simplify the point cloud, calculate the center of each voxel grid, and replace all points in the grid with the center.

7. The method for low-cost foreign body detection and classification on airport runways as claimed in claim 3, characterized in that: The invention also includes converting a point cloud image into a bird's-eye view image, and the method for converting a point cloud image into a bird's-eye view image includes the following steps: S401, create a rectangular area to completely cover the sector area of ​​2.5m-50m with a laser radar field of view angle of 14°; S402, creating a filter to retain the point cloud within the specified rectangle and clear the rest of the point cloud; S403, mapping point positions to pixel positions; S404, translating the origin so as to minimize the data of the origin position; S405, specifying the value to be filled in the pixel position, setting the pixel value to an integer by filling the height data, adjusting the range of the pixel value to the interval of 0-255, obtaining the minimum and maximum height values ​​from the data, and rescaling to the range of 0-255.

8. The method for low-cost foreign body detection and classification on airport runways as claimed in claim 2, characterized in that: Before each station automatically runs, point cloud data of objects with obvious features at two angles of 0 degrees and 180 degrees are saved; the laser indication method includes: After a single station detects a foreign object, the station closest to the foreign object is determined, and the transformation matrix A between the point clouds collected by the two stations is calculated using the relative position of the single station to the nearest station; The point where the foreign body is detected by the single station is point X, the angle ∠XO1Point12 between point X and the single station is calculated, the two-dimensional turntable is moved to the angle, and the laser pointer is turned on; The point corresponding to point X in the detection of the nearest station is X'=AX, and the angle between X' and the nearest station is calculated as ∠ X'O2Point21, move the two-dimensional platform for detecting the nearest site to the said angle and turn on the laser pointer. The point where the two laser lines intersect is the location of the foreign object indication.

Images