Method and device for intelligent recognition of fob target based on voxel feature fusion

Abstract

This invention discloses a method and apparatus for intelligent recognition of FOD targets based on voxel feature fusion. The method includes: performing ground calibration on the ground point cloud of the area to be detected to obtain a calibrated point cloud space; dividing the calibrated point cloud space into multiple voxels based on the point cloud density, such that only one voxel exists along the Z-axis direction, where the Z-axis direction is the height direction of the point cloud space; fusing features from the point cloud in each of the multiple voxels to obtain a new point cloud; segmenting the new point cloud to obtain FOD point clouds; and clustering the FOD point clouds to obtain different classes of FODs for detection. This invention solves the technical problems of poor performance in detecting FOD size and poor real-time performance in related technologies.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electronic detection, in particular to a method and device for FOD target intelligent cognition based on voxel feature fusion. BACKGROUND

[0002] Common FOD detection technologies include FOD detection technologies based on optical sensors, FOD detection technologies based on millimeter wave radars and FOD detection technologies based on laser radars. FOD detection technologies based on optical sensors are difficult to obtain depth information of FOD, have complex algorithm implementation and poor real-time performance; FOD detection technologies based on millimeter wave radars are limited by their own working principles, and it is difficult to further improve the performance of detecting FOD size; FOD detection technologies based on laser radars have high resolution, real-time performance and good performance in detecting FOD size, and are relatively accurate in distance measurement and obstacle identification, but this method is still in the initial stage, and the detection performance needs to be studied.

[0003] FOD detection is crucial for the safe takeoff and operation of an aircraft, and the existing FOD detection technologies are difficult to further improve the performance of detecting FOD size. In order to ensure the safety of passengers, it is necessary to further improve the performance of detecting FOD size.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] Embodiments of the present application provide a method and device for FOD target intelligent cognition based on voxel feature fusion, to at least solve the technical problems of poor performance of detecting FOD size and poor real-time performance of FOD detection in related technologies.

[0006] According to an aspect of an embodiment of the present application, a method for FOD target intelligent cognition based on voxel feature fusion is provided, comprising: performing ground calibration on ground point clouds of a region to be detected to obtain a calibrated point cloud space; dividing the calibrated point cloud space into a plurality of voxels in combination with point cloud density, so that there is only one voxel along the Z-axis direction, wherein the Z-axis direction is the height direction of the point cloud space; performing feature fusion according to the point clouds in each voxel of the plurality of voxels to obtain new point clouds; segmenting the new point clouds to obtain FOD point clouds, and clustering the FOD point clouds to detect different types of FOD.

[0007] According to another aspect of the embodiment of the present application, there is also provided a device for intelligent recognition of FOD targets based on voxel feature fusion, comprising: a calibration module configured to perform ground calibration on a ground point cloud of a region to be detected to obtain a calibrated point cloud space; a division module configured to divide the calibrated point cloud space into a plurality of voxels in combination with point cloud density, so that there is only one voxel along a Z-axis direction, wherein the Z-axis direction is a height direction of the point cloud space; a fusion module configured to perform feature fusion according to point clouds in each voxel of the plurality of voxels to obtain new point clouds; a segmentation and clustering module configured to segment the new point clouds to obtain FOD point clouds, and cluster the FOD point clouds to detect FODs of different categories.

[0008] In the embodiment of the present application, ground calibration is performed on a ground point cloud of a region to be detected to obtain a calibrated point cloud space; the calibrated point cloud space is divided into a plurality of voxels in combination with point cloud density, so that there is only one voxel along a Z-axis direction, wherein the Z-axis direction is a height direction of the point cloud space; feature fusion is performed according to point clouds in each voxel of the plurality of voxels to obtain new point clouds; the new point clouds are segmented to obtain FOD point clouds, and the FOD point clouds are clustered to detect FODs of different categories, thereby solving the technical problems of poor performance of detecting FOD size and poor real-time performance of FOD detection in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0009] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0010] Figure 1 is a flowchart of a method for intelligent recognition of FOD targets based on voxel feature fusion according to a first embodiment of the present application;

[0011] Figure 2 is a flowchart of a method for intelligent recognition of FOD targets based on voxel feature fusion according to a second embodiment of the present application;

[0012] Figure 3 is a flowchart of a method for intelligent recognition of FOD targets based on voxel feature fusion according to a third embodiment of the present application;

[0013] Figure 4 is a schematic diagram of voxel grid division according to an embodiment of the present application;

[0014] Figure 5 is a schematic diagram of a fusion process of point clouds according to an embodiment of the present application;

[0015] Figure 6This is a flowchart of a method for intelligent cognition of FOD targets based on voxel feature fusion according to the fourth embodiment of this application;

[0016] Figure 7 This is a schematic diagram of the structure of a device for intelligent cognition of FOD targets based on voxel feature fusion according to an embodiment of this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Example 1

[0020] According to embodiments of the present invention, a method for intelligent cognition of FOD targets based on voxel feature fusion is provided, such as... Figure 1 As shown, the method includes:

[0021] Step S102: Perform ground calibration on the ground point cloud of the area to be detected to obtain the calibrated point cloud space.

[0022] In an exemplary embodiment, before performing ground calibration on the ground point cloud of the area to be detected, the method further includes: determining the height and position of the lidar placement according to the actual aircraft runway scenario; and dividing the area to be detected into a cuboid according to the actual scenario, wherein the length, width, and height of the cuboid are set based on the point cloud density and the size of the FOD to be detected.

[0023] Wherein, the long side of the cuboid is the distance range at which the lidar can detect the FOD, the width of the cuboid is selected based on the lidar's field of view and the actual scene, and the height of the cuboid is the height of the lidar's field of view.

[0024] In an exemplary embodiment, ground calibration of the ground point cloud of the area to be detected includes: detecting the ground equation of the area to be detected based on the Random Sample Consensus (RANSAC) plane detection method; calculating a rotation matrix based on the ground normal vector before calibration and the coordinate system normal vector of the lidar; and multiplying the ground point cloud of the area to be detected by the rotation matrix to obtain a horizontally calibrated ground point cloud, thereby obtaining the calibrated point cloud space. The length, width, and height of the point cloud space are L, ... m W m h

[0025] Step S104: Divide the calibrated point cloud space into multiple voxels based on the point cloud density, such that there is only one voxel along the Z-axis direction, wherein the Z-axis direction is the height direction of the point cloud space.

[0026] In an exemplary embodiment, dividing the calibrated point cloud space into multiple voxels based on the point cloud density includes: determining the length L of each voxel based on the point cloud density ρ, the number of points n in each voxel, the growth factor λ, and the scaling factor α; determining the width W of each voxel based on the length L and the scaling factor α; and dividing the calibrated point cloud space into multiple voxels based on the length L and the width W.

[0027] Step S106: Perform feature fusion based on the point cloud of each of the plurality of voxels to obtain a new point cloud.

[0028] In an exemplary embodiment, feature fusion based on the point clouds of each of the plurality of voxels to obtain a new point cloud includes: using the coordinates of the center of each voxel on the X-axis and Y-axis as the coordinates of the new point cloud on the X-axis and Y-axis, wherein the direction of the X-axis is the direction of the wide side of each voxel, and the direction of the Y-axis is the direction of the long side of each voxel; weighting and fusing the average z′ of the Z-axis coordinates of all corrected point clouds in each voxel with the average reflectance to obtain the coordinates of the new point cloud on the Z-axis; and obtaining the new point cloud based on the obtained coordinates of the X-axis, Y-axis, and Z-axis.

[0029] Step S108: Segment the new point cloud to obtain FOD point cloud, and cluster the FOD point cloud to detect different types of FOD.

[0030] In an exemplary embodiment, segmenting the new point cloud to obtain a FOD point cloud includes: detecting the ground of the new point cloud based on the Random Sampling Consensus (RANSAC) plane detection method; calculating the distance L″ from the new point cloud to the detected ground based on the ground equation; and clearing the point when the distance L″ is less than a preset threshold, until the new point cloud has been traversed to obtain a FOD point cloud with the distance L″ greater than the preset threshold.

[0031] In an exemplary embodiment, after clustering the FOD point clouds, the method further includes: performing cuboid labeling based on the various types of FOD point clouds, wherein performing cuboid labeling further includes: finding the original point cloud contained in the voxel corresponding to each type of FOD point cloud in the clustered FOD point clouds; finding the minimum value x along the X-axis direction in each type of original point cloud. min and maximum value x max Minimum value y along the Y-axis min and maximum value y max Minimum value z along the Z-axis min and maximum value z max Calculate the difference x between the maximum and minimum values ​​along the X-axis in the original point cloud for each class. i =x max -x min The difference between the maximum and minimum values ​​along the Y-axis, y i =y max -y min The difference between the maximum and minimum values ​​along the Z-axis, z i =z max -z min According to x min y min z min Forming a new point cloud p i =(x min ,y min ,z min Taking the new point cloud as the starting point of the cuboid, and x i =x max -x min y i =y max -y min z i =z max -z min Each type of FOD is identified by the side length of the cuboid.

[0032] This application proposes an intelligent cognitive method for FOD targets based on voxel feature fusion. By dividing and organizing voxel data into new point cloud data, a large amount of point cloud data is reduced while highlighting FOD point cloud data, which effectively improves the speed of FOD detection and the performance of detecting FOD size.

[0033] Example 2

[0034] To address the problems existing in the prior art, this embodiment provides another intelligent cognitive method for FOD targets based on voxel feature fusion.

[0035] First, the point cloud of the region of interest (ROI) is ground-calibrated, and the ROI is divided into voxels based on the point cloud density, ensuring that only one voxel exists along the Z-axis. Feature fusion is then performed on the point cloud within each voxel to obtain a new point cloud. This new point cloud is then segmented, clustered, and identified to ultimately detect FOD (Foreign Object Defect). This method reduces the amount of point cloud data while highlighting FOD, effectively improving the speed of FOD detection and the performance in detecting FOD size.

[0036] like Figure 2 As shown in this embodiment, the FOD target intelligent cognition method based on voxel feature fusion includes:

[0037] Step S202: Define the region of interest;

[0038] In an exemplary embodiment, the delineation of the region of interest in step S202 includes the following steps: based on the actual aircraft runway scenario, the lidar is placed at a fixed height and fixed position through continuous adjustments; based on the actual scenario, the region of interest is divided into a cuboid; the length, width, and height of the cuboid are set by comprehensively considering the coverage density of the point cloud and the size of the FOD to be detected, the long side of the cuboid is the distance range of the FOD that the lidar can detect, the width of the cuboid is selected according to the field of view of the lidar and the experimental environment, and the height of the cuboid is selected according to the height of the field of view of the lidar.

[0039] Step S204: The collected ground point cloud is calibrated using the RANSAC method;

[0040] In an exemplary embodiment, step S204, which involves calibrating the acquired ground point cloud using the RANSAC method, includes the following steps:

[0041] The ground equation ax+by+cz+d=0 was detected using the RANSAC-based plane detection method in the PCL library.

[0042] The rotation matrix R is calculated based on the ground normal vector n(a,b,c) before calibration and the lidar coordinate system normal vector n1(0,0,1).

[0043] Multiply the point cloud P within the region of interest by the rotation matrix R to obtain the horizontally calibrated point cloud P'.

[0044] Step S206: Divide the calibrated point cloud space into voxels in combination with the point cloud density, such that there is only one voxel in the Z-axis direction. Perform feature fusion based on all the point clouds in each voxel, and finally calculate a new point cloud.

[0045] In an exemplary embodiment, the step of dividing the calibrated point cloud space into voxels in combination with the point cloud density in step S206 includes the following steps:

[0046] Given that the number of point clouds in a unit voxel is n, the total number of point clouds is N, and the voxel side length is L×L×h. Divide the region to be detected into M voxels according to the voxel side length. Since there are few aircraft pavement interference objects and obstacles, and it is relatively空旷 in the height direction, only one voxel is divided in the height direction, so the side length of the voxel in the height direction is h.

[0047] Estimate the voxel side length in combination with the point cloud density. The specific expression is:

[0048] S = L m ×W m

[0049]

[0050]

[0051] where S represents the area of the point cloud in the XOY plane, L m represents the length of the region to be detected, W m represents the width of the region to be detected, M represents the number of voxels into which the region to be detected is divided, and L' represents the estimated voxel side length.

[0052] Through the above formula, find the average density ρ = N / m of the voxels containing point clouds under the side length L'. m is the actual voxels except for the empty voxels. If ρ < n, then L' = L' + λ; if ρ > n, then L' = L' - λ, where λ is the growth factor, until ρ = αn. Here, α is a proportionality factor. Finally, make L = L'. In actual data acquisition, it is found that when there is FOD, the outliers of the point cloud in the X-axis direction are larger than those in the Y-axis direction, so the voxel side length W in the X-axis direction is W = (1 / α)L.

[0053] After multiple iterations, find the appropriate voxel side lengths L, W, h to divide the point cloud P'.

[0054] In an exemplary embodiment, the step of performing feature fusion on all the point clouds in each voxel in step S206 includes the following steps:

[0055] To convert the point cloud from unordered to ordered, there is at most one point in each voxel in the new point cloud, and the x″ and y″ of the new point cloud are taken as the center of the voxel.

[0056] Since the detected FOD may be small and it is difficult to detect with ordinary algorithms, in order to highlight the FOD, the weighted fusion of the average value z′ of the Z-axis coordinate values of all corrected point clouds in the voxel and the average reflectivity is used as the new point cloud z″, and the formula is as follows:

[0057]

[0058] In the formula, g is the weighted value of the point cloud value, is the average value of all corrected point cloud z′ in each voxel, is the average value of the reflectivity of all point clouds in each voxel, and the finally newly generated point cloud is P″(x″, y″, z″).

[0059] Step S208, segment and cluster the newly generated point cloud to obtain the point cloud information of the FOD;

[0060] In an exemplary embodiment, the segmentation of the newly generated point cloud in step S208 includes the following steps:

[0061] Use the plane detection method based on RANSAC in the PCL library to detect the ground equation a'x + b'y + c'z + d' = 0; where, x represents the x value (x″) of the newly generated point cloud, y represents the y value (y″) of the newly generated point cloud, and z represents the z value (z″) of the newly generated point cloud.

[0062] Calculate the distance L″ from any point (x″, y″, z″) after voxel division to this plane as:

[0063]

[0064] Among them, a', b', c', and d' are coefficients automatically generated by the RANSAC method in the pcl library according to the newly generated point cloud.

[0065] When L″ ≤ ξ (ξ = 0.5), then this point is cleared, and finally only the FOD point cloud remains in the space.

[0066] In an exemplary embodiment, the clustering of the newly generated point cloud in step S208 includes the following steps:

[0067] Set the distance threshold r = 0.05;

[0068] Calculate the distance d between two point clouds after segmentation. When d < r, the two points are classified as the same type of FOD point cloud.

[0069] Step S210: Mark each type of FOD with a cuboid.

[0070] In an exemplary embodiment, step S210 of identifying various types of FODs using cuboids includes the following steps: identifying the original point cloud contained in the voxel corresponding to each type of point cloud after clustering; calculating p in each type of original point cloud. i =(x min ,y min ,z min (i = 1, 2…N), with p i =(x min ,y min ,z min (x) is the starting point of the cuboid. i =x max -x min y i =y max -y min z i =z max -z min Each type of FOD is identified by the side length of the cuboid.

[0071] Specifically, find the minimum value x along the X-axis in each type of original point cloud. min and maximum value x max Minimum value y along the Y-axis min and maximum value y max Minimum value z along the Z-axis min and maximum value z max Calculate the difference x between the maximum and minimum values ​​along the X-axis in the original point cloud for each class. i =x max -x min The difference between the maximum and minimum values ​​along the Y-axis, y i =y max -y min The difference between the maximum and minimum values ​​along the Z-axis, z i =z max -z min According to x min y min z min Forming a new point cloud p i =(x min ,y min ,z min Taking the new point cloud as the starting point of the cuboid, and x i y i z i Identify each type of FOD by the side length of the cuboid.

[0072] This embodiment proposes an intelligent cognitive method for FOD targets based on voxel feature fusion. By dividing and organizing voxel data into new point cloud data, a large number of point clouds are reduced while highlighting FOD point clouds, which effectively improves the speed of FOD detection and the performance of detecting FOD size.

[0073] Example 3

[0074] This embodiment provides yet another intelligent cognitive method for FOD targets based on voxel feature fusion. For example... Figure 3 As shown, the method includes the following steps:

[0075] Step S302: Define the area;

[0076] Based on the actual aircraft runway scenario, the lidar was continuously adjusted to a fixed height and position. The region of interest was divided into a cuboid according to the actual scenario. The length, width, and height of the cuboid were set by comprehensively considering the coverage density of the point cloud and the size of the FOD to be detected. The long side of the cuboid is the range of distance at which the lidar can detect the FOD. The width of the cuboid is selected according to the lidar's field of view and the experimental environment. The height of the cuboid is selected according to the height of the lidar's field of view.

[0077] Step S304: Calibrate the collected ground point cloud;

[0078] The ground equation ax+by+cz+d=0 was detected using the RANSAC-based planar detection method in the PCL library; the rotation matrix R was calculated based on the ground normal vector n(a,b,c) before calibration and the lidar coordinate system normal vector n1(0,0,1); the point cloud P within the area of ​​interest was multiplied by the rotation matrix R to obtain the horizontally calibrated point cloud P′.

[0079] Step S306: Divide the calibrated point cloud space into voxels based on the point cloud density, so that there is only one voxel along the Z-axis. Perform feature fusion based on all the point clouds in each voxel, and finally calculate the new point cloud.

[0080] In step S306, the calibrated point cloud space is divided into voxels based on the point cloud density. Figure 4 As shown. Step 306 includes the following steps:

[0081] Given that the number of point clouds within a unit voxel is n and the total number of point clouds is N, let the side length of the voxel be L×L×h. Divide the voxel into M voxels according to the side length. Since there are few interferences and obstacles on the aircraft runway and the airspace is relatively open in the altitude direction, only one voxel is divided in the altitude direction. Therefore, the side length of the voxel in the altitude direction is h.

[0082] The voxel edge length is estimated by combining the point cloud density. The specific expression is:

[0083] S = L m ×L m

[0084]

[0085]

[0086] Find the average density ρ = N / m of the point cloud voxels with side length L′, where m is the actual voxels except for the empty voxels. If ρ < n, then L′ = L′ + λ; if ρ > n, then L′ = L′ - λ, where λ is the growth factor, until ρ = αn. Here, α is a proportionality factor. Finally, make L = L′. In actual data acquisition, it is found that when there is FOD, the outliers of the point cloud in the X-axis direction are larger than those in the Y-axis direction. Therefore, the voxel side length W in the X-axis direction is W = (1 / α)L.

[0087] After multiple iterations, find the appropriate voxel side lengths L, W, and h to divide the point cloud P′.

[0088] Among them, for all the point clouds in each voxel in step S306, feature fusion is performed as Figure 5 shown, including the following steps:

[0089] To convert the point cloud from disordered to ordered, there is at most one point in each voxel in the new point cloud, and the x″ and y″ of the new point cloud take the center of the voxel.

[0090] Since the detected FOD may be small, it is difficult to detect it with ordinary algorithms. Therefore, to highlight the FOD, the average value of the Z-axis coordinate values of all the corrected point clouds in the voxel is weighted and fused with the average reflectivity as the new point cloud z″. The formula is as follows:

[0091]

[0092] In the formula, g is the weighted value of the point cloud value, and the finally newly generated point cloud is P″(x″, y″, z″).

[0093] Step S308: Obtain the point cloud information of the FOD;

[0094] Segment and cluster the newly generated point cloud to obtain the point cloud information of the FOD. Among them, the segmentation of the newly generated point cloud in step S308 includes the following steps:

[0095] Use the plane detection method based on RANSAC in the PCL library to detect the ground equation a'x + b'y + c'z + d' = 0;

[0096] The distance L″ from any point (x″, y″, z″) after the computational voxel division to this plane is:

[0097]

[0098] When L″ ≤ ξ (ξ = 0.5), then this point is cleared, and finally only the FOD point cloud remains in the space.

[0099] Among them, the clustering of the newly generated point cloud in step S308 includes the following steps:

[0100] Set the distance threshold r = 0.05;

[0101] Calculate the distance d between two point clouds after segmentation. When d < r, the two points are classified as the same type of FOD point cloud.

[0102] Step S3110: Identify each type of FOD with a cuboid.

[0103] Among them, the identification of each type of FOD with a cuboid in step S3110 includes the following steps:

[0104] Find out the original point cloud contained in the voxel corresponding to each type of point cloud after clustering;

[0105] Calculate p in each type of original point cloud i =(x min , y min , z min )(i = 1, 2…N). Taking p i =(x min , y min , z min ) as the starting point of the cuboid, x i = x max - x min , y i = y max - y min , z i = z max - z min as the side length of the cuboid to identify each type of FOD.

[0106] Example 4

[0107] This embodiment discloses an intelligent cognitive method for FOD targets based on voxel feature fusion, which is applicable to the field of airport FOD detection. In this embodiment, (1) the region of interest is delineated; (2) the collected ground point cloud is calibrated using the RANSAC method; (3) the calibrated point cloud space is divided into voxels based on the point cloud density, so that there is only one voxel along the Z-axis; feature fusion is performed on all the point clouds in each voxel, and finally a new point cloud is calculated; (4) the newly generated point cloud is segmented and clustered to obtain the point cloud information of FOD; (5) various types of FOD are labeled with cuboids.

[0108] like Figure 6 As shown, the method includes the following steps:

[0109] Step S602: Define the region of interest;

[0110] Step S604: The collected ground point cloud is calibrated using the RANSAC method;

[0111] Step S606: Divide the calibrated point cloud space into voxels based on the point cloud density, so that there is only one voxel along the Z-axis. Perform feature fusion based on all the point clouds in each voxel, and finally calculate the new point cloud.

[0112] Step S608: Segment and cluster the newly generated point cloud to obtain the point cloud information of FOD;

[0113] Step S610: Mark each type of FOD with a cuboid.

[0114] In an exemplary embodiment, the delineation of the region of interest in step S602 includes the following steps: based on the actual aircraft runway scenario, the lidar is placed at a fixed height and fixed position through continuous adjustments; based on the actual scenario, the region of interest is divided into a cuboid; the length, width, and height of the cuboid are set by comprehensively considering the coverage density of the point cloud and the size of the FOD to be detected, the long side of the cuboid is the distance range at which the lidar can detect the FOD, the width of the cuboid is selected according to the lidar's field of view and the experimental environment, and the height of the cuboid is selected according to the height of the lidar's field of view.

[0115] In an exemplary embodiment, the calibration of the acquired ground point cloud using the RANSAC method in step S604 includes the following steps: detecting the ground equation ax+by+cz+d=0 using the RANSAC-based plane detection method in the PCL library; calculating the rotation matrix R based on the ground normal vector n(a,b,c) before calibration and the lidar coordinate system normal vector n1(0,0,1); and multiplying the point cloud P within the area of ​​interest by the rotation matrix R to obtain the horizontally calibrated point cloud P′.

[0116] In an exemplary embodiment, dividing the calibrated point cloud space into voxels in step S606 in combination with the point cloud density includes the following steps: Given that the number of point clouds in a unit voxel is n, the total number of point clouds is N, assuming the voxel side length is L×L×h, M voxels are divided according to the voxel side length. Since there are few aircraft pavement interference objects and obstacles, and it is relatively空旷 in the height direction, only one voxel is divided in the height direction; estimate the voxel side length in combination with the point cloud density. The specific expression is:

[0117] S = L m ×L m

[0118]

[0119]

[0120] In the formula, L′ represents the estimated voxel side length, S represents the area of the point cloud in the XOY plane, and the average density ρ = N / m of the voxel containing the point cloud under the size of the side length L′ is obtained, where m is the actual voxel except for the empty voxel. If ρ < n, then L = L′ + λ; if ρ > n, then L′ = L′ - λ, where λ is the growth factor until ρ = αn. Here, α is a proportionality factor. Finally, make L = L′. In actual data acquisition, it will be found that when there is FOD, the outliers of the point cloud in the X-axis direction are larger than those in the Y-axis direction. Therefore, the voxel side length W in the X-axis direction is W = (1 / α)L.

[0121] After multiple iterations, find the appropriate voxel side lengths L, W, and h to divide the point cloud P′.

[0122] In an exemplary embodiment, feature fusion of all point clouds in each voxel in step S606 includes the following steps: To convert the point cloud from disordered to ordered point cloud, there is at most one point in each voxel in the new point cloud, and the x″ and y″ of the new point cloud take the center of the voxel. Since the detected FOD may be small and it is difficult to detect with ordinary algorithms, in order to highlight the FOD, use the average value of the Z-axis coordinate values of all corrected point clouds in the voxel And the average reflectivity are weighted and fused as the new point cloud z″. The formula is as follows:

[0123]

[0124] In the formula, g is the weighted value of the point cloud value, and the finally newly generated point cloud is P″(x″, y″, z″).

[0125] In an exemplary embodiment, the segmentation of the newly generated point cloud in step S608 includes the following steps: detecting the ground equation a'x + b'y + c'z + d' = 0 using the plane detection method based on RANSAC in the PCL library; calculating the distance L″ from any point (x″, y″, z″) after voxel division to the plane as:

[0126]

[0127] When L″ ≤ ξ (ξ = 0.5), this point is cleared, and finally only the FOD point cloud remains in the space.

[0128] In an exemplary embodiment, the clustering of the newly generated point cloud in step S608 includes the following steps: setting the distance threshold r = 0.05; calculating the distance d between two point clouds after segmentation, and when d < r, the two points are classified as the same type of FOD point cloud.

[0129] In an exemplary embodiment, the cuboid identification of each type of FOD in step S610 includes the following steps: finding the original point cloud contained in the voxel corresponding to each type of point cloud after clustering; finding the minimum value xmin and the maximum value xmax in the X-axis direction, the minimum value ymin and the maximum value ymax in the Y-axis direction, and the minimum value zmin and the maximum value zmax in the Z-axis direction in each type of original point cloud, and calculating the difference between the maximum value and the minimum value in the X-axis direction xi = xmax - xmin, the difference between the maximum value and the minimum value in the Y-axis direction yi = ymax - ymin, and the difference between the maximum value and the minimum value in the Z-axis direction zi = zmax - zmin in each type of original point cloud; forming a new point cloud p i =(x min ,y min ,z min ), using the new point cloud as the starting point of the cuboid, and using x i = x max - x min , y i = y max - y min , z i = z max - z min as the side lengths of the cuboid to identify each type of FOD.

[0130] This application is mainly used for the detection of foreign objects on the aircraft runway. This method generates a new point cloud through voxel division, reduces the computational amount required in the FOD detection process, shortens the FOD detection time, improves the performance of detecting the size of FOD in FOD detection, and can provide an efficient and feasible technical method for the FOD detection field, which is beneficial to the application and development of lidar in the FOD detection technology.

[0131] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0133] Example 5

[0134] According to embodiments of the present invention, a device for intelligent cognition of FOD targets based on voxel feature fusion is also provided, such as... Figure 7 The device shown includes: a calibration module 52, a partitioning module 54, a fusion module 56, and a segmentation and clustering module 58.

[0135] The calibration module 52 is configured to perform ground calibration on the ground point cloud of the area to be detected, and obtain the calibrated point cloud space.

[0136] The partitioning module 54 is configured to divide the calibrated point cloud space into multiple voxels based on the point cloud density, such that there is only one voxel along the Z-axis direction, wherein the Z-axis direction is the height direction of the point cloud space.

[0137] The fusion module 56 is configured to perform feature fusion based on the point cloud in each of the plurality of voxels to obtain a new point cloud;

[0138] The segmentation and clustering module 58 is configured to segment the new point cloud to obtain a FOD point cloud, and cluster the FOD point cloud to detect different classes of FOD.

[0139] In one exemplary embodiment, the device further includes a location determination module, and is further configured to, before performing ground calibration on the ground point cloud of the area to be detected, the method further includes: determining the height and position of the lidar placement based on an actual aircraft runway scenario; and dividing the area to be detected into a cuboid based on the actual scenario, wherein the length, width, and height of the cuboid are set based on the point cloud density and the size of the FOD to be detected.

[0140] Wherein, the long side of the cuboid is the distance range at which the lidar can detect the FOD, the width of the cuboid is selected based on the lidar's field of view and the actual scene, and the height of the cuboid is the height of the lidar's field of view.

[0141] In an exemplary embodiment, the calibration module 52 is further configured to perform ground calibration on the ground point cloud of the area to be detected, including: detecting the ground equation of the area to be detected based on the Random Sampling Consensus (RANSAC) plane detection method; calculating a rotation matrix based on the ground normal vector before calibration and the coordinate system normal vector of the lidar; and multiplying the ground point cloud of the area to be detected by the rotation matrix to obtain the horizontally calibrated ground point cloud, so as to obtain the calibrated point cloud space.

[0142] In an exemplary embodiment, the partitioning module 54 is further configured to: determine the length L of each voxel based on the point cloud density ρ, the number of points n in each voxel, the growth factor λ, and the scaling factor α; determine the width W of each voxel based on the length L and the scaling factor α; and partition the calibrated point cloud space into multiple voxels based on the length L and the width W.

[0143] In an exemplary embodiment, the fusion module 56 is configured to: perform feature fusion based on the point cloud of each of the plurality of voxels to obtain a new point cloud, including: using the coordinate values ​​of the center of each voxel on the X-axis and Y-axis as the coordinate values ​​of the new point cloud on the X-axis and Y-axis, wherein the direction of the X-axis is the direction of the wide side of each voxel, and the direction of the Y-axis is the direction of the long side of each voxel; and averaging the Z-axis coordinate values ​​of all corrected point clouds within each voxel. The coordinates of the new point cloud on the Z-axis are obtained by weighted fusion with the average reflectance; based on the obtained coordinates of the X-axis, Y-axis, and Z-axis, the new point cloud is obtained.

[0144] In an exemplary embodiment, the segmentation and clustering module 58 is further configured to: segment the new point cloud to obtain a FOD point cloud, including: detecting the ground of the new point cloud based on the Random Sample Consensus (RANSAC) plane detection method; calculating the distance L″ from the new point cloud to the detected ground based on the ground equation; and clearing the point when the distance L″ is less than a preset threshold, until the new point cloud has been traversed to obtain a FOD point cloud with the distance L″ greater than the preset threshold.

[0145] In one exemplary embodiment, the device further includes an identification module configured to: perform cuboid identification based on the various types of FOD point clouds, wherein performing cuboid identification further includes: finding the original point cloud contained in the voxel corresponding to each type of FOD point cloud after clustering; and finding the minimum value x along the X-axis in each type of original point cloud. min and maximum value x max Minimum value y along the Y-axis min and maximum value y max Minimum value z along the Z-axis min and maximum value z max Calculate the difference x between the maximum and minimum values ​​along the X-axis in the original point cloud for each class. i =x max -x min The difference between the maximum and minimum values ​​along the Y-axis, y i =y max -y min The difference between the maximum and minimum values ​​along the Z-axis, z i =z max -z min According to x min y min z min Forming a new point cloud p i =(x min ,y min ,z min Taking the new point cloud as the starting point of the cuboid, and x i =x max -x min y i =y max -y min z i =z max -z min Each type of FOD is identified by the side length of the cuboid.

[0146] The apparatus provided in this embodiment can implement the methods in embodiments 1 to 4, which will not be described again here.

[0147] Example 6

[0148] Embodiments of the present invention also provide a storage medium having a program stored thereon, which, when run, enables a computer to execute the methods of embodiments 1 to 4.

[0149] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0150] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0151] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0152] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for FOD target intelligent cognition based on voxel feature fusion, characterized in that, The method comprises: ground calibration is performed on the ground point cloud of the region to be detected to obtain a calibrated point cloud space; the calibrated point cloud space is divided into a plurality of voxels in combination with the point cloud density, so that there is only one voxel in the Z-axis direction, wherein the Z-axis direction is the height direction of the point cloud space; feature fusion is performed on the point cloud in each voxel of the plurality of voxels to obtain a new point cloud; the new point cloud is segmented to obtain a FOD point cloud, and the FOD point cloud is clustered to obtain FODs of different categories to detect the FODs; wherein dividing the calibrated point cloud space into a plurality of voxels in combination with the point cloud density comprises: determining the length L of each voxel based on the point cloud density ρ, the number of point clouds n in each voxel, the growth factor λ and the proportion factor α; determining the width W of each voxel based on the length L and the proportion factor α; and dividing the calibrated point cloud space into a plurality of voxels based on the length L and the width W; The feature fusion according to the point cloud in each voxel in the plurality of voxels obtains a new point cloud, including: taking the coordinate values of the center of each voxel on the X axis and the Y axis as the coordinate values of the new point cloud on the X axis and the Y axis, wherein the direction of the X axis is the direction of the wide side of each voxel, and the direction of the Y axis is the direction of the long side of each voxel; taking the average value of the Z axis coordinate values of all the corrected point clouds in each voxel The weighted fusion with the average reflectivity obtains the coordinate value of the new point cloud on the Z axis; and the new point cloud is obtained based on the obtained coordinate values of the X axis, the Y axis and the Z axis. wherein segmenting the new point cloud to obtain a FOD point cloud comprises: detecting the ground of the new point cloud based on a random sample consensus (RANSAC) plane detection method; calculating the distance L" of the new point cloud to the detected ground based on a ground equation; and when the distance L" is less than a preset threshold, clearing the point until the new point cloud is traversed, so that only the FOD point cloud with the distance L" greater than the preset threshold remains in the final space.

2. The method of claim 1, wherein, Before ground calibration is performed on the ground point cloud of the region to be detected, the method further comprises: determining the height and position of the laser radar based on the actual aircraft runway scene; dividing the region to be detected into a cuboid based on the actual scene, wherein the length, width and height of the cuboid are set based on the point cloud density and the size of the FOD to be detected.

3. The method of claim 2, wherein, The long side of the cuboid is the distance range within which the FOD can be detected by the laser radar, the width of the cuboid is selected according to the field of view range of the laser radar and the actual scene, and the height of the cuboid is the height of the field of view range of the laser radar.

4. The method of claim 1, wherein, The ground calibration of the ground point cloud of the region to be detected comprises: detecting the ground equation of the region to be detected based on a random sample consensus (RANSAC) plane detection method; calculating a rotation matrix based on the normal vector of the ground before calibration and the normal vector of the coordinate system of the laser radar; multiplying the ground point cloud of the region to be detected by the rotation matrix to obtain the horizontally calibrated ground point cloud, thereby obtaining the calibrated point cloud space.

5. The method of claim 1, wherein, The clustering of the FOD point cloud comprises: calculating the distance between each two FOD point clouds after segmentation; when the distance between each two FOD point clouds is less than a preset distance threshold, the FOD point clouds are classified into the same category of point clouds, and finally the FOD point clouds of different categories are obtained.

6. The method of claim 5, wherein, After the FOD point cloud is clustered, the method further comprises identifying the cuboid based on the FOD point cloud, wherein identifying the cuboid further comprises: finding the original point cloud contained in each voxel corresponding to each category of FOD point cloud after clustering; Find the minimum value x along the X-axis direction in each category of original point cloud min and the maximum value x max , the minimum value y along the Y-axis direction min and the maximum value y max , the minimum value z along the Z-axis direction min and the maximum value z max , the difference between the maximum value and the minimum value along the X-axis direction in each category of original point cloud x i = x max - x min , the difference between the maximum value and the minimum value along the Y-axis direction y i = y max - y min , the difference between the maximum value and the minimum value along the Z-axis direction z i = z max - z min ; According to x min , y min , z min , the new point cloud p i = (x min , y min , z min ) is composed, and each type of FOD is identified with the new point cloud as the starting point of the cuboid and x i , y i , z i as the edge length of the cuboid.

7. An apparatus for FOD target intelligent cognition based on voxel feature fusion, characterized in that, ​ The calibration module is configured to perform ground calibration on a ground point cloud of a to-be-detected area to obtain a calibrated point cloud space; The division module is configured to divide the calibrated point cloud space into a plurality of voxels in combination with point cloud density, so that there is only one voxel in the Z-axis direction, wherein the Z-axis direction is a height direction of the point cloud space; The fusion module is configured to perform feature fusion according to the point cloud in each voxel of the plurality of voxels to obtain a new point cloud; The segmentation and clustering module is configured to segment the new point cloud to obtain a FOD point cloud, and cluster the FOD point cloud to obtain FODs of different categories to detect FODs; The device is further configured to determine the length L of each voxel based on the point cloud density ρ, the number n of point clouds in each voxel, a growth factor λ, and a proportion factor α; determine the width W of each voxel based on the length L and the proportion factor α; and divide the calibrated point cloud space into a plurality of voxels based on the length L and the width W. The device is further configured to: take the coordinate values of the center of each voxel on the X-axis and the Y-axis as the coordinate values of the new point cloud on the X-axis and the Y-axis, wherein the direction of the X-axis is the direction of the wide side of each voxel, and the direction of the Y-axis is the direction of the long side of each voxel; take the average value of the Z-axis coordinate values of all the corrected point clouds in each voxel as the Z-axis coordinate value of the new point cloud The Z-axis coordinate value of the new point cloud is obtained by weighted fusion with the average reflectivity; and the new point cloud is obtained based on the obtained X-axis, Y-axis and Z-axis coordinate values. The device is further configured to detect the ground of the new point cloud based on a random sample consensus (RANSAC) plane detection method; calculate the distance L" of the new point cloud to the detected ground based on a ground equation; and clear the point when the distance L" is less than a preset threshold, until the new point cloud is traversed, so that only the FOD point cloud with the distance L" greater than the preset threshold is left in the final space.

Images