A method for low-altitude obstacle recognition based on the fusion of lidar and visual sensors

Abstract

This invention discloses a low-altitude obstacle recognition method based on the fusion of lidar and visual sensors. The method includes: lidar data of the low-altitude environment provided by the lidar; visual data of the low-altitude environment provided by the visual sensor; and fusion processing of the lidar data and visual data to identify low-altitude obstacles. The advantages of this invention are: by fusing lidar data and visual data, the two data complement each other, enabling multi-angle and multi-faceted detection of obstacles, effectively avoiding the problem of single sensor failure in specific environments, thereby improving the reliability and accuracy of low-altitude obstacle recognition.

Description

Technical Field

[0001] This invention relates to low-altitude obstacle recognition for aircraft, and more particularly, to a low-altitude obstacle recognition method based on the fusion of lidar and visual sensors. Background Technology

[0002] With the increasing complexity of modern combat missions, aircraft need to complete various low-altitude flight operations, making flight safety and low-altitude obstacle recognition issues a major concern. Currently, most obstacle recognition is accomplished through environmental perception platforms composed of various sensors, and some progress has been made. However, in complex low-altitude environments, environmental interference can affect obstacle recognition accuracy, leading to a decline in recognition performance, especially for detailed information and small-scale obstacles, where the accuracy still falls far short of the requirements for low-altitude flight. Summary of the Invention

[0003] The purpose of this invention is to solve the problems of the prior art and provide a low-altitude obstacle recognition method based on the fusion of lidar and visual sensors.

[0004] To achieve the above objectives, the present invention provides a technical solution: a low-altitude obstacle recognition method based on the fusion of lidar and visual sensors, comprising: lidar data of the low-altitude environment provided by the lidar; visual data of the low-altitude environment provided by the visual sensor; and fusion processing of the lidar data and visual data to identify low-altitude obstacles.

[0005] As a preferred scheme for low-altitude obstacle recognition, it includes: automatic calibration mechanism step S1, adaptive resolution matching step S2, network backend optimization step S3, data layer fusion step S4, and deep feature fusion step S5.

[0006] As a preferred scheme for low-altitude obstacle recognition methods, the automatic calibration mechanism step S1 includes: sub-step S11, defining the visual image reference coordinate system as {V}, defining the radar reference coordinate system as {L}, and defining the transformation parameter ζ. VL =(t x ,t y ,t z Let φ, θ, ψ represent translations along the x-axis, y-axis, and z-axis, and rotations around the x, y, and z axes, respectively; homogeneous coordinates are used, utilizing ζ. VL Construct the transformation matrix T VL This involves converting 3D points in the visual image reference coordinate system into points in the radar reference coordinate system. For the 3D point cloud in the lidar, this is done using... Representing distance measurements, a depth estimate is obtained through a global matching stereo algorithm, mapping each pixel to a coordinate system (x, y, z), thus producing a point cloud-like structure. The similarity between two clouds is used to determine the transition between two sensors; where, It represents the original coordinate system {L} of the laser. This represents the original coordinate system {V} of the vision sensor; sub-step S12, find the geometric shape represented in the original point cloud, and find the geometric shape P. io A subset of points:

[0007] Using the planar shape from sub-step S11, a consensus-based planar sample segmentation method is executed to determine the planar model; in sub-step S13, the four circles on the alignment target are segmented, with the circle centers serving as key points between two clouds, and Euclidean distance is used for clustering, with the cluster centroids used in the alignment phase; and in sub-step S14, the iterative nearest point algorithm is used to minimize the sum of point-to-point distances between the cluster centroid clouds.

[0008] As a preferred scheme for low-altitude obstacle recognition methods, the adaptive resolution matching step S2 includes: extracting an n*n segment from the depth map D, defined as y i The pixels in the segment are numbered sequentially from 1 to 2. n Number the pixels and fill the remaining pixels with Gaussian regression values.

[0009] As a preferred embodiment of the low-altitude obstacle recognition method, the network backend optimization step S3 includes: sub-step S31, initialization operation, for each pixel i, let: Q I (l) are mutually independent marginal distributions. Sub-step S32, information transmission: Before the network model converges, the image after initialization is Gaussian blurred using a Gaussian kernel, let: Sub-step S33, weighted summation: Perform a weighted summation on the output of the Gaussian filter, letting... And, in sub-step S34, add univariate potential energy by directly subtracting the univariate potential energy obtained from the convolutional neural network from the output obtained in the previous step, letting...

[0010]

[0011] As a preferred scheme for low-altitude obstacle recognition, step S4, data layer fusion: Sub-step S41, construct a rectangle that can contain all scattered points and record the positions of the four vertices D1, D2, D3, and D4 of the rectangle; Sub-step S42, randomly select a point P1 from the scattered points and form four triangles with the four vertices of the rectangle; Sub-step S43, take any point P2 that is different from P1 from the scattered points, determine the vertex of the triangle where P2 is located, construct the circumcircle of all triangles containing these three vertices, and find the triangle containing point P2; when these triangles have a common edge, delete the common edge, and connect P2 with the vertices of the remaining triangles to complete the insertion of a point; Sub-step S44, optimize the formed local triangulation results according to the principle of triangulation, and treat the triangle with a common edge as a quadrilateral; according to the spatial triangulation... The process begins with the circle criterion: constructing the circumcircle of one of the triangles, determining if the four vertices of the quadrilateral are within the circumcircle. If they are, the diagonals of the quadrilateral are swapped to complete local optimization. Sub-step S45 repeats sub-steps S43 and S44 until all scattered points are inserted, completing the triangulation. Sub-step S46 assigns numbers to the discrete points and the triangles generated after DT triangulation, recording the endpoints of each triangle. Sub-step S47 calculates and records the center of the circumcircle of each triangle. Sub-step S48 iterates through each triangle, finding three triangles adjacent to the current triangle. Sub-step S49, if three adjacent triangles exist, connect the circumcenters of the three adjacent triangles to the circumcenter of the current triangle; these three sides are the Vino edges. If no adjacent triangles exist, the perpendicular bisector of the outermost triangle is taken as the Vino edge.

[0012] As a preferred scheme for low-altitude obstacle recognition, step S5, deep feature fusion includes: sub-step S51, using the back-end optimization model of the network in step one, taking the original image acquired by the visual sensor as the front-end input of the network model to obtain univariate potential energy; sub-step S52, inputting the obtained univariate potential energy, the Gaussian filtering result of the original image, and the Gaussian filtering result of the radar image into the back-end network and performing iteration; and sub-step S53, optimizing the classification prediction result, and the generated error can be backpropagated to the front-end network to achieve an end-to-end processing mode.

[0013] Compared with the prior art, the beneficial effects of the present invention are at least as follows: by fusing lidar data and visual data, the two complement each other, enabling multi-angle and multi-faceted detection of obstacles, effectively avoiding the problem of single sensor failure in specific environments, thereby improving the reliability and accuracy of low-altitude obstacle recognition. Attached Figure Description

[0014] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. It should be noted that these descriptions of embodiments are intended to aid in understanding the invention and do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0016] See Figure 1 The figure shows a low-altitude obstacle recognition method based on the fusion of lidar and visual sensors.

[0017] The low-altitude obstacle identification method is embedded in software within any device on an aircraft that has data acquisition, processing, output, and storage functions, such as ground proximity warning equipment, navigation systems, flight control systems, and other avionics equipment.

[0018] The described low-altitude obstacle recognition method enables the simultaneous capture of various physical properties of the external world by heterogeneous sensors. It addresses differences in coordinate systems, temporal and spatial resolution, and data formats among sensor data streams. Data preprocessing filters and selects image data from all sensors, eliminating unsuitable data. An automatic calibration mechanism establishes correspondences between data obtained from different angles, matching each data point output by the sensor with the corresponding pixel in the visual sensor output, aligning the data streams spatially and temporally. A convolutional neural network backend optimization mechanism enhances sensitivity to small-scale information. The method integrates LiDAR and visual sensor images at both the data level and high-order feature levels to resolve spatial and category information of obstacles.

[0019] The specific process of the low-altitude obstacle recognition method is as follows:

[0020] Step S1, Automatic Calibration Mechanism: Sub-step S11, defines the visual image reference coordinate system as {V}, the radar reference coordinate system as {L}, and defines the transformation parameter ζ. VL =(t x ,t y ,t z Let φ, θ, ψ represent translations along the x-axis, y-axis, and z-axis, and rotations about the x, y, and z axes, respectively. Homogeneous coordinates are used, utilizing ζ. VL Construct the transformation matrix T VL This involves converting 3D points in the visual image reference coordinate system into points in the radar reference coordinate system. For the 3D point cloud in the lidar, this is done using... Representing distance measurements, a depth estimate is obtained through a global matching stereo algorithm, mapping each pixel to a coordinate system (x, y, z), thus producing a point cloud-like structure. The transition between two sensors is determined by the similarity between two clouds. Wherein, P o l It represents the original coordinate system {L} of the laser. This is a representation of the original coordinate system {V} of the vision sensor. Sub-step S12: Find the geometric shape represented by the original point cloud, and find the geometric shape P. io A subset of points: Using the planar shape from sub-step S11, a consensus-based planar sample segmentation method is executed to determine the planar model. Sub-step S13 involves segmenting the four circles on the alignment target, using the circle centers as key points between two clouds, and clustering using Euclidean distance, with the cluster centroids used in the alignment phase. Sub-step S14 employs an iterative nearest-point algorithm to minimize the sum of point-to-point distances between the cluster centroid clouds.

[0021] Step S2, Adaptive resolution matching: Extract an n*n segment from the depth map D, defined as y i The pixels in the segment are numbered sequentially from 1 to 2. n Number the pixels and fill the remaining pixels with Gaussian regression values.

[0022] Step S3, Network Backend Optimization Model: Sub-step S31, Initialization Operation, for each pixel i, let: Q I (l) are mutually independent marginal distributions. Sub-step S32, information transmission: Before the network model converges, the image after initialization is Gaussian blurred using a Gaussian kernel, let: Sub-step S33, weighted summation: Perform a weighted summation on the output of the Gaussian filter, letting... Sub-step S34: Add univariate potential energy by directly subtracting the univariate potential energy obtained from the convolutional neural network from the output obtained in the previous step, letting...

[0023] Step S4, Data Layer Fusion: Sub-step S41, Construct a rectangle that can contain all scattered points and record the positions of the four vertices D1, D2, D3, and D4 of the rectangle; Sub-step S42, Randomly select a point P1 from the scattered points and form four triangles with the four vertices of the rectangle; Sub-step S43, Using any point P2 that is different from P1 in the scattered points, determine the vertex of the triangle where P2 is located, draw the circumcircle of all triangles containing these three vertices, and find the triangle containing point P2. If these triangles have a common edge, delete the common edge and connect P2 with the vertices of the remaining triangles to complete the insertion of a point; Sub-step S44, Optimize the formed local triangulation results according to the principle of triangulation, and treat triangles with common edges as quadrilaterals; According to the empty circle criterion of triangulation, draw the circumcircle of one of the triangles, and determine whether the four vertices of the quadrilateral are inside the circumcircle. If they are inside the circumcircle, swap the diagonals of the quadrilateral to complete the local optimization. Sub-step S45: Repeat sub-steps S43 and S44 until all scattered points are inserted and the triangulation is completed; Sub-step S46: Number the discrete points and the triangles generated by DT triangulation, and record the endpoints of each triangle; Sub-step S47: Calculate and record the circumcenter of each triangle; Sub-step S48: Traverse each triangle and find the three triangles adjacent to the current triangle; Sub-step S49: If there are three adjacent triangles, connect the circumcenters of the three adjacent triangles to the circumcenter of the current triangle, and these three sides are the Vino edges; if there are no adjacent triangles, the perpendicular bisector of the outermost side is taken as the Vino edge.

[0024] Step S5, Deep Feature Fusion: Sub-step S51, Optimize the backend model using the network from Step 1, taking the original image acquired by the visual sensor as the frontend input of the network model to obtain univariate potential energy; Sub-step S52, Input the obtained univariate potential energy, the Gaussian filtering result of the original image, and the Gaussian filtering result of the radar image into the backend network and iterate; Sub-step S53, Optimize the classification prediction result, and the generated error can be backpropagated to the frontend network to achieve an end-to-end processing mode.

[0025] The above description merely illustrates embodiments of the present invention and is quite specific and detailed; however, it should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A low-altitude obstacle identification method based on fusion of lidar and visual sensors, characterized in that, include: The lidar provides lidar data for the low-altitude environment; Visual data of the low-altitude environment is provided by a visual sensor; The LiDAR data and visual data are fused together to identify low-altitude obstacles.

2. The low obstacle recognition method according to claim 1, characterized in that, include: The automatic calibration mechanism steps are S1, adaptive resolution matching steps are S2, network backend optimization steps are S3, data layer fusion steps are S4, and deep feature fusion steps are S5.

3. The low-altitude obstacle identification method according to claim 2, characterized in that, The automatic calibration mechanism step S1 includes: sub-step S11, defining the visual image reference coordinate system as {V}, defining the radar reference coordinate system as {L}, and defining the transformation parameter ζ. VL =(t x ,t y ,t z Let φ, θ, ψ represent translations along the x-axis, y-axis, and z-axis, and rotations around the x, y, and z axes, respectively; homogeneous coordinates are used, utilizing ζ. VL Construct the transformation matrix T VL This involves converting 3D points in the visual image reference coordinate system into points in the radar reference coordinate system. For the 3D point cloud in the lidar, this is done using... Representing distance measurements, a depth estimate is obtained through a global matching stereo algorithm, mapping each pixel to a coordinate system (x, y, z), thus producing a point cloud-like structure. The similarity between two clouds is used to determine the transition between two sensors; where, It represents the original coordinate system {L} of the laser. This represents the original coordinate system {V} of the vision sensor; sub-step S12, find the geometric shape represented in the original point cloud, and find the geometric shape P. io A subset of points: Using the planar shape from sub-step S11, a consensus-based planar sample segmentation method is executed to determine the planar model; in sub-step S13, the four circles on the alignment target are segmented, with the circle centers serving as key points between two clouds, and Euclidean distance is used for clustering, with the cluster centroids used in the alignment phase; and in sub-step S14, the iterative nearest point algorithm is used to minimize the sum of point-to-point distances between the cluster centroid clouds.

4. The low-altitude obstacle identification method according to claim 2, characterized in that, The adaptive resolution matching step S2 comprises: extracting a n*n patch from the depth map D, defined as y i . The pixels in the patch are numbered from 1 to 2 n , and the rest of the pixels are filled with the Gaussian regression process value.

5. The low-altitude obstacle identification method according to claim 2, characterized in that, Network backend optimization step S3 includes: sub-step S31, initialization operation, for each pixel i, let: Q I (l) are mutually independent marginal distributions. Sub-step S32, information transmission: Before the network model converges, the image after initialization is Gaussian blurred using a Gaussian kernel, let: Sub-step S33, weighted summation: Perform a weighted summation on the output of the Gaussian filter, letting... And, in sub-step S34, add univariate potential energy by directly subtracting the univariate potential energy obtained from the convolutional neural network from the output obtained in the previous step, letting...

6. The low-altitude obstacle identification method according to claim 2, characterized in that, Step S4, Data Layer Fusion: Sub-step S41, Construct a rectangle that can contain all scattered points and record the positions of the four vertices D1, D2, D3, and D4 of the rectangle; Sub-step S42, Randomly select a point P1 from the scattered points and form four triangles with the four vertices of the rectangle; Sub-step S43, Using any point P2 that is different from P1 in the scattered points, determine the vertex of the triangle where P2 is located, construct the circumcircle of all triangles containing these three vertices, and find the triangle containing point P2; If these triangles have a common edge, delete the common edge, and connect P2 with the vertices of the remaining triangles to complete the insertion of a point; Sub-step S44, Optimize the formed local triangulation results according to the principle of triangulation, and treat triangles with common edges as quadrilaterals; According to the empty circle criterion of triangulation, construct one of them... For each triangle, the circumcircle is used to determine if the four vertices of the quadrilateral are inside the circumcircle. If they are, the diagonals of the quadrilateral are swapped to complete local optimization. Sub-step S45 repeats sub-steps S43 and S44 until all scattered points are inserted, completing the triangulation. Sub-step S46 assigns numbers to the discrete points and the triangles generated by DT triangulation, recording the endpoints of each triangle. Sub-step S47 calculates and records the center of the circumcircle of each triangle. Sub-step S48 iterates through each triangle, finding three triangles adjacent to the current triangle. Sub-step S49, if three adjacent triangles exist, the circumcenters of the three adjacent triangles are connected to the circumcenter of the current triangle; these three sides are the Vino edges. If no adjacent triangles exist, the perpendicular bisector of the outermost triangle is taken as the Vino edge.

7. The low-altitude obstacle identification method according to claim 2, characterized in that, Step S5, deep feature fusion includes: sub-step S51, using the back-end optimization model of the network in step one, taking the original image acquired by the visual sensor as the front-end input of the network model to obtain univariate potential energy; sub-step S52, inputting the obtained univariate potential energy, the Gaussian filtering result of the original image, and the Gaussian filtering result of the radar image into the back-end network and iterating; and sub-step S53, optimizing the classification prediction result, and the generated error can be backpropagated to the front-end network to achieve an end-to-end processing mode.

Images