A machine vision-based obstacle avoidance method for special vehicles at airports

Abstract

The present invention discloses a machine vision-based obstacle avoidance method for airport special vehicles, comprising the following steps: simulating the airport scene, using a monocular camera and a Velodyne 64-line laser radar to collect data information; preprocessing the point cloud data collected by the sensor; using the RandLA‑Net model to directly process the training data set and extract features for the point cloud data; using the convolutional neural network ResNet to extract features from the RGB image; using two types of features as inputs of the fusion network, combining and abstracting the features, outputting the 3D bounding box of the target object, detecting obstacle information, and achieving obstacle avoidance. The present invention applies the fused multi-dimensional feature information to the field of civil aviation ground vehicle services for the first time, and achieves safe obstacle avoidance for airport special vehicles by identifying obstacles.

Description

Technical Field

[0001] The invention belongs to the field of autonomous obstacle detection of special vehicles of airports in ground service tasks, and particularly relates to an obstacle avoidance and anti-collision method for special vehicles based on vision fusion. Background Art

[0002] Since the beginning of the 21st century, my country has comprehensively promoted the development of my country's civil aviation industry from many aspects, including airport construction planning, aviation network layout, and air transport services. As one of the important ground support equipment at airports, airport special vehicles are mainly responsible for providing ground support services for aircraft, which is an important factor in ensuring the orderly take-off of flights. However, in the airport environment, all vehicles rely on manual operation and manual command throughout the entire process when performing tasks, which inevitably leads to improper human operation, friction between vehicles and aircraft, and damage to aircraft. Therefore, the autonomous obstacle avoidance technology of airport special vehicles is a necessary technology for the safe development of the future aviation industry.

[0003] The obstacle avoidance technology of vehicles is divided into three parts: target detection, target tracking and path planning. Target detection, also known as moving obstacle detection, mainly monitors the moving scene in real time through cameras or radars, and analyzes the captured image data to obtain obstacle information. However, since special vehicles require high precision, high safety and high operating costs when operating in airport environments, it is not advisable to continue to use manual operation and manual command methods. At present, common on-board sensors include millimeter wave radar, lidar, ultrasonic radar, infrared sensor and camera, and each sensor plays its own advantages. The fusion of camera and lidar based on computer vision is the most common. The data obtained by the camera is a two-dimensional image, which has the advantages of high resolution and low cost. The disadvantage is that it cannot obtain accurate three-dimensional information. The advantage of radar sensors is that they can obtain accurate three-dimensional data information, but the disadvantage is that the data does not have high resolution and the equipment cost is high. The fusion of the advantages of the two sensors makes the task completion result based on both high resolution of the data and accurate three-dimensional scene information. The F-PointNet and SIFRNet models are based on the fusion of decision and feature layers. The main idea of ​​this fusion is to generate a candidate box of an object through one kind of data, and then combine the candidate box with another kind of data to generate the final object detection result. The ContFuse and PointPainting models are based on feature layer fusion and have higher accuracy than the former. The above two types of fusion technologies are based on data layer fusion and have limited accuracy and efficiency. Summary of the invention

[0004] The present invention adopts the idea of ​​task layer fusion, extracts features from images and point cloud data respectively, and finally fuses the features to complete the detection task. In order to meet the actual situation of large-scale airport scenes, the RandLA-Net model is used to process millions of point cloud data. At the same time, in order to improve processing efficiency, ResNet (residual neural network) is used as a 2D feature extraction method, and then the two are used under the CNN fusion model to realize 3D Bounding Box prediction, improve obstacle detection accuracy, and thus achieve special vehicle obstacle avoidance.

[0005] To achieve the above object, the technical solution of the present invention is:

[0006] A machine vision-based obstacle avoidance method for airport special vehicles comprises the following steps:

[0007] S1. Simulate the airport scene and collect data information using cameras and lidar;

[0008] S2. Preprocessing the point cloud data collected by the sensor;

[0009] S3. Use the RandLA-Net model to directly process the training data set and extract features from the point cloud data;

[0010] S4. Use the residual neural network ResNet to extract features from RGB images;

[0011] S5. Fusion of the two types of data, through the combination of these features abstraction, output of the target object's 3D bounding box, detection of obstacle information, to achieve obstacle avoidance.

[0012] Furthermore, the step S1 is specifically as follows:

[0013] S11. Use Unity3D software to create a 120m*120m airport real-time scene map and export the 3D scene in .obj format;

[0014] S12. Import the scene graph into the blensor software, select the Velodyne 64-line laser radar, adjust the scanning distance to 120m, and save the acquired point cloud data as a .pcd file;

[0015] S13. Adjust the monocular camera and the laser radar to the same angle and collect images as the input data required for the experiment;

[0016] S14. Import the .pcd file into CloudCompare software, add labels to the objects in the scene, and save it as a .txt text file as the input data required for the experiment.

[0017] Furthermore, the step S2 is specifically as follows:

[0018] S21. Grid sampling is performed on the scene. The entire scene is evenly divided into several cubes. The points in each cube are sampled. The side length of the grid selected here is 0.04 voxels. A strategy of randomly selecting a point in each grid is adopted. All sampled data are used to train the model. Assuming that the original scene has N points, n points are selected after random sampling.

[0019] S22. The X, Y, and Z three-dimensional coordinates of the n sampling points are translated so that X, Y, and Z are all greater than or equal to 0, and the RGB color parameters are normalized (the value range of the three RGB parameters is 0-255, and one embodiment is R / 255, G / 255, and B / 255);

[0020] S23. A KDTree is established for the downsampled point coordinates in each scene, which replaces the traditional KNN algorithm to find neighboring points and achieve the purpose of using the downsampled points to predict the original points.

[0021] Furthermore, the step S3 is specifically as follows:

[0022] S31. Obtain the K nearest neighboring points of each sampling point in S23 based on the Euclidean distance, where the three-dimensional coordinates of the sampling point are represented by p i , the three-dimensional coordinate set of K neighboring points is expressed as {p i1 …p ik …p iK};

[0023] S32. The three-dimensional coordinates p of the sampling point i , the three-dimensional coordinates p of the neighboring point k ik , relative coordinates (p i -p ik ) and the Euclidean distance ‖p i -p ik ‖ connect to form a relative point position code, and the encoded relative point position coordinates are used Indicates that the calculation formula is:

[0024] S33. The three-dimensional coordinates p of the neighboring point k ik Corresponding features and encoded relative point position coordinates Connect them together to get new point features. In order to avoid the loss of useful information, the output neighboring point feature vector Aggregated together, it is represented as a set of local geometric features of sampling point i The calculation formula is:

[0025] S34. Since the relative positions of the K neighboring points from the center point are different, the attention mechanism is used to automatically learn and aggregate the useful information in the feature set of the neighboring points; g(·) is a shared function composed of a multi-layer perceptron, which learns a separate score for each point, called the attention score The calculation formula is: Where: W is the learnable weight of the shared multilayer perceptron; the final feature of each sampling point i is It is expressed as the weighted sum of all neighborhood point features and attention scores, and the calculation formula is:

[0026] S35. In order to preserve the overall details of the point cloud and fully display the global features, steps S31-S34 are repeated twice to expand the receptive field from K neighboring points to K 2 Compared with directly increasing the K value in the K nearest neighbor search, this is a cheaper and more efficient way to increase the receptive field of each point and promote feature propagation between neighboring points.

[0027] Furthermore, the step S4 is specifically as follows:

[0028] S41. The network input is 224*224*3. After 5 convolutions for feature extraction, the feature map size is 224 / 2. 5 *224 / 2 5 , so the output is 7*7*2048, and then the pooling layer converts it into a feature vector.

[0029] Furthermore, the step S5 is specifically as follows:

[0030] S51. Taking the feature vector obtained in steps S34 and S41 as input, taking the 3D points as dense spatial anchor points, and predicting the spatial offset of each point to the nearby box corner position after passing through several fully connected layers;

[0031] S52. Use the loss function to regress the spatial offset of each point in the point cloud. The loss function L includes the offset loss function between the ground truth box and the predicted box, and the score loss function L score , spatial transformation regularization loss function L stn , the calculation formula of the loss function L is: Where: n represents the number of sampling points, is the offset between the ground truth box corner position and the i-th input point, Contains the predicted offset;

[0032] S53. Score function loss L scoreIt is used to let the network learn the spatial offset from the point to the nearest target box. There are two scoring functions, namely supervised scoring loss and unsupervised scoring loss. The supervised scoring loss trains the network to predict whether a point is in the target box. The offset regression loss of the i-th input point is expressed as The binary classification loss is expressed as Then the supervised scoring loss function L score1 The calculation formula is: Where: n represents the number of sampling points, m i ∈{0, 1}, used to indicate whether the i-th input point is in the target bounding box; the unsupervised scoring loss is to let the network directly understand which points may give the best hypothesis. The purpose of training the network is to generate high confidence in the points that produce good predictions. The formula includes two competing loss terms, among which: the score of the corner prediction error and the confidence c i Proportional, then the unsupervised scoring loss function L score2 The calculation formula is: Where: n represents the number of sampling points, ω is the weight factor, and its value is 0.1;

[0033] S54. Select points with high scores as prediction results to obtain the 3D bounding box of the object.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention applies the fused multi-dimensional feature information to the field of civil aviation safety for the first time, and realizes safe obstacle avoidance of special vehicles at airports by identifying obstacles. First, the airport scene is simulated, and data information is collected using a monocular camera and a Velodyne 64-line laser radar; then the point cloud data is preprocessed, and the RandLA-Net model is used to extract features from the training data set, and the convolutional neural network ResNet is used to extract features from the RGB image; finally, the fusion network is used to take the two types of features as input, and by combining and abstracting these features, the 3D bounding box of the target object is output, obstacle information is detected, and obstacle avoidance is achieved. The advantage of the present invention is that the image data and point cloud data are extracted and then fused. This approach greatly retains the 2D and 3D data information, so the accuracy of target detection can be improved; at the same time, experiments show that the random sampling method used by the RandLA-Net model not only retains the local features of the point cloud data but also reduces the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the present invention.

[0037] Figure 2 Flowchart for feature extraction of point cloud data by RandLA-Net. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the embodiments.

[0039] like Figure 1 and 2 As shown, a machine vision-based obstacle avoidance method for airport special vehicles includes the following steps:

[0040] S1. Simulate the airport scene and collect data information using cameras and lidar;

[0041] S11. Use Unity3D software to create a 120m*120m airport real-time scene map and export the 3D scene in .obj format;

[0042] S12. Import the scene image into the blensor software, select the Velodyne 64-line laser radar, adjust the scanning distance to 120m, and save the acquired point cloud data as a .pcd file;

[0043] S13. Adjust the monocular camera and the laser radar to the same angle and collect images as the input data required for the experiment;

[0044] S14. Import the .pcd file into CloudCompare software, add labels to the objects in the scene, and save it as a .txt text file as the input data required for the experiment.

[0045] S2. Preprocessing the point cloud data collected by the sensor;

[0046] S21. Grid sampling is performed on the scene. The entire scene is evenly divided into several cubes. The points in each cube are sampled. The side length of the grid selected here is 0.04 voxels. A strategy of randomly selecting a point in each grid is adopted. All sampled data are used to train the model. Assuming that the original scene has N points, n points are selected after random sampling.

[0047] S22. The X, Y, and Z three-dimensional coordinates of the n sampling points are translated so that X, Y, and Z are all greater than or equal to 0, and the RGB color parameters are normalized (the value range of the three RGB parameters is 0-255, and as an embodiment, the specific values ​​can be R / 255, G / 255, and B / 255);

[0048] S23. A KDTree is established for the downsampled point coordinates in each scene, which replaces the traditional KNN algorithm to find neighboring points and achieve the purpose of using the downsampled points to predict the original points.

[0049] S3. Use the RandLA-Net model to directly process the training data set and extract features from the point cloud data;

[0050] S31. Obtain the K nearest neighboring points of each sampling point in S23 based on the Euclidean distance, where the three-dimensional coordinates of the sampling point are represented by p i , the three-dimensional coordinate set of K neighboring points is expressed as {p i1 …p ik …p iK};

[0051] S32. The three-dimensional coordinates p of the sampling point i , the three-dimensional coordinates p of the neighboring point k ik , relative coordinates (p i -p ik ) and the Euclidean distance ‖p i -p ik ‖ connect to form a relative point position code, and the encoded relative point position coordinates are used Indicates that the calculation formula is:

[0052] S33. The three-dimensional coordinates p of the neighboring point k ik Corresponding features and encoded relative point position coordinates Connect them together to get new point features. In order to avoid the loss of useful information, the output neighboring point feature vector Aggregated together, it is represented as a set of local geometric features of sampling point i The calculation formula is:

[0053] S34. Since the relative positions of the K neighboring points from the center point are different, the attention mechanism is used to automatically learn and aggregate the useful information in the feature set of the neighboring points; g(·) is a shared function composed of a multi-layer perceptron, which learns a separate score for each point, called the attention score The calculation formula is: Where: W is the learnable weight of the shared multilayer perceptron; the final feature of each sampling point i is It is expressed as the weighted sum of all neighborhood point features and attention scores, and the calculation formula is:

[0054] S35. In order to preserve the overall details of the point cloud and fully display the global features, steps S31-S34 are repeated twice to expand the receptive field from K neighboring points to K 2 Compared with directly increasing the K value in the K nearest neighbor search, this is a cheaper and more efficient way to increase the receptive field of each point and promote feature propagation between neighboring points.

[0055] S4. Use the residual neural network ResNet to extract features from RGB images;

[0056] S41. The network input is 224*224*3. After 5 convolutions for feature extraction, the feature map size is 224 / 2. 5 *224 / 2 5 , so the output is 7*7*2048, and then the pooling layer converts it into a feature vector.

[0057] S5. Fusion of two types of data, by combining and abstracting these features, outputs the 3D bounding box of the target object, measures obstacle information, and achieves obstacle avoidance.

[0058] S51. Taking the feature vector obtained in steps S34 and S41 as input, taking the 3D points as dense spatial anchor points, and predicting the spatial offset of each point to the nearby box corner position after passing through several fully connected layers;

[0059] S52. Use the loss function to regress the spatial offset of each point in the point cloud. The loss function L includes the offset loss function between the ground truth box and the predicted box, and the score loss function L score , spatial transformation regularization loss function L stn , the calculation formula of the loss function L is: Where: n represents the number of sampling points, is the offset between the ground truth box corner position and the i-th input point, Contains the predicted offset;

[0060] S53. Score function loss L score It is used to let the network learn the spatial offset from the point to the nearest target box. There are two scoring functions, namely supervised scoring loss and unsupervised scoring loss. The supervised scoring loss trains the network to predict whether a point is in the target box. The offset regression loss of the i-th input point is expressed as The binary classification loss is expressed as Then the supervised scoring loss function L score1 The calculation formula is: Where: n represents the number of sampling points, m i ∈{0, 1}, used to indicate whether the i-th input point is in the target bounding box; the unsupervised scoring loss is to let the network directly understand which points may give the best hypothesis. The purpose of training the network is to generate high confidence in the points that produce good predictions. The formula includes two competing loss terms, among which: the score of the corner prediction error and the confidence c i Proportional, then the unsupervised scoring loss function L score2 The calculation formula is: Where: n represents the number of sampling points, ω is the weight factor, and its value is 0.1;

[0061] S54. Select points with high scores as prediction results to obtain the 3D bounding box of the object.

[0062] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Without departing from the spirit and principles of the present invention, any modifications, equivalent substitutions, improvements, etc. made to the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A machine vision-based obstacle avoidance method for airport special vehicles, characterized in that: The following steps are involved: S1. Simulate the airport scene and collect data information using cameras and lidar; S2. Preprocessing the point cloud data collected by the sensor; S3. Use the RandLA-Net model to directly process the training data set and extract features from the point cloud data; S4. Use the residual neural network ResNet to extract features from RGB images; S5. Fusion of the two types of data, by combining and abstracting these features, outputting the 3D bounding box of the target object, detecting obstacle information, and achieving obstacle avoidance; The step S5 is specifically as follows: S51. Taking the feature vector obtained in steps S34 and S41 as input, taking the 3D points as dense spatial anchor points, and predicting the spatial offset of each point to the nearby box corner position after passing through several fully connected layers; S52. Use the loss function to regress the spatial offset of each point in the point cloud. The loss function L includes the offset loss function between the ground truth box and the predicted box, and the score loss function L score , spatial transformation regularization loss function L stn , the calculation formula of the loss function L is: Where: n represents the number of sampling points, is the offset between the ground truth box corner position and the i-th input point, Contains the predicted offset; S53. Score function loss L score It is used to let the network learn the spatial offset from the point to the nearest target box. There are two scoring functions, namely supervised scoring loss and unsupervised scoring loss. The supervised scoring loss trains the network to predict whether a point is in the target box. The offset regression loss of the i-th input point is expressed as The binary classification loss is expressed as Then the supervised scoring loss function L score1 The calculation formula is: Where: n represents the number of sampling points, m i ∈{0, 1}, used to indicate whether the i-th input point is in the target bounding box; the unsupervised scoring loss is to let the network directly understand which points may give the best hypothesis. The purpose of training the network is to generate high confidence in the points that produce good predictions. The formula includes two competing loss terms, among which: the score of the corner prediction error and the confidence c i Proportional, then the unsupervised scoring loss function L score2 The calculation formula is: Where: n represents the number of sampling points, ω is the weight factor, and its value is 0.1; S54. Select points with high scores as prediction results to obtain the 3D bounding box of the object.

2. The machine vision-based obstacle avoidance method for airport special vehicles according to claim 1, characterized in that: The step S1 is specifically as follows: S11. Use Unity3D software to create a 120m*120m airport real-time scene map and export the 3D scene in .obj format; S12. Import the scene graph into the blensor software, select the Velodyne 64-line laser radar, adjust the scanning distance to 120m, and save the acquired point cloud data as a .pcd file; S13. Adjust the monocular camera and the laser radar to the same angle and collect images as the input data required for the experiment; S14. Import the .pcd file into CloudCompare software, add labels to the objects in the scene, and save it as a .txt text file as the input data required for the experiment.

3. The machine vision-based obstacle avoidance method for airport special vehicles according to claim 1, characterized in that: The step S2 is specifically as follows: S21. Grid sampling is performed on the scene. The entire scene is evenly divided into several cubes. The points in each cube are sampled. The side length of the grid selected here is 0.04 voxels. A strategy of randomly selecting a point in each grid is adopted. All sampled data are used to train the model. Assuming that the original scene has N points, n points are selected after random sampling. S22. Translate the X, Y, and Z three-dimensional coordinates of the n sampling points so that X, Y, and Z are all greater than or equal to 0, and normalize the RGB color parameters; S23. A KDTree is established for the downsampled point coordinates in each scene, which replaces the traditional KNN algorithm to find neighboring points and achieve the purpose of using the downsampled points to predict the original points.

4. The machine vision-based obstacle avoidance method for airport special vehicles according to claim 1, characterized in that: The step S3 is specifically as follows: S31. Obtain the K nearest neighboring points of each sampling point in S23 based on the Euclidean distance, where the three-dimensional coordinates of the sampling point are represented by p i , the three-dimensional coordinate set of K neighboring points is expressed as {p i1 …p ik …p iK }; S32. The three-dimensional coordinates p of the sampling point i , the three-dimensional coordinates p of the neighboring point k ik , relative coordinates (p i -p ik ) and the Euclidean distance ‖p i -p ik ‖ connect to form a relative point position code, and the encoded relative point position coordinates are used It means that the calculation formula is: S33. The three-dimensional coordinates p of the neighboring point k ik Corresponding features and encoded relative point position coordinates Connect them together to get new point features. In order to avoid the loss of useful information, the output neighboring point feature vector k=1…K are aggregated together to represent the local geometric feature set of sampling point i The calculation formula is: S34. Since the relative positions of the K neighboring points from the center point are different, the attention mechanism is used to automatically learn and aggregate the useful information in the feature set of the neighboring points; g(·) is a shared function composed of a multi-layer perceptron, which learns a separate score for each point, called the attention score The calculation formula is: Where: W is the learnable weight of the shared multilayer perceptron; the final feature of each sampling point i is It is expressed as the weighted sum of all neighborhood point features and attention scores, and the calculation formula is: S35. In order to preserve the overall details of the point cloud and fully display the global features, steps S31-S34 are repeated twice to expand the receptive field from K neighboring points to K 2 Compared with directly increasing the K value in the K nearest neighbor search, this is a cheaper and more efficient way to increase the receptive field of each point and promote feature propagation between neighboring points.

5. The machine vision-based obstacle avoidance method for airport special vehicles according to claim 1, characterized in that: The step S4 is specifically as follows: S41. The network input is 224*224*3. After 5 convolutions for feature extraction, the feature map size is 224 / 2. 5 *224 / 2 5 , so the output is 7*7*2048, and then the pooling layer converts it into a feature vector.

Images