Method for locating holes on the surface of an aircraft based on two-dimensional images and three-dimensional point clouds

Abstract

The application discloses a kind of based on two-dimensional image and three-dimensional point cloud's aircraft surface hole positioning measurement method, belong to aircraft defect detection technical field.Firstly, two-dimensional image and three-dimensional point cloud data of the surface of aircraft to be overhauled are collected using binocular structured light camera;For two-dimensional image, YOLOX target detection algorithm is used to locate hole defect;By camera calibration, stereo matching, space three-dimensional coordinate solution, hole defect two-dimensional coordinate frame is mapped into three-dimensional point cloud, and target point cloud is intercepted;Then, down-sampling is realized using voxelization grid method, and the number of point clouds is controlled;Based on RANSAC algorithm, hole fitting and measurement of three-dimensional point cloud are realized, and spatial coordinates, area and depth information of hole defect are obtained.The application combines the characteristics of two-dimensional image and three-dimensional point cloud, realizes the positioning and accurate measurement of hole defect under the complex environment of aircraft surface, improves the robustness of algorithm, balances the detection speed and detection accuracy, and provides hole defect spatial position, area and depth information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aircraft defect detection, and particularly relates to an aircraft surface hole positioning and measuring method based on two-dimensional images and three-dimensional point clouds. BACKGROUND

[0002] In order to ensure the safe operation of the aircraft during the service period, the aircraft needs to be overhauled before each take-off and after landing. During the flight process, the aircraft bears the constantly changing atmospheric pressure under different natural environments, the impact of external objects and the collision of tiny particles in the air, and the working environment is very harsh. Long-term in this state will cause many tiny damages on the surface of the aircraft and then evolve into hole defects, which poses a serious threat to the flight safety of the aircraft. At present, the detection of aircraft surface defects mainly relies on experienced manual visual detection, which has a large working strength and low detection efficiency. Therefore, it is urgent to position and measure the aircraft surface hole defects efficiently and accurately, and to improve the detection level of the aircraft surface defects.

[0003] With the rapid development of science and technology, computer vision technology has been recognized as a key component for improving inspection and monitoring problems in many research fields. Images are a major way for computer vision technology to analyze data, and the visual information contained therein is similar to the information obtained by human eyesight. Due to this similarity, the computer can implement visual inspection similar to the eyes of an inspector. In the problem of aircraft surface defect detection, thanks to the rapid development of image sensors in recent decades, the detection problem of two-dimensional defects has achieved fruitful research results. However, due to the influence of factors such as light source and surface material on the imaging quality of the image, the defect detection result becomes poor, and only the brightness information of the target object on the surface can be obtained from the image data, and usually only the two-dimensional contour information of the hole defect can be obtained, and the depth information cannot be obtained, so that the two-dimensional image cannot effectively express the three-dimensional information of the defect. In recent years, the distance sensors such as binocular structured light cameras and laser radars have made the collection of three-dimensional point cloud data of the object surface become more and more simple, and the research work on point cloud data has also become a research hotspot in many fields. Point cloud can present the three-dimensional structure of the part and obtain a large amount of information that two-dimensional images cannot obtain, and the noise such as light change and color distortion has little effect on the point cloud structure, so the robustness of the detection algorithm can be enhanced. However, the amount of original three-dimensional point cloud data is very large, and if the detection algorithm is directly based on the original point cloud, the detection time may be too long.

[0004] From the appeal background, it can be analyzed that the positioning and measurement of the holes defects on the surface of the aircraft need to solve the following key problems: (1) The algorithm model must have intelligence, high robustness and generalization. The algorithm must be able to detect different positions on the surface of the aircraft, so as to realize intelligent detection instead of manual detection and ensure the flight safety of the aircraft. (2) The algorithm model must have high efficiency. The aircraft maintenance has a short air period, so the algorithm must be able to quickly complete the positioning and measurement of the hole defects on the surface of the aircraft in the service air period of the aircraft, and ensure the punctuality of the flight. (3) The algorithm model must have high precision. The algorithm must be able to accurately measure the hole position, area and depth, provide accurate defect information for the aircraft maintenance personnel, and improve the quality inspection quality. SUMMARY

[0005] In view of the problems of low detection efficiency and poor accuracy of the existing aircraft surface defect detection method, the present application provides an aircraft surface hole positioning and measurement method based on two-dimensional image and three-dimensional point cloud.

[0006] The aircraft surface hole positioning and measurement method based on two-dimensional image and three-dimensional point cloud provided by the present application has the following steps:

[0007] S1, a binocular structured light camera is used to collect 2D images and 3D point cloud data of the aircraft to be maintained.

[0008] S2, a YOLOX target detection network based on deep learning is used to position the hole defects of the 2D image.

[0009] S21, a trained YOLOX target detection model is loaded to perform target detection processing on the 2D image.

[0010]

[0011] S22, if no hole defect is detected, the position is moved, and the target detection of the next point is repeated.

[0012] S23, if a hole defect is detected, the image frame coordinates are obtained.

[0013] S3, camera calibration and three-dimensional calculation are performed to realize data registration from two-dimensional image to three-dimensional point cloud; including the following sub-steps:

[0014] S31, camera calibration: the basis for restoring the three-dimensional coordinates of the object is to obtain the position and attitude relationship between the two cameras. The Zhang Zhengyou plane calibration algorithm is used, and the radial distortion and tangential distortion are considered when solving the camera parameters. During calibration, the images shot by the two cameras are obtained multiple times, then single target calibration is performed to obtain the internal and external parameters of the two cameras, and then double target calibration is performed with these parameters as initial values.

[0015] S32. Stereo matching acquires spatial location data based on the relationship between parallaxes and matches all pixels in the image using phase matching. It uses the epipolar slope of each pixel as the matching criterion, combined with the condition of equal phase, to complete the matching process for pixels across the entire image.

[0016] S33. Solving and measuring three-dimensional spatial coordinates, expressing the focal lengths of the left and right cameras as follows: and Using the left camera coordinate system as the world coordinate system, let the projection coordinates of any point P in space in the world coordinate system (left camera coordinate system) be (X, Y, Z), and its coordinates in the left camera image plane coordinate system be expressed as: The image of the right camera is represented in the plane coordinate system as follows: The three-dimensional coordinates of point P in the world coordinate system are calculated as follows:

[0017]

[0018] In the formula, t y and t z This represents two parameters in the camera translation matrix T; k 31 、k 32 、k 33 、k 21 、k 22 、k 23 This represents the parameters in the camera rotation matrix R.

[0019] S4. 3D point cloud data preprocessing reduces the number of points without damaging the geometry of the point cloud itself.

[0020] The methods for preprocessing 3D point cloud data are as follows:

[0021] S41. By mapping the 2D coordinate frame to the 3D point cloud data through step S3, the 3D point cloud data is extracted, thereby filtering out irrelevant point cloud data and accelerating the model speed.

[0022] S42. Based on the point cloud data coordinate set, obtain... Maximum value on the three coordinate axes and minimum value ;

[0023] S43. Set the side length of the voxel grid to b. ;

[0024] S44、According to The maximum and minimum values of the three coordinate axes are used to obtain the minimum bounding box side length of the point cloud

[0025]

[0026] S45、Calculate the size of the voxel grid:

[0027]

[0028] S46、Calculate the index of each point in the point cloud in the small voxel grid ;

[0029]

[0030] S47、Sort the elements in from small to large, calculate the center of gravity of each small voxel grid, and replace all points in the small grid with the center of gravity.

[0031] S5、Based on the RANSAC algorithm, the hole fitting and measurement of the three-dimensional point cloud are obtained. The area of the hole defect on the surface of the aircraft , position information and depth information measurement.

[0032] 2. The aircraft surface hole positioning and measurement method based on two-dimensional image and three-dimensional point cloud, characterized in that step S5 includes the following sub-steps:

[0033] S51、Randomly select a subset from the original data set , as the assumed inlier, estimate the hole model parameters according to the subset ;

[0034] S52、Traverse all data in the data set except the subset , if the data point is within a given error , it is marked as an inlier, otherwise it is marked as an outlier;

[0035] S53、All inliers form a consensus set, if the number of points in the consensus set meets a given threshold , then re-estimate the hole model parameters using all inliers in the consensus set, and then end the algorithm;

[0036] S54、If the number of inliers in the consensus set is less than the threshold , then select a new subset and repeat S51-S53;

[0037] S55, after the secondary iteration, selecting a consistent set with the most inner points, re-estimating the hole model parameters with all the inner points in the consistent set, and then ending the algorithm;

[0038] S56, calculating the hole center coordinates (x, y, z) x m ，y m ，z m ) and hole radius r according to the estimated hole model parameters , calculating the area of the aircraft surface hole defect , position information and depth information .

[0039] 3. The aircraft surface hole positioning and measuring method based on two-dimensional images and three-dimensional point clouds, characterized in that in step S56, the calculation formula of the area of the aircraft surface hole defect , position information is as follows:

[0040]

[0041]

[0042] r represents the hole radius; x m ，y m ，z m x and y represent the hole center coordinates.

[0043] Depth information is calculated according to the point cloud coordinates of two vertices on one side of the three-dimensional cuboid obtained by projection:

[0044] .

[0045] Compared with the prior art, the present application has the advantages of:

[0046] ​(1) The application adopts a YOLOX target detection network based on deep learning, uses a binocular structured light camera to collect two-dimensional images, and performs target cropping, horizontal flipping, exposure adjustment, noise addition and Mosaic data enhancement on the pictures, so that the model has high robustness to images obtained in different environments, after YOLOX locates the hole defect position on the two-dimensional image, the coordinates of the target position upper left, lower left, upper right and lower right are mapped to three-dimensional point cloud, and the positioning point cloud data is intercepted. Compared with directly using a deep learning-based target positioning algorithm to train and detect three-dimensional data, the detection efficiency of the algorithm is greatly improved.

[0047] (2) The target positioning method based on two-dimensional coordinate mapping three-dimensional point cloud of the application positions the two-dimensional target detection frame in the three-dimensional point cloud through camera calibration, stereo matching and spatial three-dimensional coordinate solving, filters out a large amount of point cloud data irrelevant to the target component, and improves the algorithm model efficiency.

[0048] (3) In the method of the application, the voxelization grid method of three-dimensional point cloud realizes down-sampling, a three-dimensional voxel grid is created through input point cloud data, a hash function is used to approximate other points in the voxel with the center of gravity of all points in each voxel, so that all points in the voxel are finally represented by a center of gravity point, and after processing all voxels, the filtered point cloud is obtained, the number of points is reduced without destroying the geometric structure of the point cloud, and the algorithm speed in the shape fitting recognition process is improved.

[0049] (4) In order to obtain the position, area and depth information of the hole defect, the application adopts RANSAC to fit the cylinder formed by the hole defect, the method of RANSAC fitting cylinder can effectively eliminate noise and outliers, and can also eliminate the influence caused by the collection angle, so that the algorithm has high robustness.

[0050] Other advantages, objects and features of the application will be partly embodied by the following description, and partly understood by those skilled in the art through research and practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The flowchart of the aircraft surface hole positioning and measuring method based on two-dimensional images and three-dimensional point clouds of the application.

[0052] Figure 2 The schematic diagram of two-dimensional image hole positioning.

[0053] Figure 3 YOLOX target detection network structure.

[0054] Figure 4 Binocular camera model.

[0055] Figure 5 For binocular vision calibration process.

[0056] Figure 6 For target three-dimensional point cloud registration schematic diagram.

[0057] Figure 7 For RANSAC algorithm-based three-dimensional point cloud hole fitting result schematic diagram.

[0058] Figure 8 For the principle diagram of measuring the spatial position, area and depth information of the hole defect. DETAILED DESCRIPTION

[0059] The preferred embodiments of the present application will be described below with reference to the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0060] As Figure 1 shown, the two-dimensional image and three-dimensional point cloud-based aircraft surface hole positioning and measuring method of the present application comprises the following steps:

[0061] Step 1: Use a binocular structured light camera to collect aircraft surface hole defect data. That is, configure a defect visual detection device at an aircraft maintenance point, and a maintenance personnel collects 2D images and 3D point cloud data of the aircraft surface to be repaired by using a binocular structured light camera carried on the device.

[0062] When collecting image samples, a random shooting method is used to make the shooting conditions of the samples as variable as possible, to ensure the diversity of the samples. A binocular structured light camera is configured at the aircraft maintenance point to obtain high-precision point cloud data and high-quality image information. The maintenance personnel collects aircraft surface hole defects according to the order and distance of the machine inspection, and collects the picture samples in a local-to-global manner to ensure that there is a considerable amount of samples under coarse measurement angle and fine measurement angle. For the same damage site, a multiple shooting method is used, and the shooting conditions should not be limited, but the pictures taken each time should not be repeated and should be significantly different.

[0063] Step 2: Hole defect positioning based on deep learning target detection network YOLOX.

[0064] Since the number of collected point clouds is in the order of millions, directly training and detecting the original three-dimensional point cloud data based on deep learning will consume a large amount of unnecessary computing power and time. Therefore, the two-dimensional image of the hole defect can be trained and detected based on the deep learning target detection network, and then the three-dimensional point cloud data can be located through coordinate mapping. The two-dimensional image hole defect positioning uses the YOLOX target detection network, and the detection visualization result is as shown in Figure 2 .

[0065] The YOLOX algorithm belongs to a single-stage target detection algorithm, which integrates target classification and target positioning in the same network for prediction to realize "end-to-end" detection. The network structure of the YOLOX algorithm is as shown in Figure 3 The CSPDarknet is a backbone feature extraction network of the YOLOX, and the input picture will be first subjected to feature extraction in the CSPDarknet. The extracted feature layer is a feature set of the input picture. In the backbone part, three feature layers are obtained for the construction of the next network, and the three feature layers are effective feature layers. The FPN is a reinforced feature extraction network of the YOLOX, and the three effective feature layers obtained in the backbone part will be subjected to feature fusion in this part. The purpose of the feature fusion is to combine feature information of different scales. In the FPN part, the effective feature layers obtained are used to continue to extract features. In the YOLOX, the structure of the PAnet used in the YOLOV4 is also used, which not only performs up-sampling on the features to realize feature fusion, but also performs down-sampling on the features to realize feature fusion again. The YOLO Head is a classifier and regressor of the YOLOX. Through the CSPDarknet and the FPN, three reinforced effective feature layers can be obtained. Each feature layer has a width, a height and a number of channels. At this time, the feature map can be regarded as a set of feature points one by one, and each feature point has a number of features in the channel. The work actually done by the YOLO Head is to judge whether the feature points have objects corresponding to them. The YOLO Head is divided into two parts, which are realized respectively, and are integrated together only when the final prediction is performed.

[0066] Considering that each image in the training set contains about 2-3 targets, most of the prior boxes do not contain targets, which is easy to cause imbalance between positive and negative samples during training. The cross-entropy loss function used in the classification error of the YOLOX is difficult to solve the problem of sample imbalance, and the focal loss has the function of adjusting sample imbalance and difficult-to-learn samples. The present application adopts the following focal loss function as the loss function.

[0067]

[0068]

[0069]

[0070] In the formula, , indicates the probability of sample prediction, is a balance parameter for controlling the influence of positive and negative samples on the total loss, and can adjust the influence of sample imbalance on the loss, is a focal parameter for controlling the focus of difficult samples on the total loss. The focal loss is used to replace the cross-entropy loss function of the original YOLOX model, so that the model pays more attention to learning difficult-to-distinguish samples.

[0071] Step 3: Implement data registration of two-dimensional images to three-dimensional point clouds.

[0072] The binocular vision system is to simulate the human eyes to take the same object in space from different angles by using two cameras, then find the corresponding points in the two images, and then recover the coordinates of the object in three-dimensional space according to the principle of triangulation. The basis for recovering the three-dimensional coordinates of the object is to obtain the position and attitude relationship between the two cameras, i.e. binocular camera calibration is needed. The binocular camera model is as shown in Figure 4 The calibration algorithm uses Zhang Zhengyou's plane calibration algorithm, and considers radial distortion and tangential distortion when solving camera parameters. During calibration, the images taken by the two cameras are obtained multiple times, then single target calibration is performed to obtain the internal and external parameters of the two cameras, and then double target calibration is performed with these parameters as initial values. The specific calibration process is as shown in Figure 5 .

[0073] In the process of three-dimensional data acquisition, the purpose of stereo matching is to obtain matching pixel points in two images corresponding to the same space point. Through the relationship between the parallax, the data information of the spatial position is obtained, and all the pixels in the image are matched according to the phase matching method. By taking the polar line slope of each pixel point in the image as the matching basis, and combining with the phase equal judgment condition, the matching process of the pixel points in the whole image range is completed. This method has good performance in noise resistance level, matching speed and accuracy.

[0074] Through camera calibration, the focal length of the camera and the rotation matrix and translation vector between the left and right cameras are obtained, and through stereo matching algorithm, the image coordinates of the actual space points on the object corresponding to the projection points on the left and right cameras are obtained, and the three-dimensional world coordinates of the space points can be calculated by using the principle of triangulation.

[0075] The method for solving the spatial three-dimensional coordinates is: the focal lengths of the left and right cameras are represented as and . The left camera coordinate system is taken as the world coordinate system, and an arbitrary point in the world coordinate system (left camera coordinate system) is taken as the projection point coordinate in the world coordinate system (left camera coordinate system), which is , and the coordinate in the left camera image plane coordinate system is represented as , and the projection point coordinate in the right camera coordinate system is , and the coordinate in the right camera image plane coordinate system is represented as . Therefore, according to the triangular relationship of perspective transformation, we have:

[0076]

[0077] The rotation matrix of the left camera coordinate system transformed to the right camera coordinate system is:

[0078]

[0079] The translation matrix is:

[0080]

[0081] Then:

[0082]

[0083] Combined with the above formula, the space The three-dimensional coordinates of the point in the world coordinate system are:

[0084]

[0085] After positioning the hole defect position on the two-dimensional image by YOLOX, the coordinates of the target position upper left, lower left, upper right and lower right are mapped to the three-dimensional point cloud, and the positioning point cloud data is intercepted, and the visualization result is as shown in Figure 6 , which greatly improves the detection efficiency of the algorithm.

[0086] Step 4: three-dimensional point cloud data preprocessing

[0087] In the previous steps 2 and 3, hole target detection and two-dimensional image to three-dimensional point cloud data registration are performed, and the corresponding hole defects in the point cloud are labeled by a three-dimensional bounding box. The number of target point clouds extracted is in the order of hundreds of thousands, and a large number of point clouds will reduce the algorithm processing speed. The present application uses a voxel grid method of three-dimensional point cloud to realize down-sampling. A three-dimensional voxel grid is created by inputting point cloud data, and a hash function is used to approximate the other points in the voxel by using the center of gravity of all points in each voxel. In this way, all points in the voxel are finally represented by a center of gravity point. After processing all voxels, the filtered point cloud is obtained, which reduces the number of points without destroying the geometric structure of the point cloud, and improves the algorithm speed in the shape fitting recognition process.

[0088] Step 5: three-dimensional point cloud hole fitting and measurement based on RANSAC algorithm

[0089] The target point cloud obtained by step 4 is processed, and since the shape of the target object is similar to a cylinder, the present application uses the RANSAC (Random Sample Consensus) algorithm to realize the fitting and segmentation of the cylindrical model, which has the characteristics of fewer iterations and strong anti-gross-error ability.

[0090] The RANSAC algorithm first selects a subset of data from the original data set Randomly select a subset , Let be an interior point, and then apply the following equation of the cylinder to the axis of the cylinder. The coordinates of the previous point cylindrical axis Direction vector The calculation is performed using seven parameters, including the cylinder radius r0.

[0091]

[0092] Traversing the dataset Subsets All data points outside the set are marked as interior points if they are within a given error e (e=0.01); otherwise, they are marked as exterior points. All interior points form a consensus set. If the number of points in the consensus set satisfies a given threshold T (T=50), the cylindrical model parameters are re-estimated using all interior points in the consensus set, and then the algorithm terminates.

[0093] If the number of interior points in the uniform set is less than the threshold Then a new subset is selected. Repeat the aforementioned steps, after In the next iteration, a consensus set with the largest number of interior points is selected. The hole model parameters are re-estimated using all interior points in the consensus set, and then the algorithm terminates. The RANSAC algorithm-based 3D point cloud hole fitting results are as follows: Figure 7 As shown.

[0094] The radius of the hole is obtained from the estimated hole model parameters. and the coordinates of the cylinder center ( x m ，y m ，z m Complete the area of ​​holes and defects on the aircraft surface. and spatial location Measurement, depth information The measurement can be directly obtained from the two vertices on one side of the three-dimensional cuboid obtained by projection. The point cloud coordinates are calculated. The calculation principle is as follows: Figure 8 As shown.

[0095]

[0096] In summary, the aircraft surface hole positioning and measuring method based on the combination of two-dimensional images and three-dimensional point clouds of the application uses a binocular structured light camera to collect two-dimensional images and three-dimensional point cloud data of the surface of the aircraft to be overhauled; for the two-dimensional images, a YOLOX target detection algorithm is used to locate the hole defects; the two-dimensional coordinate frame of the hole defects is mapped into the three-dimensional point cloud through camera calibration, stereo matching and spatial three-dimensional coordinate solving, and the target point cloud is intercepted; then the voxel grid method is used to realize down-sampling and control the number of point clouds; the RANSAC algorithm is used to realize the hole fitting and measurement of the three-dimensional point cloud, and the spatial coordinates, area and depth information of the hole defects are obtained. The application combines the characteristics of two-dimensional images and three-dimensional point clouds, and realizes the positioning and accurate measurement of hole defects under the complex environment of the surface of the aircraft based on the deep learning target detection algorithm and the traditional point cloud method, improves the robustness of the algorithm, balances the detection speed and detection accuracy, provides the full spatial position, area and depth information of the hole defects, and provides an important reference for the maintenance work of the maintenance personnel.

[0097] The above is only a preferred embodiment of the application, and does not limit the application in any form. Although the application has been disclosed as above, it is not intended to limit the application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the application, and any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the application are still within the scope of the technical solution of the application.

Claims

1. A method for locating and measuring holes on aircraft surfaces based on two-dimensional images and three-dimensional point clouds, characterized in that, The steps are as follows: S1. Use a binocular structured light camera to acquire 2D images and 3D point cloud data of the surface of the aircraft to be inspected; S2. The YOLOX object detection network based on deep learning is used to locate holes and defects in 2D images; S3, camera calibration and 3D calculation, to achieve data registration from 2D images to 3D point clouds; Includes the following sub-steps: S31, Camera calibration; S32. Stereo matching: Spatial location data is obtained through the relationship between parallaxes, and all pixels in the image are matched according to the phase matching method. S33. Solving and measuring three-dimensional spatial coordinates, expressing the focal lengths of the left and right cameras as follows: and Using the left camera coordinate system as the world coordinate system, let the coordinates of any point P in space projected onto the world coordinate system be (X, Y, Z), and its coordinates in the left camera image plane coordinate system be expressed as follows: The image of the right camera is represented in the plane coordinate system as follows: The three-dimensional coordinates of point P in the world coordinate system are calculated as follows: In the formula, t y and t z This represents two parameters in the camera translation matrix T; k 31 、k 32 、k 33 、k 21 、k 22 、k 23 These represent the parameters in the camera rotation matrix R; S4. 3D point cloud data preprocessing reduces the number of points without damaging the geometry of the point cloud itself; S5. Based on the RANSAC algorithm, fit and measure the 3D point cloud hole area to obtain the area of ​​hole defects on the aircraft surface. Location information and depth information Measurement.

2. The method for locating and measuring holes on aircraft surfaces based on two-dimensional images and three-dimensional point clouds as described in claim 1, characterized in that, Step S5 includes the following sub-steps: S51, From the original dataset Randomly select a subset , For the assumed interior points, according to the subset Estimate the parameters of the hole model; S52, Traversing the Dataset Subsets All data outside of the given error, if the data points are within the given error range. If it is inside, mark it as an inside point; otherwise, mark it as an outside point. S53. All interior points form a consistent set. If the number of points in the consistent set satisfies a given threshold... If the parameters of the hole model are re-estimated using all interior points in the uniform set, then the algorithm ends. S54. If the number of interior points in the uniform set is less than the threshold. Then a new subset is selected. And repeat S51-S53; S55, Passing In the next iteration, select a consensus set with the largest number of interior points, re-estimate the hole model parameters using all interior points in the consensus set, and then terminate the algorithm. S56. The center coordinates of the hole obtained from the estimated hole model parameters ( x m ，y m ，z m ) and hole radius Calculate the area of ​​holes and defects on the aircraft surface. Location information and depth information .

3. The method for locating and measuring holes on aircraft surfaces based on two-dimensional images and three-dimensional point clouds as described in claim 2, characterized in that, In step S56, the area of ​​the hole defect on the aircraft surface is... Location information The calculation formula is as follows: Indicates the radius of the hole; x m ，y m ，z m Indicates the coordinates of the center of the hole; Depth Information The two vertices on one side of the three-dimensional cuboid obtained by projection The point cloud coordinates were calculated as follows: 。 4. The method for locating and measuring holes on aircraft surfaces based on two-dimensional images and three-dimensional point clouds as described in claim 1, characterized in that, Step S2 includes the following sub-steps: S21. Load the trained YOLOX object detection model and perform object detection processing on the 2D image; S22. If no hole defect is detected, move the position and repeat S21 to perform target detection at the next point. S23. If a hole defect is detected, obtain the coordinates of the image border.

5. The method for locating and measuring aircraft surface holes based on two-dimensional images and three-dimensional point clouds as described in claim 4, characterized in that, In step S21, the Focal loss function is used as the loss function in the classification error of the YOLOX object detection model.

6. The method for locating and measuring holes on aircraft surfaces based on two-dimensional images and three-dimensional point clouds as described in claim 1, characterized in that, In step S32, the matching process of pixels in the entire image range is completed by using the epipolar slope of each pixel in the image as the matching basis and combining it with the judgment condition of equal phase.

7. The method for locating and measuring holes on aircraft surfaces based on two-dimensional images and three-dimensional point clouds as described in claim 1, characterized in that, Step S4, the method for preprocessing 3D point cloud data is as follows: S41. By mapping the 2D coordinate frame to the 3D point cloud data through step S3, the 3D point cloud data is extracted, thereby filtering out irrelevant point cloud data and accelerating the model speed. S42. Based on the point cloud data coordinate set, obtain... Maximum value on the three coordinate axes and minimum value ; S43. Set the side length of the voxel grid to b; S44, according to The minimum bounding box side length of the point cloud is obtained by finding the maximum and minimum values ​​of the three coordinate axes. ; S45. Calculate the dimensions of the voxel mesh; S46. Calculate the index of each voxel grid in the point cloud. ; S47, will The elements are sorted in ascending order, the centroid of each voxel grid is calculated, and all points within the grid are replaced by the centroid.

Images