Intelligent aircraft surface crack detection and measurement method combining image and point cloud

Abstract

The invention discloses an aircraft surface crack intelligent detection and measurement method combining an image and a point cloud. The method comprises the following steps: carrying out two-dimensional image and three-dimensional point cloud data acquisition on an aircraft surface; performing crack defect identification on the acquired two-dimensional image based on a target detection model; aligning the two-dimensional image with the three-dimensional point cloud, and acquiring corresponding crack defect three-dimensional point cloud data according to the crack defect position identified from the two-dimensional image; and performing filtering and down-sampling processing on the obtained three-dimensional point cloud data of the crack defects, and performing crack defect size measurement on the processed three-dimensional point cloud data of the crack defects to obtain length, width and depth information of the crack defects. An omnibearing defect detection system is constructed by fusing two-dimensional image processing and three-dimensional point cloud analysis technologies; the texture features of the two-dimensional image and the depth information of the three-dimensional point cloud are organically combined, so that the detection capability of the system is improved, and accurate identification and size measurement of the aircraft surface crack defects are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft surface crack detection, and in particular to an intelligent detection and measurement method for aircraft surface cracks combining images and point clouds. Background Art

[0002] The performance and reliability of large-scale equipment are crucial to a nation's competitiveness, and the manufacturing and maintenance quality of critical equipment like aircraft is particularly crucial. During their service, aircraft are often subjected to external environmental influences such as small object impacts and atmospheric pressure, which can cause surface cracks. These defects not only shorten the aircraft's service life but also threaten its structural integrity, potentially leading to serious safety incidents. Therefore, timely and accurate detection, location, and measurement of aircraft surface defects are crucial.

[0003] Traditional aircraft surface defect detection relies primarily on manual visual inspection, which is not only time-consuming and labor-intensive, but also prone to missed and false detections due to fatigue, making it difficult to guarantee efficient and accurate inspections. With the development of computer vision technology, image-based automated inspection methods have gained widespread application. Two-dimensional images can intuitively reflect the surface characteristics of defects. Combined with advanced algorithms such as deep learning, they can rapidly locate and measure defects. However, 2D image inspection methods have significant limitations: they cannot obtain defect depth information and are susceptible to lighting conditions, compromising the reliability of inspection results. The development of 3D point cloud technology has provided a new solution for defect detection. Point cloud data acquired using devices such as binocular cameras and structured light can fully characterize the 3D geometric features of defects, enabling not only defect depth measurement but also strong environmental adaptability. However, the sheer volume of point cloud data makes direct processing a significant computational burden, reducing inspection efficiency. Therefore, combining 2D image and 3D point cloud technologies to leverage their respective strengths for rapid and accurate defect detection and comprehensive dimensional measurement has become a key research direction.

[0004] While considerable research has been conducted on equipment surface defect detection, methods that effectively combine image recognition with point cloud processing remain insufficient. In particular, for intelligent detection of subtle defects such as aircraft surface cracks, key issues remain to be further explored, including precise registration of 2D images and 3D point cloud data, extraction of effective feature information, and guaranteed measurement accuracy. Therefore, developing an intelligent detection and measurement method for aircraft surface cracks based on image and point cloud fusion has significant theoretical and practical implications.

[0005] From the above background, we can analyze that the following key points need to be addressed in the positioning and measurement of surface defects on large equipment: (1) The detection system must have excellent versatility and adaptability, and be able to accurately identify and locate cracks on the surface of the equipment, regardless of their distribution location and size. This intelligent and robust detection capability can truly realize automated detection and replace manual inspection, thereby ensuring the safe use of large equipment. (2) Detection efficiency is an important indicator of the system. In order to ensure that the normal operation cycle of the equipment is not affected, the detection system needs to complete the identification, positioning and size measurement of defects within the specified time limit and have rapid processing capabilities. (3) Measurement accuracy is a key requirement of the system. The detection system must be able to accurately quantify the spatial position and geometric dimensions (including length, width, depth, etc.) of various defects, provide maintenance personnel with reliable defect data support, and facilitate the subsequent maintenance work and quality assurance. Summary of the Invention

[0006] The present invention provides an intelligent detection and measurement method for aircraft surface cracks that combines images and point clouds to solve the problems of high computational complexity, low detection efficiency, and substandard detection accuracy in existing aircraft surface crack defect detection methods.

[0007] According to a first aspect, an embodiment provides a method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds, the method comprising: Collect 2D images and 3D point cloud data of the aircraft surface; Crack defect recognition is performed on the collected two-dimensional images based on the trained target detection model; Align the two-dimensional image and the three-dimensional point cloud, and obtain the corresponding three-dimensional point cloud data of the crack defect according to the crack defect position identified from the two-dimensional image; The obtained three-dimensional point cloud data of crack defects are filtered and down-sampled, and the crack defect size is measured on the processed three-dimensional point cloud data of crack defects to obtain the length, width and depth information of the crack defects.

[0008] Furthermore, crack defect recognition is performed on the collected two-dimensional images based on the trained target detection model, specifically including: The collected two-dimensional images are preprocessed, and training and test sets are constructed to train and test the YOLOv8 target detection model.

[0009] Furthermore, the collected two-dimensional images are preprocessed, and training and test sets are constructed to train and test the YOLOv8 object detection model, including: The Mixup data augmentation method is used to expand the image dataset, including: randomly selecting samples and corresponding labels in the image dataset to generate mixed new images and labels:

[0010] in, are the coordinates of the generated mixed new image, , is a random coordinate point in the data set The coordinates of is the random sampling coefficient; Use the local mean-based image reduction algorithm to uniformly scale the two-dimensional image to the size required for target detection model training, including: adjusting the original image matrix to the target size by the scaling factor. If the original image matrix is , the scaling factor is , then the reduced image matrix is ; Normalize the pixel values ​​of the image to between 0 and 1, including: for each pixel of the image matrix , set the maximum pixel value and pixel minimum , normalized pixel value for:

[0011] The processed image dataset is divided into training set and test set, the training parameters are configured and the target detection model is trained. The obtained target detection model is used to identify crack defects on the aircraft surface.

[0012] Furthermore, the two-dimensional image and the three-dimensional point cloud are aligned, and corresponding three-dimensional point cloud data of the crack defect is obtained according to the crack defect position identified from the two-dimensional image, specifically including: Establishing a spatial mapping relationship between the two-dimensional image and the three-dimensional point cloud, including: projecting each three-dimensional point in the three-dimensional point cloud data onto the two-dimensional image plane through camera calibration parameters; For each point in the point cloud, calculate the corresponding pixel coordinates on the two-dimensional image plane, including: according to the point cloud data row number Inverse the coordinates of the corresponding pixel points:

[0013] in, is the point cloud data row number, is the coordinate of the corresponding pixel point on the two-dimensional image plane; Determine pixel coordinates Is it in the crack defect rectangle box identified by the target detection model? The rectangle box is composed of the upper left pixel coordinate point. and the lower right pixel coordinate point Composition, of which Indicates yes The horizontal coordinate of the point, Indicates yes The vertical coordinate of the point, Indicates yes The horizontal coordinate of the point, Indicates yes The vertical coordinate of the point; If the pixel coordinates corresponding to the points in the point cloud Meet the conditions , then the points in the point cloud that meet the conditions are marked as crack defect point cloud.

[0014] Furthermore, the obtained three-dimensional point cloud data of crack defects is filtered and down-sampled, specifically including: Adopt the normal-based point cloud bilateral filtering method, including: Calculate the normal of each point in the point cloud and build a spatial index structure for fast neighborhood search; Perform neighborhood search for each point in the point cloud based on spatial distance and spatial standard deviation parameters Calculating spatial weights , based on the normal angle and the normal standard deviation parameter Calculating normal weights , perform weighted averaging based on the weights and update the position of the points:

[0015] Based on the current processing point , points in the neighborhood ,point Normal ,point Normal , spatial weight function and the normal weight function , update the normal and output the filtered point cloud : .

[0016] Furthermore, the obtained three-dimensional point cloud data of crack defects is filtered and down-sampled, specifically including: Perform voxel downsampling on point cloud data, including: Analyze the spatial distribution characteristics of point cloud data and determine the The maximum and minimum values ​​of the three coordinate axes are used to calculate the minimum bounding box size of the point cloud; Determine the side length and size of the voxel grid, and perform voxel grid division on the minimum bounding box of the point cloud; For each non-empty voxel grid, calculate the centroid coordinates of all points inside it and replace all points in the small grid with the centroid.

[0017] Furthermore, the processed three-dimensional point cloud data of the crack defect is used to measure the size of the crack defect to obtain the length, width and depth information of the crack defect, specifically including: For length measurement, the crack is analyzed at For the distribution in the axial direction, the leftmost and rightmost voxel points are selected and the distance between the two points is calculated to obtain the crack length; For width measurement, by analyzing For the distribution in the axial direction, the top and bottom voxel points are selected and the distance between the two points is calculated to obtain the crack width; For depth measurement, the crack defect point cloud is first fitted to obtain a reference plane, and the normal vector of the reference plane is calculated. Then all voxel points are projected in the direction of the normal vector, and the actual depth of the crack is obtained by calculating the difference between the maximum projection distance and the minimum projection distance.

[0018] According to a second aspect, an embodiment provides an intelligent detection and measurement system for aircraft surface cracks that combines images and point clouds, the system comprising: Data acquisition module, used to collect two-dimensional images and three-dimensional point cloud data of the aircraft surface; The target detection module is used to identify crack defects in the collected two-dimensional images based on the trained target detection model; A crack defect point cloud determination module is used to align the two-dimensional image and the three-dimensional point cloud, and obtain the corresponding three-dimensional point cloud data of the crack defect according to the crack defect position identified from the two-dimensional image; The crack defect measurement module is used to filter and downsample the obtained three-dimensional point cloud data of crack defects, and measure the crack defect size on the processed three-dimensional point cloud data to obtain the length, width and depth information of the crack defects.

[0019] The present invention provides an intelligent detection and measurement method for aircraft surface cracks that combines images and point clouds, comprising: collecting two-dimensional images and three-dimensional point cloud data of the aircraft surface; identifying crack defects in the collected two-dimensional images based on a trained target detection model; aligning the two-dimensional images and the three-dimensional point cloud, and obtaining corresponding three-dimensional point cloud data of the crack defects based on the crack defect positions identified in the two-dimensional images; filtering and downsampling the obtained three-dimensional point cloud data of the crack defects, and measuring the crack defect size on the processed three-dimensional point cloud data of the crack defects to obtain the length, width, and depth information of the crack defects. The method of the present invention constructs a comprehensive defect detection system by integrating two-dimensional image processing and three-dimensional point cloud analysis technology; by organically combining the texture features of the two-dimensional image and the depth information of the three-dimensional point cloud, it not only improves the detection capability of the system, but also realizes the accurate identification and size measurement of crack defects on the aircraft surface; this solution significantly improves the detection efficiency while ensuring detection accuracy, provides reliable technical support for aircraft quality control and maintenance, and effectively solves various technical problems existing in traditional detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds provided by one embodiment of the present invention; Figure 2 A diagram showing the results of locating crack defects in a two-dimensional image in an intelligent detection and measurement method for aircraft surface cracks that combines images and point clouds, provided by one embodiment of the present invention; Figure 3 A result diagram of mapping a two-dimensional image to a three-dimensional point cloud in an intelligent detection and measurement method for aircraft surface cracks combining images and point clouds provided by one embodiment of the present invention; Figure 4 A schematic diagram of the logical structure of an intelligent detection and measurement system for aircraft surface cracks combining images and point clouds, provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The application will be described in further detail below with specific reference being made to the drawings. Like elements in different embodiments are denoted by like reference numerals. In the following description, numerous specific details are described to provide a thorough understanding of the application. However, it will be apparent to one skilled in the art that the application can be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order not to unnecessarily obscure the application. In addition, the features described in the specification, operations or steps can be combined in any suitable manner in various embodiments. Also, the various steps of the methods described can be performed in any suitable order, as would be apparent to one skilled in the art. Therefore, the order in which the features are described in the specification and the order in which the steps of the methods are described are not an essential order, unless otherwise specified.

[0022] In addition, the features, operations or steps described in the specification can be combined in any suitable manner in various embodiments. Also, the various steps of the methods described can be performed in any suitable order, as would be apparent to one skilled in the art. Therefore, the order in which the features are described in the specification and the order in which the steps of the methods are described are not an essential order, unless otherwise specified.

[0023] The first embodiment of the application provides an intelligent detection and measurement method for aircraft surface cracks combined with images and point clouds, which will be described in detail below. Figure 1

[0024] As shown in FIG. 1, in step S100, two-dimensional image and three-dimensional point cloud data of the aircraft surface are collected. Figure 1 The above steps specifically include that the staff moves to the crack detection position of the aircraft surface, collects two-dimensional image data by using a Basler camera, and collects three-dimensional point cloud data by using a binocular structured light camera.

[0025] S110, two-dimensional image collection: a Balser camera is used to shoot four defects of perforation, scratch, crack and pit on the surface of large equipment under different distances, light intensities and angles. It is ensured that enough defect image data under coarse measurement and fine measurement angles are shot to ensure that the trained model can correctly identify defects under various conditions.

[0026] S120, three-dimensional point cloud collection: a binocular structured light camera is used to shoot PLY format three-dimensional point cloud data of the surface of large equipment under different conditions. The collected point cloud data will correspond to the two-dimensional image data, which will be used for subsequent defect mapping and three-dimensional positioning.

[0027] As shown in FIG. 2, in step S200, the collected two-dimensional image data is preprocessed.

[0028] Figure 1 ​​As shown, in step S200, crack defects are identified on the collected two-dimensional image based on the trained target detection model.

[0029] This embodiment uses the convolutional neural network YOLOV8 to locate and identify crack defects.

[0030] The above steps specifically include: S210 uses the Mixup data augmentation method to expand the image dataset, including: randomly selecting samples and corresponding labels in the image dataset to generate mixed new images and labels:

[0031] in, are the coordinates of the generated mixed new image, , is a random coordinate point in the data set The coordinates of is the random sampling coefficient; S220, using a local mean-based image reduction algorithm to uniformly scale the two-dimensional image to the size required for target detection model training, including: adjusting the original image matrix to the target size by a scaling factor, if the original image matrix is , the scaling factor is , then the reduced image matrix is ; S230, normalizing the pixel values ​​of the image to between 0 and 1, including: for each pixel of the image matrix , set the maximum pixel value and pixel minimum , ,normalized pixel value for:

[0032] S240: Configure training parameters and train a target detection model, and use the obtained target detection model to identify crack defects on the aircraft surface.

[0033] In this embodiment, during the data processing stage, the Mixup image enhancement technology is used to construct an operation with a "convex" property, effectively expanding the training data set and improving the generalization ability of the model. At the same time, the image is scaled and normalized to ensure the stability and efficiency of the model training. The image is finely labeled using visualization tools such as labelimg to generate a TXT file that meets the training requirements of the YOLOv8 model. During the model training stage, the YOLOv8 target detection model is used. This model can accurately identify and locate surface defects of large equipment (aircraft) by introducing the backbone network of the C2F module, the neck layer of feature fusion, and the precise output head structure. Finally, the trained YOLOv8 weight file is used to test the captured defect pictures or videos, quickly obtain the type information and two-dimensional coordinates of the defect, and map this information into a three-dimensional point cloud to achieve three-dimensional visual positioning and identification of defects, providing strong technical support for the quality control and maintenance of large equipment.

[0034] During the training and testing phases, the pre-processed image data is divided into training and test sets according to a certain ratio, and then fed into the YOLOv8 network with configured training parameters. This step is usually performed on a high-performance server to ensure the smooth progress of model training. Once the model training is completed and the trained weight file is obtained, these weights can be used to test the defect images or videos newly taken by the 2D camera. Through these tests, crack defects on the surface of large equipment (aircraft) can be quickly and accurately identified, and the two-dimensional coordinate position of the crack defect in the image can also be obtained, such as Figure 2 shown.

[0035] In actual tests, using an RTX 3060Ti graphics card to process a 4096×3000 resolution image, detection took only 0.02 seconds, fully meeting real-time detection requirements and achieving nearly 100% accuracy. Through innovative image preprocessing methods and Mixup data enhancement technology, the object detection model demonstrated excellent generalization and robustness.

[0036] like Figure 1 As shown, in step S300, the two-dimensional image and the three-dimensional point cloud are aligned, and corresponding three-dimensional point cloud data of the crack defect is obtained according to the crack defect position identified from the two-dimensional image.

[0037] The above steps specifically include: S310, establishing a spatial mapping relationship between the two-dimensional image and the three-dimensional point cloud, including: projecting each three-dimensional point in the three-dimensional point cloud data onto the two-dimensional image plane through camera calibration parameters; S320, for each point in the point cloud, calculate the corresponding pixel coordinates on the two-dimensional image plane, including: according to the row number of the point cloud data Inverse the coordinates of the corresponding pixel points:

[0038] in, is the point cloud data row number, is the coordinate of the corresponding pixel point on the two-dimensional image plane; S330, determine pixel coordinates Is it in the crack defect rectangle box identified by the target detection model? The rectangle box is composed of the upper left pixel coordinate point. and the lower right pixel coordinate point Composition, of which Indicates yes The horizontal coordinate of the point, Indicates yes The vertical coordinate of the point, Indicates yes The horizontal coordinate of the point, Indicates yes The vertical coordinate of the point; if the pixel coordinates corresponding to the point in the point cloud Meet the conditions , then the points in the point cloud that meet the conditions are marked as crack defect point cloud.

[0039] This embodiment uses a high-precision industrial camera to capture a two-dimensional image of the aircraft surface, analyzes the image through a pre-trained YOLOv8 deep learning model, accurately identifies and locates the position of the crack defect, and obtains the bounding box coordinates of the defect area. At the same time, a three-dimensional structured light camera is used to collect point cloud data of the same area to obtain the three-dimensional geometric information of the surface. Secondly, a spatial mapping relationship between the two-dimensional image and the three-dimensional point cloud is established: each three-dimensional point in the three-dimensional point cloud data is projected onto the two-dimensional image plane through the camera calibration parameters. This process needs to consider the intrinsic parameter matrix and extrinsic parameter matrix of the camera to ensure the accuracy of the projection. For each point in the point cloud, its corresponding pixel coordinates on the two-dimensional image plane are calculated. Finally, data matching and screening are performed: the projected point cloud coordinates are compared with the crack defect rectangular frame detected by YOLOv8 to determine whether each point falls within the crack defect area. Specifically, the pixel coordinates of the check point meet the constraints of the rectangular frame: the horizontal coordinate is between the left and right boundaries of the rectangular frame, and the vertical coordinate is between the upper and lower boundaries of the rectangular frame. Points that meet these conditions are marked as crack defect point clouds for subsequent processing and analysis. The alignment results are as follows: Figure 3 shown.

[0040] like Figure 1 As shown, in step S400, the obtained three-dimensional point cloud data of the crack defect is filtered and down-sampled, and the crack defect size is measured on the processed three-dimensional point cloud data of the crack defect to obtain the length, width and depth information of the crack defect.

[0041] The above steps specifically include: S410 uses a normal-based point cloud bilateral filtering method, including: S411, calculate the normal of each point in the point cloud and establish a spatial index structure for fast neighborhood search; S412, perform neighborhood search for each point in the point cloud based on spatial distance and spatial standard deviation parameters Calculating spatial weights , based on the normal angle and the normal standard deviation parameter Calculating normal weights , perform weighted averaging based on the weights and update the position of the points:

[0042] S413, based on the current processing point , points in the neighborhood ,point Normal ,point Normal , spatial weight function and the normal weight function , update the normal and output the filtered point cloud : .

[0043] This embodiment uses a normal-based bilateral filtering algorithm to accurately reduce the noise of point cloud data. First, the normal vector of each point in the point cloud is calculated and a spatial index structure is established to provide support for fast neighborhood search. Then, a bilateral filtering operation is performed on each point: the neighborhood points are searched through the spatial index structure, the spatial weight is calculated based on the spatial distance, the normal weight is calculated based on the normal angle, and the position of the point is updated by weighted average based on these two weights. Finally, based on the updated point position, the normal information of the current point and the neighborhood point, and the corresponding weight function, the normal of the point cloud is updated and the filtered result is output. This method can not only effectively remove noise, but also well maintain key geometric features such as defect edges, laying the foundation for subsequent defect feature extraction and precise measurement.

[0044] S420, performing voxel downsampling processing on the point cloud data, including: S421, analyze the spatial distribution characteristics of point cloud data and determine the point cloud The maximum and minimum values ​​of the three coordinate axes are used to calculate the minimum bounding box size of the point cloud; Specifically, according to the point cloud data coordinate set, we can obtain The maximum value on the three coordinate axes and minimum value ;according to The maximum and minimum values ​​of the three coordinate axes are used to obtain the minimum bounding box side length of the point cloud :

[0045] S422, determining the side length and size of the voxel grid, and performing voxel grid division on the minimum bounding box of the point cloud; Specifically, set the side length of the voxel grid to , the three-dimensional dimensions of the voxel grid are:

[0046] S423, for each non-empty voxel grid, calculate the centroid coordinates of all points inside it, and replace all points in the small grid with the centroid; Specifically, calculate the index of each point cloud in the voxel grid :

[0047] Will Sort the elements in the grid from small to large, calculate the centroid of each voxel grid, and replace all points in the grid with the centroid.

[0048] This embodiment implements voxel downsampling of point cloud data. First, the spatial distribution characteristics of the point cloud data are analyzed and the point cloud is calculated. The maximum and minimum values ​​on the three coordinate axes determine the spatial range of the point cloud. Then, the appropriate voxel grid side length is set, the size of the minimum bounding box is calculated based on the spatial range of the point cloud, and the voxel grid is divided accordingly. For each point in the point cloud, the voxel grid index to which it belongs is calculated, and the point sets within the same voxel are classified. Finally, for each non-empty voxel grid, the centroid coordinates of all points inside it are calculated, and all points within the voxel are replaced by this centroid point, thereby achieving uniform simplification of the point cloud. This downsampling method not only ensures the uniformity of the point cloud distribution, but also significantly reduces the amount of data, improves the efficiency of subsequent processing, and at the same time well maintains the geometric characteristics of the point cloud.

[0049] S430, measuring the crack defect size on the processed three-dimensional point cloud data to obtain information on the length, width, and depth of the crack defect, specifically including: S431, for length measurement, by analyzing the crack in For the distribution in the axial direction, the leftmost and rightmost voxel points are selected and the distance between the two points is calculated to obtain the crack length; Specifically, select coordinates Corresponding The crack length is calculated at the voxel points on the left and right sides; S432, for width measurement, by analyzing For the distribution in the axial direction, the top and bottom voxel points are selected and the distance between the two points is calculated to obtain the crack width; Specifically, select coordinates Corresponding The crack width is calculated at the voxel points on the upper and lower sides; S433: For depth measurement, first perform plane fitting on the crack defect point cloud to obtain a reference plane, and calculate the normal vector of the reference plane. Then, project all voxel points onto the normal vector direction, and calculate the difference between the maximum projection distance and the minimum projection distance to obtain the actual depth of the crack. Specifically, first fit the plane composed of defects, then calculate the normal vector of the defect plane, and finally select the coordinates Corresponding The projection distance of all voxel points on the normal vector is calculated, and the maximum projection distance minus the minimum projection distance is the crack depth.

[0050] This embodiment realizes the accurate size measurement of crack defects. The measurements are carried out in three dimensions: First, by analyzing the crack size in The distribution in the axial direction, select the leftmost and rightmost voxel points, calculate the distance between the two points to get the crack length; secondly, by analyzing The crack width is obtained by selecting the top and bottom voxel points and calculating their distance. Finally, for depth measurement, a more complex processing method is used. The defect point cloud is first fitted to a reference plane, the normal vector of this plane is calculated, and then all voxel points are projected in the direction of the normal vector. The actual crack depth is obtained by calculating the difference between the maximum and minimum projection distances. This 3D measurement method fully utilizes the spatial information of the point cloud data to accurately obtain the complete geometric dimensions of the crack, achieving submillimeter measurement accuracy (error <1.5mm), providing a reliable quantitative basis for defect assessment.

[0051] The embodiment of the present invention proposes an intelligent detection and measurement method for aircraft surface cracks that combines images and point clouds, which has the following advantages: 1. This embodiment of the present invention designs a method for locating and identifying surface defects on large equipment (aircraft) based on YOLOv8. First, a 2D camera is used to capture two-dimensional images of the surface of the large equipment (aircraft). Mixup data augmentation technology is then used to expand the dataset, and image scaling and normalization are performed to further optimize the data for model training. These preprocessing steps are designed to improve the model's generalization capabilities and ensure accurate location and identification under various conditions. Using the weight file generated by YOLOv8 training, the defect type and precise location coordinates can be quickly and efficiently predicted.

[0052] 2. This embodiment of the present invention innovatively proposes an image-guided point cloud defect localization method. This method first utilizes the YOLOv8 network to accurately locate and identify crack defects in a two-dimensional image, obtaining information on the defect type and location. This information is then precisely projected into a three-dimensional point cloud space through a rigorous mathematical mapping relationship. This innovative fusion approach not only accurately locates defect areas within the point cloud but also effectively filters out a large amount of background point cloud data unrelated to the defect, significantly reducing computational complexity and improving the overall processing efficiency of the system.

[0053] 3. The embodiment of the present invention uses a normal-based bilateral filtering algorithm to reduce noise in point cloud data. This method optimizes and updates the point position by calculating the normal information of each point in the point cloud and combining spatial distance weights and normal direction weights. This processing method can effectively remove the interference of noise points while well preserving the geometric characteristics of defect edges. Subsequently, a voxelized downsampling method is used to simplify the point cloud. By calculating the center of gravity of each voxel grid to replace all points within the grid, it not only ensures the uniform distribution of the point cloud but also greatly reduces the amount of data.

[0054] 4. This embodiment of the present invention proposes a precise crack size measurement method. By analyzing the distribution characteristics of the defect point cloud along different coordinate axes and combining it with the voxelized center of gravity information, it achieves accurate measurement of crack length and width. For depth measurement, the plane containing the defect point cloud is first fitted and the normal vector is calculated. The crack depth is then determined by the difference in the projected distance from the point to the plane, ensuring the accuracy and reliability of the measurement results.

[0055] Corresponding to the above-disclosed method for intelligent detection and measurement of aircraft surface cracks by combining images and point clouds, an embodiment of the present invention further discloses an intelligent detection and measurement system for aircraft surface cracks by combining images and point clouds, such as Figure 4 As shown, it specifically includes: Data acquisition module, used to collect two-dimensional images and three-dimensional point cloud data of the aircraft surface; The target detection module is configured to perform crack defect recognition on the collected two-dimensional image based on a trained target detection model. The crack defect point cloud determination module is configured to align the two-dimensional image and the three-dimensional point cloud, and obtain corresponding crack defect three-dimensional point cloud data according to the crack defect position recognized from the two-dimensional image. The crack defect measurement module is configured to perform filtering and down-sampling processing on the obtained crack defect three-dimensional point cloud data, and perform crack defect size measurement on the processed crack defect three-dimensional point cloud data to obtain length, width and depth information of the crack defect.

[0056] It should be noted that the detailed description of the aircraft surface crack intelligent detection and measurement system provided by the embodiment of the present application can refer to the related description of the aircraft surface crack intelligent detection and measurement method provided by the embodiment of the present application, which will not be repeated here.

[0057] In addition, the embodiment of the present application further provides an electronic device, which comprises a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of the aircraft surface crack intelligent detection and measurement method provided by any one of the above.

[0058] It should be noted that the detailed description of the electronic device provided by the embodiment of the present application can refer to the related description of the aircraft surface crack intelligent detection and measurement method provided by the embodiment of the present application, which will not be repeated here.

[0059] In addition, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the aircraft surface crack intelligent detection and measurement method provided by any one of the above.

[0060] It should be noted that the detailed description of the computer readable storage medium provided by the embodiment of the present application can refer to the related description of the aircraft surface crack intelligent detection and measurement method provided by the embodiment of the present application, which will not be repeated here.

[0061] Those skilled in the art can understand that all or part of the functions of various methods in the above embodiments can be realized by hardware or by a computer program. When all or part of the functions in the above embodiments are realized by a computer program, the program can be stored in a computer readable storage medium, which can include a read-only memory, a random access memory, a magnetic disk, an optical disk, a hard disk, and the like. The above functions are realized by executing the program by a computer. For example, the program is stored in a memory of a device, and the above functions are realized by executing the program in the memory by a processor. In addition, when all or part of the functions in the above embodiments are realized by a computer program, the program can also be stored in a storage medium such as a server, another computer, a disk, an optical disk, a flash disk, or a mobile hard disk, and is saved in a memory of a local device by downloading or copying, or the system of the local device is updated, and the above functions are realized by executing the program in the memory by a processor.

[0062] The above application of specific examples to the present application is described, which is only used to help understand the present application and does not limit the present application. For those skilled in the art, according to the idea of the present application, a number of simple deductions, deformations or substitutions can be made.

Claims

1. An intelligent detection and measurement method for aircraft surface cracks combining images and point clouds, characterized in that: The method comprises: Collect 2D images and 3D point cloud data of the aircraft surface; Crack defect recognition is performed on the collected two-dimensional images based on the trained target detection model; Align the two-dimensional image and the three-dimensional point cloud, and obtain the corresponding three-dimensional point cloud data of the crack defect according to the crack defect position identified from the two-dimensional image; The obtained three-dimensional point cloud data of crack defects are filtered and down-sampled, and the crack defect size is measured on the processed three-dimensional point cloud data of crack defects to obtain the length, width and depth information of the crack defects.

2. The method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds according to claim 1, characterized in that: Crack defect recognition is performed on the collected 2D images based on the trained target detection model, specifically including: The collected two-dimensional images are preprocessed, and training and test sets are constructed to train and test the YOLOv8 target detection model.

3. The method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds as claimed in claim 2, characterized in that: Preprocess the collected 2D images and construct training and test sets to train and test the YOLOv8 object detection model, including: The Mixup data augmentation method is used to expand the image dataset, including: randomly selecting samples and corresponding labels in the image dataset to generate mixed new images and labels: in, are the coordinates of the generated mixed new image, , is a random coordinate point in the data set The coordinates of is the random sampling coefficient; Use the local mean-based image reduction algorithm to uniformly scale the two-dimensional image to the size required for target detection model training, including: adjusting the original image matrix to the target size by the scaling factor. If the original image matrix is , the scaling factor is , then the reduced image matrix is ; Normalize the pixel values ​​of the image to between 0 and 1, including: for each pixel of the image matrix , set the maximum pixel value and pixel minimum , normalized pixel value for: The processed image dataset is divided into training set and test set, the training parameters are configured and the target detection model is trained. The obtained target detection model is used to identify crack defects on the aircraft surface.

4. The method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds according to claim 1, characterized in that: Align the 2D image and the 3D point cloud, and obtain the corresponding 3D point cloud data of the crack defect based on the crack defect position identified from the 2D image, specifically including: Establishing a spatial mapping relationship between the two-dimensional image and the three-dimensional point cloud, including: projecting each three-dimensional point in the three-dimensional point cloud data onto the two-dimensional image plane through camera calibration parameters; For each point in the point cloud, calculate the corresponding pixel coordinates on the two-dimensional image plane, including: according to the point cloud data row number Inverse the coordinates of the corresponding pixel points: in, is the point cloud data row number, is the coordinate of the corresponding pixel point on the two-dimensional image plane; Determine pixel coordinates Is it in the crack defect rectangle box identified by the target detection model? The rectangle box is composed of the upper left pixel coordinate point. and the lower right pixel coordinate point Composition, of which Indicates yes The horizontal coordinate of the point, Indicates yes The vertical coordinate of the point, Indicates yes The horizontal coordinate of the point, Indicates yes The vertical coordinate of the point; If the pixel coordinates corresponding to the points in the point cloud Meet the conditions , then the points in the point cloud that meet the conditions are marked as crack defect point cloud.

5. The method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds according to claim 1, characterized in that: The obtained three-dimensional point cloud data of crack defects is filtered and down-sampled, specifically including: Adopt the normal-based point cloud bilateral filtering method, including: Calculate the normal of each point in the point cloud and build a spatial index structure for fast neighborhood search; Perform neighborhood search for each point in the point cloud based on spatial distance and spatial standard deviation parameters Calculating spatial weights , based on the normal angle and the normal standard deviation parameter Calculating normal weights , perform weighted averaging based on the weights and update the position of the points: Based on the current processing point , points in the neighborhood ,point Normal ,point Normal , spatial weight function and the normal weight function , update the normal and output the filtered point cloud : 。 6. The method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds according to claim 1, characterized in that: The obtained three-dimensional point cloud data of crack defects is filtered and down-sampled, specifically including: Perform voxel downsampling on point cloud data, including: Analyze the spatial distribution characteristics of point cloud data and determine the The maximum and minimum values ​​of the three coordinate axes are used to calculate the minimum bounding box size of the point cloud; Determine the side length and size of the voxel grid, and perform voxel grid division on the minimum bounding box of the point cloud; For each non-empty voxel grid, calculate the centroid coordinates of all points inside it and replace all points in the small grid with the centroid.

7. The method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds according to claim 1, characterized in that: The processed three-dimensional point cloud data of crack defects is used to measure the size of the crack defects to obtain the length, width and depth information of the crack defects, including: For length measurement, the crack is analyzed at For the distribution in the axial direction, the leftmost and rightmost voxel points are selected and the distance between the two points is calculated to obtain the crack length; For width measurement, by analyzing For the distribution in the axial direction, the top and bottom voxel points are selected and the distance between the two points is calculated to obtain the crack width; For depth measurement, the crack defect point cloud is first fitted to obtain a reference plane, and the normal vector of the reference plane is calculated. Then all voxel points are projected in the direction of the normal vector, and the actual depth of the crack is obtained by calculating the difference between the maximum projection distance and the minimum projection distance.

8. An intelligent detection and measurement system for aircraft surface cracks combining images and point clouds, characterized in that: The system comprises: Data acquisition module, used to collect two-dimensional images and three-dimensional point cloud data of the aircraft surface; The target detection module is used to identify crack defects in the collected two-dimensional images based on the trained target detection model; A crack defect point cloud determination module is used to align the two-dimensional image and the three-dimensional point cloud, and obtain the corresponding three-dimensional point cloud data of the crack defect according to the crack defect position identified from the two-dimensional image; The crack defect measurement module is used to filter and downsample the obtained three-dimensional point cloud data of crack defects, and measure the crack defect size on the processed three-dimensional point cloud data to obtain the length, width and depth information of the crack defects.

9. An electronic device, characterized in that: The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of the method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for intelligent detection and measurement of aircraft surface cracks combining images and point clouds as described in any one of claims 1 to 7.

Images