Aircraft surface damage area segmentation method based on adaptive multi-modal fusion

Abstract

The invention relates to an aircraft surface damage area segmentation method based on adaptive multi-modal fusion, belongs to the technical field of nondestructive testing, and solves the technical problem of low segmentation precision of an aircraft skin curved surface falling damage area. Obtaining a grey-scale map, a depth map and a point cloud map of the aircraft surface damage skin; synthesizing the grey-scale map, the depth map and the point cloud map, and performing preprocessing, feature extraction, feature conversion, image display and projection and weighted fusion on the synthesized grey-scale map, the synthesized depth map and the synthesized point cloud map to obtain a total edge probability map of skin damage; and performing targeted screening, continuity detection, connectivity detection, convex hull construction, marking and skin falling damage area division on the total edge probability graph. The method is used for aircraft skin health monitoring, maintenance guarantee and operation and maintenance.

Description

Technical Field

[0001] The present invention belongs to the technical field of non-destructive testing, and in particular relates to a method for segmenting damaged areas on an aircraft surface based on adaptive multimodal fusion. Background Art

[0002] Because aircraft skins are constantly subjected to aerodynamic pressure, extreme temperature cycles, and external impacts during high-speed flight, they are susceptible to damage such as shedding and delamination of the surface coating. Repairing damaged coatings is a crucial part of routine aircraft maintenance. Detecting damaged aircraft coatings and rapidly segmenting the damage location is a prerequisite for subsequent repair work. Existing methods for detecting aircraft skin damage primarily rely on manual visual inspection, deep learning-based image detection, and 3D reconstruction-based point cloud segmentation.

[0003] Chinese invention patent CN118314388A discloses a method and system for aircraft skin damage detection based on the FC-YOLO network model. This method includes obtaining a raw data set, dividing it into a training set and a validation set, constructing an initial FC-YOLO network model, obtaining the FC-YOLO network model, and outputting aircraft skin damage identification results from the aircraft skin damage image to be detected. In practical engineering applications, this method requires a large amount of damage image data for training and can only identify the damage type, failing to segment the edge contours of the damaged area.

[0004] In his master's thesis, "Research on Aircraft Skin Damage Point Cloud Segmentation Methods Based on Dynamic Graph Convolution" (Nanjing University of Aeronautics and Astronautics, March 2023), Zhang Wenrui proposed a dynamic graph convolutional network based on feature updating to segment point cloud datasets of damaged areas, thereby achieving the purpose of identifying damage defects. This method does not require the construction of a damage dataset in advance, and theoretically has the function of segmenting the edge contour information of the damaged area. However, the author only explains how to use this method to identify the type of aircraft skin damage, and does not explain how to obtain the damage edge contour based on this method. Moreover, the accuracy of segmentation is very limited by simply using point clouds.

[0005] In his master's thesis, "Surface Defect Detection and Characterization of Aircraft Skin Based on 3D Point Clouds" (Xi'an University of Technology, May 2022), Zhang Yan proposed an improved local contrast saliency defect detection method and a 3D point cloud-based defect detection and quantitative characterization method for 2D image defect detection and 3D point cloud segmentation of aircraft skin, respectively. This method effectively segmented the edges of 3D damaged areas, but the principle remained solely based on point cloud segmentation. Image-based detection served as only one component of the research, without integrating multimodal data, which limited the improvement in final detection accuracy.

[0006] The above methods have the following technical problems when it comes to quickly and accurately segmenting the shedding damage of the skin surface:

[0007] Aircraft skins are typically complex curved surfaces. Image-based damage segmentation schemes cannot handle the geometric distortion caused by the curved surface characteristics of the skin and can only segment the damage location on a two-dimensional image. In addition, to achieve high accuracy, a large number of damage image datasets must be constructed in advance, which limits their application scenarios. Although damage segmentation methods based on three-dimensional point clouds can extract the edge contours of the damaged area on the surface, they are heavily dependent on the initial accuracy of the point cloud. On the other hand, the information contained in the simple point cloud data is limited, which limits the accuracy of the segmentation. In addition, most of the current research on damage area segmentation is carried out in ideal lighting conditions in the laboratory. For objects with certain reflective properties such as aircraft skin, the above methods are inevitably affected by lighting conditions. Once the lighting conditions change, it will cause serious interference to the image and point cloud information collected by the sensor, further limiting its application in actual engineering. Summary of the Invention

[0008] In order to overcome the shortcomings of low accuracy, large amount of skin data set, laser interference and two-dimensional reflection in the segmentation of damaged areas on aircraft skin surfaces, the present invention proposes a method for segmenting damaged areas on aircraft surfaces based on adaptive multimodal fusion.

[0009] The technical solution adopted by the present invention to solve the technical problem is:

[0010] A method for segmenting damaged areas on an aircraft surface based on adaptive multimodal fusion includes the following steps:

[0011] Step S1, collecting aircraft skin damage data: using a surface structured light acquisition system to collect damage data of the damaged skin on the aircraft surface, and obtaining a grayscale image, a depth image and a point cloud image of the damaged skin on the aircraft surface.

[0012] Step S2: extract the total edge probability map of skin damage:

[0013] The first step is synthesis: using the high dynamic range imaging (HDR) synthesis algorithm, the RGB grayscale image, depth map and point cloud image of the damaged skin on the aircraft surface are synthesized to obtain an HDR image.

[0014] The second step is to preprocess the HDR image: perform gamma correction on the synthesized grayscale image and the synthesized depth image respectively to obtain the preprocessed grayscale image and the preprocessed depth image; voxelize the synthesized point cloud image and truncate the invalid part of the coordinate data to obtain a valid synthesized point cloud image; extract the geometric features of the valid synthesized point cloud image, and use the least squares algorithm to smooth the valid synthesized point cloud image to obtain the preprocessed point cloud image.

[0015] The third step is feature extraction: feature extraction is performed on the preprocessed grayscale image and the preprocessed depth image to obtain grayscale gradient features and depth gradient features; feature extraction is performed on the preprocessed point cloud image to obtain curvature features and normal vector features.

[0016] The fourth step is feature conversion: the grayscale gradient features, depth gradient features, curvature features and normal vector features are converted respectively, that is, normalized to [0, 1] and nonlinearly processed to obtain grayscale edge probability features, depth edge probability features, curvature edge probability features and normal vector edge probability features.

[0017] The fifth step is to display and project the image into two-dimensional space: the image displays the grayscale edge probability features and the depth edge probability features to obtain the grayscale edge probability map and the depth edge probability map; the curvature edge probability features and the normal vector edge probability features are projected into two-dimensional space to obtain the curvature edge probability map and the normal vector edge probability map.

[0018] Step 6: Weighted fusion: Adaptive weighted fusion method is used to weightedly fuse the grayscale edge probability map, depth edge probability map, curvature edge probability map and normal vector edge probability map to obtain the total edge probability map.

[0019] Step S3, region segmentation:

[0020] The total edge probability map is subjected to targeted screening, continuity detection, connectivity detection, convex hull construction, and marking to demarcate the skin shedding damage area.

[0021] The above-mentioned method for segmenting damaged areas on the aircraft surface further includes step S4, calculating the damage segmentation accuracy.

[0022] The data processing system is used to calculate the segmentation accuracy of the skin loss damage area.

[0023] In the above-mentioned method for segmenting damaged areas on the aircraft surface, the surface structured light acquisition system includes a surface structured light camera and a data processing system.

[0024] In the above-mentioned method for segmenting the damaged area on the aircraft surface, the surface structured light acquisition system is the RVC-P5330 surface structured light acquisition aircraft skin damage data system.

[0025] In the above-mentioned method for segmenting damaged areas on an aircraft surface, step S2 further comprises:

[0026] In the first step:

[0027] Select at least 5 RGB grayscale images, depth images and point cloud images with different exposure times, and use the HDR synthesis algorithm to capture the details of the highlights, midtones and shadows respectively to obtain HDR images, namely, synthetic grayscale images, synthetic depth maps and synthetic point cloud images.

[0028] In the third step:

[0029] Perform feature extraction on the preprocessed grayscale image and preprocessed depth image, and use the Soble operator to extract the gradient value G of a single pixel in the x direction. x And the gradient value G in the y direction y :

[0030]

[0031]

[0032] Where I(i,j) is the pixel value of the input image, that is, the pixel value of the preprocessed grayscale image or the preprocessed depth map, i is the pixel index in the x direction, j is the pixel index in the y direction, G x (i,j) and G y (i, j) are the weight matrices of the Sobel operator in the horizontal and vertical directions, respectively, as follows:

[0033]

[0034] G x and G y It can capture the grayscale changes of pixels on the 3x3 grid in the x and y directions, and thus represent the gradient changes. x and G y , extract the gradient amplitude G, and the calculation formula is as follows:

[0035]

[0036] Perform feature extraction on the pre-processed point cloud image, and use the operator to extract the curvature feature and normal vector feature of the pre-processed point cloud image. Define the point cloud in the pre-processed point cloud image as P = {p1, p2, ..., p n}, construct a kdtree search to obtain its neighborhood points.

[0037] For each point p i , calculate its position vector v relative to the target point p i :

[0038] vi =p i -p (6)

[0039] Calculate the position vector v i The covariance matrix Conv(p i ):

[0040]

[0041] For the covariance matrix Conv(p i ) performs eigenvalue decomposition and obtains eigenvalues ​​λ1, λ2, and λ3, with the relationship λ1≤λ2≤λ3.

[0042] The curvature feature σ of the preprocessed point cloud i Defined as:

[0043]

[0044] Among them, λ min =min{λ1,λ2,λ3} is the minimum eigenvalue.

[0045] The principal component analysis method is used to obtain the normal vector features of the pre-processed point cloud image. Assume that there is an arbitrary point p on the point cloud P in the pre-processed point cloud image i and its adjacent point p i+1 , their respective minimum eigenvalues ​​λ min The corresponding eigenvector is v i 、v i+1 , normalize the eigenvector to get the corresponding normal vector feature n i 、n i+1 :

[0046]

[0047] In the fourth step:

[0048] First, the grayscale gradient features and depth gradient features are transformed as follows:

[0049] Normalize to the range of [0,1], crop and process outliers to obtain grayscale gradient features and depth gradient features with low outliers; normalize the grayscale gradient features and depth gradient features with low outliers to the range of [0,1], and perform nonlinear scaling to obtain grayscale edge probability features and depth edge probability features.

[0050] Secondly, perform feature transformation on the curvature feature to obtain the curvature edge probability feature. Specifically:

[0051] Normalize to the range of [0,1], crop and process outliers to obtain the curvature gradient features with low outliers; normalize the curvature gradient features with low outliers to the range of [0,1], perform nonlinear scaling, and obtain the curvature edge probability features.

[0052] Finally, the normal vector features are transformed to obtain the normal vector edge probability features. Specifically:

[0053] For each point in the preprocessed point cloud image, the sum of the normal vector angles between it and its surrounding neighboring points is calculated; normalized to the range of [0,1], outliers are cropped and processed to obtain normal vector gradient features with low outliers; normal vector gradient features with low outliers are normalized to the range of [0,1], nonlinearly scaled, and normal vector edge probability features are obtained.

[0054] In the fifth step:

[0055] Using the internal parameters of the surface structured light camera, the point cloud image is projected back to the two-dimensional depth map, and then the position of the pixel point in the two-dimensional depth map is projected into the point cloud image to find the point cloud point and depth Figure 2 The mapping relationship between dimensional pixels is used to obtain the curvature edge probability map and the normal vector edge probability map.

[0056] In step 6:

[0057] The grayscale edge probability map, depth edge probability map, curvature edge probability map and normal vector edge probability map are weighted, and an adaptive fusion weight ratio module is introduced to complete the feature fusion process. The fusion formula is as follows:

[0058] P all =ω gra P gra +ω dep P dep +ω cur P cur +ω nor P nor (10)

[0059] Among them, P all is the total marginal probability map, P gra is the grayscale edge probability map, P dep is the depth edge probability map, P cur is the curvature edge probability map, P nor is the normal vector edge probability map, ω k is the weight, k∈{gra,dep,cur,nor}.

[0060] The adaptive fusion weight ratio module assigns a dynamic weight ω to each feature mode k , the calculation formula is as follows:

[0061]

[0062] Each eigenmode is assigned a dynamic weight ω k The value range of is [0,1], and the sum of all modal weights is 1.

[0063] Significance factor α k (x) = exp(λ k Var Ω(x) (P k )) is used to measure the response strength of all modes k∈{gra,dep,cur,nor} in the local window Ω(x). Var Ω(x) is the variance of the edge probability map, which quantifies the significance of the edge response of the mode in the local area, and the modal correlation gain coefficient λ k Controls the influence of significance factors on weights.

[0064] β k (x) = 1-exp(-γ k μ k (x)) is the reliability factor of the noise level of the mode of interest, μ k (x) is the signal-to-noise ratio of the mode in the local area, γ k is the modal attenuation coefficient. When the modal signal-to-noise ratio is low, β k The value of (x) decreases, thereby reducing the weight of the modality in the fusion, effectively suppressing noise interference, and vice versa.

[0065] In the above-mentioned aircraft surface damage area segmentation method, in the sixth step of step S2, the weight is set to ω gra =0.52,ω dep =0.17,ω cur =0.11,ω nor =0.20.

[0066] In the above-mentioned method for segmenting damaged areas on an aircraft surface, step S3 further comprises:

[0067] The first step is targeted screening to remove the non-edge area clusters and noise area clusters of the total edge probability map to obtain the cluster total edge probability map.

[0068] In the second step, the continuity of the cluster total edge probability map is tested, so that the continuity is defined as the sum of the spatial distances between each point and the three adjacent points is within the set threshold, and the continuous total edge probability map of each point is obtained.

[0069] The third step is to perform connectivity testing on the continuous total edge probability map. Connectivity is defined as the sum of the differences between the gradient directions of each point and its three neighboring points, ensuring that the sum is within a set threshold. This yields the edge pixel set 1, 2, ... n for each damaged area. The gradient direction θ is extracted from the gradient value in the grayscale image. The gradient direction θ is calculated as follows:

[0070]

[0071] The fourth step is to construct the convex hull. Using the gradient direction θ, perform non-maximum suppression on the edge pixel set of the damaged area to sharpen the edge. Then, construct a two-dimensional convex hull for the sharpened edge region to obtain the damaged area pixel set 1, 2...n.

[0072] The fifth step is to mark the pixel set of each damaged area and divide the skin-shedding damaged area.

[0073] The beneficial effects of the present invention are:

[0074] A method for segmenting aircraft surface damage regions based on adaptive multimodal fusion solves the problem of damage segmentation when a large sample size is insufficient to construct a dataset. By introducing an adaptively designed fusion weight ratio module, multimodal fusion of five or more RGB images, five or more corresponding depth maps, and five or more corresponding point cloud data is required to achieve effective segmentation of surface damage.

[0075] A method for segmenting aircraft surface damage regions based on adaptive multimodal fusion achieves high-precision segmentation of aircraft skin shedding damage, with an accuracy of 0.02mm. For the damaged areas of aircraft skin shedding, the collected data is synthesized using an HDR synthesis algorithm. After preprocessing and feature extraction, including grayscale gradient features, depth gradient features, curvature features, and normal vector features, the multimodal edge probability map is adaptively fused and clustered to achieve high-precision damage segmentation.

[0076] A method for segmenting aircraft surface damage regions based on adaptive multimodal fusion is proposed. It is compatible with the suppression of background reflection interference from metal / composite skins and greatly improves the detection effect of skin shedding damage on curved surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a flow chart of a damaged area segmentation method according to embodiment 1 of the present invention;

[0078] Figure 2 This is a schematic diagram of a structured light acquisition system according to an embodiment of the present invention;

[0079] Figure 3 This is a flowchart of HDR synthesis image according to embodiment 1 of the present invention;

[0080] Figure 4 This is the HDR synthesized grayscale image according to the first embodiment of the present invention;

[0081] Figure 5 This is a grayscale image after γ correction in Example 1 of the present invention;

[0082] Figure 6 This is the point cloud surface after MLS smoothing in Example 1 of the present invention;

[0083] Figure 7 This is a flow chart of gradient feature conversion in accordance with the first embodiment of the present invention;

[0084] Figure 8 This is a flow chart of curvature feature conversion in accordance with the first embodiment of the present invention;

[0085] Figure 9 This is a flow chart of normal vector feature conversion in accordance with the first embodiment of the present invention;

[0086] Figure 10 This is a segmentation effect diagram of Example 1 of the present invention.

[0087] Figure numerals: 1. Surface structured light camera, 2. Data processing system, 3. Aircraft surface damaged skin. DETAILED DESCRIPTION

[0088] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0089] Example 1

[0090] like Figure 1 As shown, a method for segmenting aircraft surface damage regions based on adaptive multimodal fusion includes the following steps:

[0091] Step S1: Use the surface structured light acquisition system to obtain grayscale images, depth images, and point cloud images. The specific process is:

[0092] Surface structured light acquisition system Figure 2 As shown in FIG, the system consists of an RVC-P5330 surface structured light camera and a data processing system.

[0093] The grayscale image, depth image and point cloud image of the surface damage area are obtained through the surface structured light acquisition system.

[0094] Step S2: Design an edge recognition algorithm based on adaptive multimodality to extract the total edge probability map of skin damage.

[0095] In the first step, the high dynamic range imaging (HDR) synthesis algorithm is used to synthesize the collected grayscale image, depth map and point cloud image to obtain a high-precision and light source robust synthetic grayscale image, synthetic depth map and synthetic point cloud image. This technology enhances the visual effect of photos or videos by expanding the brightness expression range in digital images and videos. At least 5 images of three types with different exposure time settings are selected respectively, and the HDR algorithm is used to capture the details of the highlights, midtones and shadows in the skin image respectively to obtain HDR images, namely synthetic grayscale images, synthetic depth maps and synthetic point cloud images. The method flow is shown in Figure 3 . Greatly improve the quality of input images, thereby improving algorithm processing results.

[0096] In the second step, the synthetic grayscale image, synthetic depth map and synthetic point cloud are preprocessed. The synthetic grayscale image and synthetic depth map are gamma corrected to obtain the preprocessed grayscale image and preprocessed depth map respectively. The synthetic point cloud image is voxelized and the least squares (MLS) is used to obtain the preprocessed point cloud image:

[0097] Since the detection object has a curved surface and is made of metal, the captured image has brightness deviation. Figure 4 Perform gamma correction on the synthesized grayscale image to make the changes in the bright and dark areas of the image smooth and avoid problems in subsequent gradient extraction. The pre-processed grayscale image is obtained. The effect after correction is shown in Figure 5 The synthesized depth map is also gamma corrected to obtain the preprocessed depth map.

[0098] For the preprocessing of the synthetic point cloud, since the numerical accuracy of the coordinates of the points in the generated initial point cloud is too large, but the actual accuracy is not high, in order to avoid the difficulty of subsequent calculations caused by excessive data length, the synthetic point cloud is first voxelized with a unit of 0.01mm, and the invalid part of the coordinate data is truncated to obtain a valid synthetic point cloud. Secondly, the geometric features of the effective synthetic point cloud are extracted. Since the surface roughness of the initial point cloud will affect the subsequent calculations, the least squares (MLS) algorithm is used to smooth the effective synthetic point cloud to make the point cloud surface smoother, and the preprocessed point cloud is obtained. The smoothed effect can be seen. Figure 6 .

[0099] The third step is to extract features from the preprocessed grayscale image and the preprocessed depth image to obtain grayscale gradient features and depth gradient features; and to extract features from the preprocessed point cloud image to obtain curvature features and normal vector features:

[0100] For the preprocessed grayscale image or preprocessed depth image, the Soble operator is used to extract the gradient value G of a single pixel in the x direction. x And the gradient value G in the y direction y :

[0101]

[0102]

[0103] Where I(i,j) is the pixel value of the input image, that is, the pixel value of the preprocessed grayscale image or the preprocessed depth map, i is the pixel index in the x direction, j is the pixel index in the y direction, G x (i,j) and G y (i, j) is the weight matrix of the Sobel operator in the horizontal and vertical directions, as follows:

[0104]

[0105] G x and G y It can capture the grayscale changes of pixels on the 3x3 grid in the x and y directions, and then represent the gradient changes. x and G y Extract the gradient magnitude G:

[0106]

[0107] Extract the features of the pre-processed point cloud image and use the operator to extract the curvature features and normal vector features of the pre-processed point cloud image. The larger the curvature value of the point, the greater the possibility of it being an edge. For any point p, define a point cloud P = {p1, p2, ..., p n}, by specifying k points, constructing a kdtree search to obtain its neighborhood points.

[0108] For each point p i , calculate its position vector relative to the target point p:

[0109] v i =p i -p (6)

[0110] Calculate the position vector v i The covariance matrix Conv(p i ):

[0111]

[0112] Covariance matrix Conv(p i ) is symmetric, so eigenvalue decomposition can be performed. i ) performs eigenvalue decomposition and obtains eigenvalues ​​λ1, λ2, and λ3, with the relationship λ1≤λ2≤λ3.

[0113] The curvature feature σ of the preprocessed point cloudi Defined as:

[0114]

[0115] Among them, λ min =min{λ1,λ2,λ3} is the minimum eigenvalue.

[0116] The principal component analysis method is used to obtain the normal vector features of the pre-processed point cloud. Assume that there is an arbitrary point p on a point cloud P. i and its adjacent point p i+1 , their respective minimum eigenvalues ​​λ min The corresponding eigenvector is v i 、v i+1 , after normalizing the eigenvector, we get the corresponding normal vector feature n i 、n i+1 :

[0117]

[0118] The fourth step is to perform feature conversion on the grayscale gradient feature, depth gradient feature, curvature feature and normal vector feature respectively. By normalizing them to the range of [0, 1] and performing nonlinear processing, the grayscale gradient edge probability feature, depth gradient edge probability feature, curvature edge probability feature and normal vector edge probability feature are obtained:

[0119] The larger the gradient magnitude, the greater the probability that the point is an edge. Therefore, after calculating the gradient set for all pixel points, feature conversion is performed after counting all gradient magnitudes, so that these gradient features are converted into features that can better represent the probability of each point being an edge, that is, edge probability features.

[0120] First, the grayscale gradient features and depth gradient features are transformed. The specific gradient transformation method is shown in Figure 7 In order to unify the representation ability of edge probabilities of different features, the gradient features need to be normalized to the range of [0,1].

[0121] Perform histogram statistics on the data. Since the proportion of true edge pixels in all pixels is very small, the vertical axis is set to an exponential form with a base of 10 to represent the number of pixels.

[0122] Abnormally high-value points will cause an imbalance in the probability distribution due to overall linear scaling. High outliers will mask the low values ​​of the true edges. It is necessary to suppress outliers to optimize the rationality of the probability mapping, thereby improving the reliability of subsequent linear weighting. For grayscale images, pixels with abnormal gradient values ​​are noise points, and their grayscale values ​​are smoothed according to the pixel values ​​of the surrounding neighborhood. In depth images, excessively large gradient values ​​are mainly caused by depth loss areas caused by occlusion, such as object edges. Although the sudden edges in these areas have actual edge significance, the calculated abnormally high gradient values ​​will disrupt the balance of probability normalization. By suppressing abnormal amplitudes while retaining their edge structure information through gradient clipping, the effectiveness of the true edge and the need to suppress abnormal noise are balanced.

[0123] After processing outliers, we obtain the grayscale gradient features and depth gradient features for low outliers. Again, all values ​​of these low outlier grayscale gradient features and depth gradient features are linearly scaled to the [0, 1] range for normalization. Then, we perform nonlinear scaling to roughly unify the edge representation capabilities, resulting in grayscale edge probability features and depth edge probability features.

[0124] Secondly, perform feature transformation on the curvature feature to obtain the curvature edge probability feature. The process is shown in Figure 8 . Normalize the gradient features to the range [0,1],

[0125] Finally, the normal vector features are transformed to obtain the normal vector edge probability features. The process is shown in Figure 9 The size of the normal vector feature cannot be directly expressed as the probability of the edge. Since the normal vectors of the points on the edge will have a large angle difference with the surrounding points, the normal vector edge probability feature is defined as the sum of the angle differences between each point and all its neighboring points within the k-neighborhood range for subsequent processing.

[0126] The fifth step is to display the grayscale edge probability features and the depth edge probability features using images to obtain the grayscale edge probability map and the depth edge probability map. The curvature edge probability features and the normal vector edge probability features are projected into two-dimensional space and displayed as images to obtain the curvature edge probability feature map and the normal vector edge probability feature map:

[0127] The point cloud image is directly projected back into the depth map through the intrinsic parameters of the surface structured light camera. The positions of the pixels in the depth map can also be projected into the point cloud image through the intrinsic parameters of the camera. The correspondence between the points in the point cloud image and the two-dimensional pixels in the depth map is found. This relationship is consistent with the relationship between the edge probability features projected into the two-dimensional space. Therefore, the curvature edge probability map and the normal vector edge probability map can be output.

[0128] In the sixth step, the adaptive weighted fusion method is used to weightedly fuse the grayscale edge probability map, depth edge probability map, curvature edge probability map and normal vector edge probability map to obtain the total edge probability map:

[0129] The grayscale edge probability map, depth edge probability map, curvature edge probability feature map and normal vector edge probability feature map are weighted. The adaptive weighted fusion method introduces an adaptive fusion weight ratio module to complete the feature fusion process. The fusion formula is as follows:

[0130] P all =ω gra P gra +ω dep P dep +ω cur P cur +ω nor P nor (10)

[0131] Among them, P all is the total marginal probability map, P gra is the grayscale edge probability map, P dep is the depth edge probability map, P cur is the curvature edge probability map, P nor is the normal vector edge probability map, ω k is the weight, k∈{gra,dep,cur,nor}.

[0132] The adaptive fusion weight ratio module assigns a dynamic weight ω to each feature mode k , the calculation formula is as follows:

[0133]

[0134] Weight ω k The value range of is [0,1], and the sum of all modal weights is 1, ensuring the rationality and stability of the fusion process.

[0135] Significance factor α k (x) = exp(λ k Var Ω(x) (P k )) is used to measure the response strength of mode k within the local window Ω(x). Var Ω(x) is the variance of the edge probability map, which quantifies the significance of the edge response of the mode in the local area, and the modal correlation gain coefficient λ k Controls the influence of significance factors on weights.

[0136] In the form of an exponential function, the significance factor effectively amplifies the difference in response intensity of different modalities in local areas, so that regions with significant edges and modalities with high variance can obtain higher weights, thereby enhancing their contribution to the fusion results.

[0137] β k (x) = 1-exp(-γ k μk (x)) is the reliability factor of the noise level of the mode of interest, μ k (x) is the signal-to-noise ratio of the mode in the local area, γ k is the modal attenuation coefficient. When the modal signal-to-noise ratio is low, β k The value of (x) decreases, thereby reducing the weight of the modality in the fusion, effectively suppressing noise interference, and vice versa.

[0138] In the example picture of this embodiment, it is set to ω gra =0.52,ω dep =0.17,ω cur =0.11,ω nor =0.20.

[0139] Step S3: Perform targeted screening, continuity detection, connectivity detection, convex hull construction, and marking on the total edge probability map to obtain the skin shedding damage area.

[0140] Using probability graph growth clustering, multiple edge sets are obtained and then region partitioning is performed. Using continuity and connectivity filtering, multiple valid edge pixel sets of the damaged region are finally obtained. After constructing the convex hull, all pixels within the convex hull are the set of all pixels in the damaged region.

[0141] In the first step, the total edge probability map is targetedly screened to remove the clusters of non-edge areas and noise areas in the total edge probability map to obtain the clustered total edge probability map.

[0142] In the second step, the continuity of the cluster total edge probability map is tested. The continuity is defined as the sum of the spatial distances between each point and the three adjacent points. The sum is within the set threshold, and a continuous total edge probability map is obtained.

[0143] The third step is to perform connectivity detection on the continuous total edge probability map. Connectivity is defined as the sum of the differences between the gradient directions θ of each point and the three adjacent points. The sum is within the set threshold, and the edge pixel set 1, 2...n of each damaged area is obtained. The gradient direction is extracted from the gradient value of the grayscale image:

[0144]

[0145] In the fourth step, the gradient direction θ is used to perform non-maximum suppression on the edge pixel set of the damaged area, sharpen the edge, and construct a two-dimensional convex hull for the sharpened edge area to obtain the damaged area pixel set 1, 2...n.

[0146] The fifth step is to mark the pixel set of each damaged area, and the division of the skin-shedding damaged area is completed. The final effect is shown in Figure 10 .

[0147] Step S4: Using a data processing system to calculate the segmentation accuracy of the skin loss damage area. After multiple tests and verification, the accuracy reached 0.02mm, which can achieve sub-millimeter segmentation of aircraft skin loss damage.

[0148] This application aims to segment aircraft skin detachment damage by using the proposed high-precision damage area segmentation method on the aircraft surface based on adaptive multimodal fusion. The resulting damage segmentation is more accurate and highly flexible, thereby effectively supporting aircraft health monitoring, maintenance and operation, and aircraft design optimization. By improving the accuracy and efficiency of damage detection, it contributes to promoting the transformation of the aviation industry from "planned maintenance" to "predictive maintenance."

Claims

1. A method for segmenting aircraft surface damage regions based on adaptive multimodal fusion, characterized in that: The following steps are involved: Step S1, collecting damage data of the aircraft skin: using a surface structured light acquisition system to collect damage data of the damaged skin on the aircraft surface, and obtaining a grayscale image, a depth image, and a point cloud image of the damaged skin on the aircraft surface; Step S2: extract the total edge probability map of skin damage: The first step is synthesis: using a high dynamic range imaging (HDR) synthesis algorithm, the RGB grayscale image, depth image, and point cloud image of the damaged skin of the aircraft surface are synthesized to obtain an HDR image. The second step is to preprocess the HDR image: perform gamma correction on the synthesized grayscale image and the synthesized depth image to obtain the preprocessed grayscale image and the preprocessed depth image; The synthetic point cloud image is voxelized and the invalid part of the coordinate data is truncated to obtain a valid synthetic point cloud image; the geometric features of the valid synthetic point cloud image are extracted and the least squares algorithm is used to smooth the valid synthetic point cloud image to obtain a preprocessed point cloud image; The third step is feature extraction: feature extraction is performed on the preprocessed grayscale image and preprocessed depth image to obtain grayscale gradient features and depth gradient features; Perform feature extraction on the preprocessed point cloud image to obtain curvature features and normal vector features; The fourth step is feature conversion: the grayscale gradient feature, depth gradient feature, curvature feature and normal vector feature are converted respectively, that is, normalized to [0, 1] and nonlinearly processed to obtain grayscale edge probability feature, depth edge probability feature, curvature edge probability feature and normal vector edge probability feature; Step 5: Image display and projection into two-dimensional space: The image displays grayscale edge probability features and depth edge probability features to obtain grayscale edge probability maps and depth edge probability maps; the curvature edge probability features and normal vector edge probability features are projected into two-dimensional space to obtain curvature edge probability maps and normal vector edge probability maps; Step 6: Weighted fusion: Adaptive weighted fusion method is used to weightedly fuse the grayscale edge probability map, depth edge probability map, curvature edge probability map and normal vector edge probability map to obtain the total edge probability map; Step S3, region segmentation: The total edge probability map is subjected to targeted screening, continuity detection, connectivity detection, convex hull construction, and marking to demarcate the skin shedding damage area.

2. The method for segmenting damaged areas on an aircraft surface according to claim 1, wherein: The method comprises step S4, calculating the damage segmentation accuracy; The data processing system is used to calculate the segmentation accuracy of the skin loss damage area.

3. The method for segmenting damaged areas on an aircraft surface according to claim 1, wherein: The surface structured light acquisition system includes a surface structured light camera and a data processing system.

4. The method for segmenting damaged areas on an aircraft surface according to claim 1, wherein: The surface structured light acquisition system is the RVC-P5330 surface structured light acquisition aircraft skin damage data system.

5. The method for segmenting damaged areas on an aircraft surface according to claim 1, wherein: The step S2 further comprises: In the first step: Select at least five RGB grayscale images, depth images, and point cloud images with different exposure times, and use an HDR synthesis algorithm to capture the details of the highlights, midtones, and shadows respectively to obtain HDR images, namely, synthesized grayscale images, synthesized depth images, and synthesized point cloud images; In the third step: Perform feature extraction on the preprocessed grayscale image and preprocessed depth image, and use the Soble operator to extract the gradient value G of a single pixel in the x direction. x And the gradient value G in the y direction y : Where I(i,j) is the pixel value of the input image, that is, the pixel value of the preprocessed grayscale image or the preprocessed depth map, i is the pixel index in the x direction, j is the pixel index in the y direction, G x (i,j) and G y (i, j) are the weight matrices of the Sobel operator in the horizontal and vertical directions, respectively, as follows: G x and G y It can capture the grayscale changes of pixels on the 3x3 grid in the x and y directions, and thus represent the gradient changes; According to G x and G y , extract the gradient amplitude G, and the calculation formula is as follows: Perform feature extraction on the pre-processed point cloud image, and use the operator to extract the curvature feature and normal vector feature of the pre-processed point cloud image; define the point cloud in the pre-processed point cloud image as P = {p1, p2, ..., p n }, construct kdtree search to get its neighborhood points; For each point p i , calculate its position vector v relative to the target point p i : v i =p i -p (6) Calculate the position vector v i The covariance matrix Conv(p i ): For the covariance matrix Conv(p i ) to perform eigenvalue decomposition and obtain eigenvalues ​​λ1, λ2, λ3, with the relationship λ1≤λ2≤λ3. The curvature feature σ of the preprocessed point cloud is i Defined as: Among them, λ min =min{λ1,λ2,λ3} is the minimum eigenvalue; The principal component analysis method is used to obtain the normal vector features of the pre-processed point cloud image; suppose that there is any point p on the point cloud P in the pre-processed point cloud image i and its adjacent point p i+1 , their respective minimum eigenvalues ​​λ min The corresponding eigenvector is v i 、v i+1 , normalize the eigenvector to get the corresponding normal vector feature n i 、n i+1 : In the fourth step: First, the grayscale gradient features and depth gradient features are transformed as follows: Normalize to the range of [0,1], crop and process outliers to obtain grayscale gradient features and depth gradient features with low outliers; normalize the grayscale gradient features and depth gradient features with low outliers to the range of [0,1], perform nonlinear scaling, and obtain grayscale edge probability features and depth edge probability features; Secondly, the curvature features are transformed to obtain the curvature edge probability features; specifically: Normalize to the range of [0,1], crop and process outliers to obtain the curvature gradient features with low outliers; normalize the curvature gradient features with low outliers to the range of [0,1], perform nonlinear scaling, and obtain the curvature edge probability features; Finally, the normal vector features are transformed to obtain the normal vector edge probability features; specifically: For each point in the preprocessed point cloud, the sum of the normal vector angles between it and its neighboring points is calculated; normalized to the range [0, 1], outliers are cropped and processed to obtain normal vector gradient features with low outliers; normal vector gradient features with low outliers are normalized to the range [0, 1], nonlinearly scaled, and normal vector edge probability features are obtained; In the fifth step: Using the intrinsic parameters of the structured light camera, the point cloud image is projected back into a 2D depth map. The positions of the pixels in the 2D depth map are then projected onto the point cloud image. The correspondence between the points in the point cloud image and the 2D pixels in the depth map is found to obtain the curvature edge probability map and the normal vector edge probability map. In step 6: The grayscale edge probability map, depth edge probability map, curvature edge probability map and normal vector edge probability map are weighted, and an adaptive fusion weight ratio module is introduced to complete the feature fusion process. The fusion formula is as follows: P all =ω gra P gra +oh dep P dep +oh cur P cur +oh nor P nor (10) Among them, P all is the total marginal probability map, P gra is the grayscale edge probability map, P dep is the depth edge probability map, P cur is the curvature edge probability map, P nor is the normal vector edge probability map, ω k is the weight, k∈{gra,dep,cur,nor}; The adaptive fusion weight ratio module assigns a dynamic weight ω to each feature mode k , the calculation formula is as follows: k∈{gra,dep,cur,nor} Each eigenmode is assigned a dynamic weight ω k The value range of is [0,1], and the sum of all modal weights is 1; Significance factor α k (x) = exp(λ k Var Ω(x) (P k )) is used to measure the response strength of all modes k∈{gra,dep,cur,nor} in the local window Ω(x); Var Ω(x) is the variance of the edge probability map, which quantifies the significance of the edge response of the mode in the local area, and the modal correlation gain coefficient λ k Control the influence of significance factors on weights; β k (x) = 1-exp(-γ k μ k (x)) is the reliability factor of the noise level of the mode of interest, μ k (x) is the signal-to-noise ratio of the mode in the local area, γ k is the modal attenuation coefficient. When the modal signal-to-noise ratio is low, β k The value of (x) decreases, thereby reducing the weight of the modality in the fusion, effectively suppressing noise interference, and vice versa.

6. The method for segmenting damaged areas on an aircraft surface according to claim 5, characterized in that: In the sixth step S2, The weight is set to ω gra =0.52,ω dep =0.17,ω cur =0.11,ω nor =0.

20.

7. The method for segmenting damaged areas on an aircraft surface according to claim 1, characterized in that: The step S3 further comprises: The first step is targeted screening to remove the non-edge area clusters and noise area clusters of the total edge probability map to obtain the cluster total edge probability map; In the second step, the continuity test is performed on the cluster total edge probability map, so that the continuity is defined as the sum of the spatial distances between each point and the three adjacent points is within the set threshold, and the continuous total edge probability map of each point is obtained; In the third step, connectivity detection is performed on the continuous total edge probability map. Connectivity is defined as the sum of the differences between the gradient directions of each point and the three adjacent points. The sum is kept within the set threshold, and the edge pixel set 1, 2...n of each damaged area is obtained. Extract the gradient direction θ from the gradient value of the grayscale image. The gradient direction θ is calculated as follows: The fourth step is to construct the convex hull; use the gradient direction θ to perform non-maximum suppression on the edge pixel set of the damaged area. Sharpen the edge, construct a two-dimensional convex hull for the sharp edge area, and obtain the pixel set 1,2...n of the damaged area; The fifth step is to mark the pixel set of each damaged area and divide the skin-shedding damaged area.

Images