An aircraft surface damage region segmentation method based on adaptive multi-modal fusion

Abstract

The present application relates to a kind of aircraft surface damage area segmentation method based on adaptive multi-modal fusion, belong to nondestructive testing technical field, solve the technical problem of low precision of aircraft skin surface peeling damage area segmentation, it includes collecting aircraft skin damage data, obtains the gray scale chart, depth chart and point cloud chart of aircraft surface damage skin;Synthesis is carried out to gray scale chart, depth chart and point cloud chart, and the steps of obtaining the total edge probability chart of skin damage are carried out to the pre-processing, feature extraction, feature conversion, image display and projection, weighted fusion of synthesized gray scale chart, synthesized depth chart and synthesized point cloud chart;Total edge probability chart is screened, continuity detection, connectedness detection, constructs convex hull, marks, and divides the steps of skin peeling damage area.This application is used for aircraft skin health monitoring, maintenance support and operation.

Description

Technical Field

[0001] This invention belongs to the field of nondestructive testing technology, specifically relating to a method for segmenting damaged areas on the surface of aircraft based on adaptive multimodal fusion. Background Technology

[0002] Because aircraft skin is constantly subjected to aerodynamic pressure, extreme temperature cycles, and impacts from external foreign objects during high-speed flight, it is highly susceptible to damage such as peeling and delamination of the coating. Repairing damaged coatings is a crucial part of routine aircraft maintenance. Detecting and quickly segmenting the damaged coating is a prerequisite for subsequent repair work. Current methods for detecting damage to aircraft skin primarily rely on manual visual inspection, deep learning-based image detection, and point cloud segmentation based on 3D reconstruction.

[0003] Chinese invention patent CN118314388A discloses a method and system for detecting aircraft skin damage based on an FC-YOLO network model. The method includes steps such as obtaining the original dataset, dividing it into training and validation sets, constructing an initial FC-YOLO network model, obtaining the final FC-YOLO network model, and outputting the aircraft skin damage identification results from the images of the aircraft skin to be detected. However, in practical engineering applications, this method requires a large amount of damage image data for training and can only identify the type of damage, failing to segment the edge contour information of the damaged area.

[0004] In his master's thesis, "Research on Point Cloud Segmentation Method for Aircraft Skin Damage Based on Dynamic Graph Convolution" (Nanjing University of Aeronautics and Astronautics, March 2023), Zhang Wenrui proposed a dynamic graph convolutional network based on feature updates to segment the point cloud dataset of the damaged region, thereby achieving the purpose of identifying damage defects. This method does not require prior construction of the damage dataset and theoretically possesses the function of segmenting the edge contour information of the damaged region. However, the author only describes using this method to identify the type of aircraft skin damage, without explaining how to obtain the damage edge contour based on it. Furthermore, simply using the point cloud for segmentation results in very limited accuracy.

[0005] In her master's thesis, "Defect Detection and Characterization of Aircraft Skin Surface Based on 3D Point Clouds" (Xi'an University of Technology, May 2022), Zhang Yan proposed an improved method for detecting defects based on local contrast saliency and a method for defect detection and quantification characterization based on 3D point clouds, respectively used for defect detection in 2D images of aircraft skin and segmentation of 3D point clouds. This method achieves good edge segmentation of 3D damaged regions; however, its principle still relies solely on point cloud segmentation, with image-based detection being only one aspect of the research. The lack of multimodal data fusion limits the improvement of final detection accuracy.

[0006] The above methods have the following main technical problems when dealing with the rapid and accurate segmentation of skin surface detachment damage:

[0007] Aircraft skin typically consists of complex curved surfaces. Image-based damage segmentation schemes cannot handle the geometric distortions caused by the surface characteristics of the skin. They can only segment damage locations on two-dimensional images, and to achieve high accuracy, a large dataset of damage images is usually required beforehand, limiting their application scenarios. While 3D point cloud-based damage segmentation methods can extract the edge contours of curved surface damage areas, they heavily rely on the initial accuracy of the point cloud. Furthermore, point cloud data alone contains limited information, both of which restrict segmentation accuracy. In addition, most current research on damage region segmentation is conducted in ideal laboratory lighting conditions. For objects like aircraft skin, which have reflective properties, the above methods are inevitably affected by lighting conditions. Changes in lighting conditions can severely interfere with the images and point cloud information acquired by sensors, further limiting their application in practical engineering. Summary of the Invention

[0008] To overcome the shortcomings of low accuracy, the need for a large amount of skin data, and the susceptibility to laser interference and two-dimensional reflection in segmenting damaged areas of aircraft skin surfaces, this invention proposes a method for segmenting damaged areas of aircraft surfaces based on adaptive multimodal fusion.

[0009] The technical solution adopted by this invention to solve its technical problem is:

[0010] A method for segmenting damaged regions on the surface of an aircraft based on adaptive multimodal fusion includes the following steps:

[0011] Step S1: Collect aircraft skin damage data: Use a surface structured light acquisition system to collect damage data on the damaged skin of the aircraft surface, and obtain grayscale images, depth maps and point cloud maps of the damaged skin of the aircraft surface.

[0012] Step S2, extract the total edge probability map of skin damage:

[0013] The first step is synthesis: using a high dynamic range imaging (HDR) synthesis algorithm, the RGB grayscale image, depth map, and point cloud image of the damaged skin on the aircraft surface are synthesized to obtain an HDR image.

[0014] The second step is to preprocess the HDR image: gamma correction is performed on the synthesized grayscale image and the synthesized depth image to obtain the preprocessed grayscale image and the preprocessed depth image; the synthesized point cloud image is voxelized to truncate the invalid parts of the coordinate data to obtain the effective synthesized point cloud image; the geometric features of the effective synthesized point cloud image are extracted, and the least squares algorithm is used to smooth the effective synthesized point cloud image to obtain the preprocessed point cloud image.

[0015] The third step is feature extraction: feature extraction is performed on the preprocessed grayscale image and the preprocessed depth image to obtain grayscale gradient features and depth gradient features; feature extraction is performed on the preprocessed point cloud image to obtain curvature features and normal vector features.

[0016] The fourth step is feature transformation: feature transformation is performed on gray-level gradient features, depth gradient features, curvature features and normal vector features respectively, that is, normalized to [0,1] and nonlinearized to obtain gray-level edge probability features, depth edge probability features, curvature edge probability features and normal vector edge probability features.

[0017] Step 5: Image display and projection into two-dimensional space: The image displays grayscale edge probability features and depth edge probability features to obtain grayscale edge probability maps and depth edge probability maps; the curvature edge probability features and normal vector edge probability features are projected into two-dimensional space to obtain curvature edge probability maps and normal vector edge probability maps.

[0018] Step 6, weighted fusion: Adaptive weighted fusion method is used to weightedly fuse the grayscale edge probability map, depth edge probability map, curvature edge probability map and normal vector edge probability map to obtain the total edge probability map.

[0019] Step S3, Region Segmentation:

[0020] The total edge probability map is subjected to targeted screening, continuity detection, connectivity detection, convex hull construction, and labeling to delineate the skin detachment damage area.

[0021] The above-mentioned method for segmenting damaged areas on the surface of an aircraft also includes step S4, which calculates the accuracy of the damage segmentation.

[0022] A data processing system was used to calculate the segmentation accuracy of the skin detachment damage area.

[0023] The aforementioned method for segmenting damaged areas on the surface of an aircraft includes a surface structured light acquisition system comprising a surface structured light camera and a data processing system.

[0024] The above-mentioned method for segmenting damaged areas on the surface of an aircraft uses a surface structured light acquisition system, such as the RVC-P5330 surface structured light acquisition system for aircraft skin damage data.

[0025] The above-mentioned method for segmenting damaged areas on the surface of an aircraft, wherein step S2 further includes:

[0026] In the first step:

[0027] Select at least 5 RGB grayscale images, depth maps, and point cloud images with different exposure times. Use the HDR compositing algorithm to capture the details of the highlights, midtones, and shadows respectively to obtain HDR images, namely, composite grayscale images, composite depth maps, and composite point cloud images.

[0028] In the third step:

[0029] Feature extraction is performed on the preprocessed grayscale image and the preprocessed depth image. The Soble operator is used to extract the gradient value G of each pixel in the x-direction. x and the gradient value G in the y direction y :

[0030]

[0031]

[0032] Where I(i,j) is the pixel value of the input image, i.e., the pixel value of the preprocessed grayscale image or the preprocessed depth image, i is the pixel index in the x-direction, j is the pixel index in the y-direction, and G... x (i,j) and G y (i,j) are the weight matrices of the Sobel operator in the horizontal and vertical directions, respectively, as follows:

[0033]

[0034] G x and G y It can capture the magnitude of pixel grayscale changes on a 3x3 grid in the x and y directions, thus representing gradient changes. According to G... x and G y The gradient magnitude G is extracted and calculated as follows:

[0035]

[0036] Feature extraction is performed on the preprocessed point cloud image, using operators to extract curvature and normal vector features. The point cloud in the preprocessed point cloud image is defined as P = {p1, p2, ..., p...}. n}, construct a kdtree to search for its neighborhood points.

[0037] For each point p i Calculate its position vector v relative to the target point p. i :

[0038] vi =p i -p (6)

[0039] Calculate the position vector v i The covariance matrix Conv(p) i ):

[0040]

[0041] For the covariance matrix Conv(p) i Eigenvalue decomposition is performed to obtain eigenvalues ​​λ1, λ2, λ3, with the relationship λ1≤λ2≤λ3.

[0042] The curvature feature σ of the preprocessed point cloud map i Defined as:

[0043]

[0044] Where, λ min =min{λ1,λ2,λ3} is the minimum eigenvalue.

[0045] Principal component analysis is used to obtain the normal vector characteristics of the preprocessed point cloud image. Let there exist any point p on the point cloud P in the preprocessed point cloud image. i and its neighboring point p i+1 Their respective minimum eigenvalues ​​λ min The corresponding feature vector is v i v i+1 After normalizing the eigenvectors, we obtain the corresponding normal vector eigenvector n. i n i+1 :

[0046]

[0047] In the fourth step:

[0048] First, feature transformation is performed on the grayscale gradient features and depth gradient features, specifically as follows:

[0049] Normalized to the range of [0,1], outliers are clipped and processed to obtain gray-level gradient features and depth gradient features with low outliers; the gray-level gradient features and depth gradient features with low outliers are normalized to the range of [0,1] and non-linearly scaled to obtain gray-level edge probability features and depth edge probability features.

[0050] Secondly, feature transformation is performed on the curvature features to obtain curvature edge probability features. Specifically:

[0051] Normalized to the range of [0,1], outliers are clipped and processed to obtain curvature gradient features with low outliers; the curvature gradient features with low outliers are normalized to the range of [0,1] and nonlinearly scaled to obtain curvature edge probability features.

[0052] Finally, feature transformation is performed on the normal vector features to obtain the marginal probability features of the normal vector. Specifically:

[0053] For each point in the preprocessed point cloud map, calculate the sum of the angles between its normal vector and the normal vectors of its surrounding neighboring points; normalize to the range of [0,1], and crop and process outliers to obtain the normal vector gradient features with low outliers; normalize the normal vector gradient features with low outliers to the range of [0,1], and perform nonlinear scaling to obtain the normal vector edge probability features.

[0054] In step five:

[0055] Using the intrinsic parameters of a structured light camera, the point cloud image is projected back onto a 2D depth map. Then, the positions of pixels in the 2D depth map are projected onto the point cloud image to find the relationship between points and depth in the point cloud image. Figure 2 By mapping the 3D pixels, we obtain the curvature edge probability map and the normal vector edge probability map.

[0056] In step six:

[0057] The grayscale edge probability map, depth edge probability map, curvature edge probability map, and normal vector edge probability map are weighted, and an adaptive fusion weight ratio module is introduced to complete the feature fusion process. The fusion formula is as follows:

[0058] P all =ω gra P gra +ω dep P dep +ω cur P cur +ω nor P nor (10)

[0059] Among them, P all For the total marginal probability map, P gra P is a grayscale edge probability map. dep For the depth edge probability map, P cur For the curvature marginal probability map, P nor For the marginal probability map of the normal vector, ω k Let be the weights, k∈{gra,dep,cur,nor}.

[0060] The adaptive fusion weight ratio module assigns dynamic weights ω to each feature modality. k The calculation formula is as follows:

[0061]

[0062] Each feature mode is assigned a dynamic weight ω. k The value of is in the range of [0,1], and the sum of all modal weights is 1.

[0063] Significance factor α k (x)=exp(λ k Var Ω(x) (P k This is used to measure the response intensity of all modes k∈{gra,dep,cur,nor} within the local window Ω(x). Ω(x) The variance of the marginal probability map quantifies the significance of the marginal response of this mode in the local region, and the mode correlation gain coefficient λ. k Control the degree of influence of significance factors on weights.

[0064] β k (x)=1-exp(-γ k μ k (x)) is the reliability factor for the noise level of the mode of interest, μ k (x) represents the signal-to-noise ratio of the mode in the local region, γ k β is the modal attenuation coefficient. When the signal-to-noise ratio of the modes is low, β k Decreasing the value of (x) reduces the weight of that mode in the fusion, effectively suppressing noise interference, and vice versa.

[0065] In the above-mentioned method for segmenting damaged areas on the surface of aircraft, in step S2, the weight is set to ω. gra =0.52, ω dep =0.17, ω cur =0.11, ω nor =0.20.

[0066] The above-mentioned method for segmenting damaged areas on the surface of an aircraft, wherein step S3 further includes:

[0067] The first step is to selectively filter and remove non-marginal region clusters and noisy region clusters from the total marginal probability map to obtain the total marginal probability map of the clusters.

[0068] The second step is to perform continuity detection on the total edge probability map of the cluster, and define continuity as the sum of the spatial distances between each point and its three neighboring points within a set threshold, so as to obtain the continuous total edge probability map of each point.

[0069] The third step involves connectivity detection on the continuous total edge probability map. Connectivity is defined as the sum of the differences in gradient directions between each point and its three neighboring points, ensuring the summation is within a set threshold. This yields the pixel set 1, 2...n for the edge of the damaged region at each location. The gradient direction θ is extracted from the gradient values ​​in the grayscale image. The formula for calculating the gradient direction θ is as follows:

[0070]

[0071] The fourth step is to construct the convex hull. Using the gradient direction θ, non-maximum suppression is applied to the pixel set at the edge of the damaged region to sharpen the edges. A two-dimensional convex hull is then constructed for the sharpened edge region to obtain the pixel set 1, 2...n of the damaged region.

[0072] The fifth step is to label the pixel set of each damaged area to delineate the skin detachment damage area.

[0073] The beneficial effects of this invention are:

[0074] An adaptive multimodal fusion-based method for segmenting surface damage regions on aircraft addresses the problem of damage segmentation when a large dataset is lacking. By introducing an adaptive fusion weight ratio module, only 5 or more RGB images, 5 or more corresponding depth images, and 5 or more corresponding point cloud data are required to perform multimodal fusion on surface damage, ultimately achieving the desired segmentation effect.

[0075] A method for segmenting surface damage regions of aircraft based on adaptive multimodal fusion is proposed, achieving high-precision segmentation of aircraft skin detachment damage with an accuracy of 0.02 mm. For detachment damage on curved aircraft skin surfaces, an HDR synthesis algorithm is used to synthesize and preprocess the collected data, extracting features including grayscale gradient features, depth gradient features, curvature features, and normal vector features. The multimodal edge probability maps are adaptively fused, and clustered regions are generated for segmentation, ultimately achieving high-precision damage segmentation.

[0076] A method for segmenting damaged areas on aircraft surfaces based on adaptive multimodal fusion is proposed, which is compatible with the suppression of background reflection interference from metal / composite material skins and greatly improves the detection effect of detachment damage from curved skins. Attached Figure Description

[0077] Figure 1 This is a flowchart of the damage region segmentation method according to Embodiment 1 of the present invention;

[0078] Figure 2 This is a schematic diagram of a structured light acquisition system according to an embodiment of the present invention;

[0079] Figure 3 This is a flowchart of the HDR image synthesis process according to Embodiment 1 of the present invention;

[0080] Figure 4 This is an HDR composite grayscale image from Embodiment 1 of the present invention;

[0081] Figure 5 This is a grayscale image after γ-correction according to Embodiment 1 of the present invention;

[0082] Figure 6 This is the point cloud surface after MLS smoothing according to Embodiment 1 of the present invention;

[0083] Figure 7 This is a flowchart of gradient feature transformation according to Embodiment 1 of the present invention;

[0084] Figure 8 This is a flowchart of curvature feature conversion according to Embodiment 1 of the present invention;

[0085] Figure 9 This is a flowchart of the normal vector feature transformation process according to Embodiment 1 of the present invention;

[0086] Figure 10 This is a diagram showing the segmentation effect of Embodiment 1 of the present invention.

[0087] Reference numerals: 1. Surface structured light camera, 2. Data processing system, 3. Damaged skin on the aircraft surface. Detailed Implementation

[0088] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0089] Example 1

[0090] like Figure 1 As shown, a method for segmenting damaged areas on the surface of an aircraft based on adaptive multimodal fusion includes the following steps:

[0091] Step S1: Using a structured light acquisition system, acquire grayscale images, depth maps, and point cloud images. The specific process is as follows:

[0092] See surface structured light acquisition system Figure 2 As shown, the system consists of the RVC-P5330 surface structured light camera and a data processing system.

[0093] Grayscale images, depth maps, and point cloud images of damaged areas on curved surfaces are obtained using a surface structured light acquisition system.

[0094] Step S2: Design an edge recognition algorithm based on adaptive multimodal mode to extract the total edge probability map of skin damage.

[0095] The first step involves using a High Dynamic Range (HDR) imaging synthesis algorithm to synthesize the acquired grayscale image, depth map, and point cloud image to obtain high-precision synthesized grayscale image, synthesized depth map, and synthesized point cloud image with robustness to different light sources. This technique enhances the visual effect of photos or videos by expanding the range of brightness expression in digital images and videos. At least five images of the three types with different exposure durations are selected, and the HDR algorithm is used to capture details of the highlights, midtones, and shadows in the skinning image to obtain HDR images, i.e., the synthesized grayscale image, synthesized depth map, and synthesized point cloud image. The method flow is described in [link to flowchart]. Figure 3 This greatly improves the quality of the input image, thereby enhancing the algorithm's processing results.

[0096] The second step involves preprocessing the synthesized grayscale image, synthesized depth image, and synthesized point cloud. Gamma correction is applied to the synthesized grayscale image and synthesized depth image to obtain preprocessed grayscale and preprocessed depth images, respectively. Voxelization and least squares (MLS) are then applied to the synthesized point cloud image to obtain the preprocessed point cloud image.

[0097] Because the object being detected has curved surfaces and is made of metal, the captured image exhibits brightness deviations. (See...) Figure 4 Gamma correction is applied to the synthesized grayscale image to smooth out the changes in bright and dark areas, avoiding problems in subsequent gradient extraction. This yields a preprocessed grayscale image. The corrected result is shown in the image below. Figure 5 The synthesized depth map is also subjected to gamma correction to obtain a preprocessed depth map.

[0098] For the preprocessing of the synthesized point cloud image, since the coordinate values ​​of the points in the generated initial point cloud have excessively high precision (but the actual precision is not high), to avoid the excessive data length causing difficulties for subsequent calculations, the synthesized point cloud image is first voxelized to a unit of 0.01 mm to truncate the invalid parts of the coordinate data, resulting in an effective synthesized point cloud image. Secondly, the geometric features of the effective synthesized point cloud image are extracted. Since the surface roughness of the initial point cloud image affects subsequent calculations, the least squares (MLS) algorithm is used to smooth the effective synthesized point cloud image, making the point cloud surface smoother, resulting in the preprocessed point cloud image. The smoothed effect is shown in [Figure number missing]. Figure 6 .

[0099] The third step involves extracting features from the preprocessed grayscale image and the preprocessed depth image to obtain grayscale gradient features and depth gradient features; and extracting features from the preprocessed point cloud image to obtain curvature features and normal vector features.

[0100] For the preprocessed grayscale image or the preprocessed depth image, the Soble operator is used to extract the gradient value G of each individual pixel in the x-direction. x and the gradient value G in the y direction y :

[0101]

[0102]

[0103] Where I(i,j) is the pixel value of the input image, i.e., the pixel value of the preprocessed grayscale image or the preprocessed depth image, i is the pixel index in the x-direction, j is the pixel index in the y-direction, and G... x (i,j) and G y (i,j) is the weight matrix of the Sobel operator in the horizontal and vertical directions, as follows:

[0104]

[0105] G x and G y It can capture the magnitude of pixel grayscale changes on a 3x3 grid in the x and y directions, thus representing gradient changes. Finally, based on G... x and G y Extract the gradient magnitude G:

[0106]

[0107] Feature extraction is performed on the preprocessed point cloud image. Operators are used to extract the curvature and normal vector features of the preprocessed point cloud image. Points with larger curvature values ​​are more likely to be edges. For any point p, a point cloud P = {p1, p2, ..., p} is defined. n By specifying k points, a kd-tree search is constructed to obtain their neighborhood points.

[0108] For each point p i Calculate its position vector relative to the target point p:

[0109] v i =p i -p (6)

[0110] Calculate the position vector v i The covariance matrix Conv(p) i ):

[0111]

[0112] Covariance matrix Conv(p) i The covariance matrix Conv(p) is symmetric, therefore eigenvalue decomposition can be performed. i Eigenvalue decomposition is performed to obtain eigenvalues ​​λ1, λ2, λ3, with the relationship λ1≤λ2≤λ3.

[0113] The curvature feature σ of the preprocessed point cloud mapi Defined as:

[0114]

[0115] Where, λ min =min{λ1,λ2,λ3} is the minimum eigenvalue.

[0116] Principal component analysis (PCA) is used to obtain the normal vector characteristics of the preprocessed point cloud image. Suppose there exists an arbitrary point p on a point cloud P. i and its neighboring point p i+1 Their respective minimum eigenvalues ​​λ min The corresponding feature vector is v i v i+1 After normalizing the eigenvectors, the corresponding normal vector eigenvector n is obtained. i n i+1 :

[0117]

[0118] The fourth step involves performing feature transformations on the grayscale gradient features, depth gradient features, curvature features, and normal vector features respectively. By normalizing them to the [0,1] range and performing nonlinear processing, the edge probability features of the grayscale gradient, depth gradient, curvature, and normal vectors are obtained.

[0119] The larger the gradient magnitude, the greater the probability that a point is an edge. Therefore, after calculating the gradient set for all pixels, feature transformation is performed after statistically analyzing all gradient magnitudes. This transforms the gradient features into features that can better represent the probability of each point being an edge, i.e., edge probability features.

[0120] First, feature transformation is performed on the grayscale gradient features and depth gradient features. See [link to details on gradient transformation methods] for more information. Figure 7 To unify the representational ability of different features to represent marginal probabilities, the gradient features need to be normalized to the range [0,1].

[0121] Histogram statistics are performed on the data. Since the proportion of true edge pixels among all pixels is very small, the vertical axis is set to an exponential form with a base of 10 to represent the number of pixels.

[0122] Outlier points can cause probability distribution imbalance due to overall linear scaling, and high outliers can mask low values ​​at real edges. Suppressing outliers is necessary to optimize the probability mapping and improve the reliability of subsequent linear weighting. For grayscale images, pixels with outlier gradient values ​​are noise; their grayscale values ​​are smoothed based on the values ​​of their surrounding pixels. In depth maps, excessively large gradient values ​​are mainly caused by occlusion leading to missing depth regions, such as object edges. While these abrupt edges have actual edge significance, the calculated outlier gradient values ​​can disrupt the balance of probability normalization. Gradient limiting suppresses outlier amplitudes while preserving edge structure information, balancing the effectiveness of real edges with the need to suppress outlier noise.

[0123] After processing outliers, the gray-level gradient features and depth gradient features of low outliers are obtained. All values ​​of the gray-level gradient features and depth gradient features of low outliers are then linearly scaled to the [0,1] interval for normalization. Afterwards, non-linear scaling is performed to ensure that the edge representation capabilities are roughly uniform, resulting in gray-level edge probability features and depth edge probability features.

[0124] Secondly, feature transformation is performed on the curvature features to obtain curvature edge probability features. The process is as follows: Figure 8 The gradient features are normalized to the range [0,1].

[0125] Finally, feature transformation is performed on the normal vector features to obtain the normal vector marginal probability features. The process is as follows: Figure 9 The magnitude of the normal vector feature cannot be directly represented as the edge probability. Since the normal vector of a point on the edge will have a large angular difference with the surrounding points, the edge probability feature of the normal vector is defined as the sum of the angular differences between the normal vectors of each point and all its neighboring points within a k-neighborhood range, and then further processing is performed.

[0126] The fifth step involves displaying the grayscale edge probability features and depth edge probability features using an image to obtain grayscale edge probability maps and depth edge probability maps. The curvature edge probability features and normal vector edge probability features are then projected onto a two-dimensional space and displayed as an image to obtain curvature edge probability feature maps and normal vector edge probability feature maps.

[0127] The point cloud map is directly reflected back to the depth map through the intrinsic parameters of the surface structured light camera. The position of the pixel in the depth map can also be projected onto the point cloud map through the camera intrinsic parameters. The correspondence between the points in the point cloud map and the two-dimensional pixels in the depth map can be found. This relationship is consistent with the relationship of the edge probability feature projected into two-dimensional space. Therefore, the curvature edge probability map and the normal vector edge probability map can be output.

[0128] Step 6: Use an adaptive weighted fusion method to weight and fuse the grayscale edge probability map, depth edge probability map, curvature edge probability map, and normal vector edge probability map to obtain the total edge probability map.

[0129] The grayscale edge probability map, depth edge probability map, curvature edge probability feature map, and normal vector edge probability feature map are weighted. An adaptive weighted fusion method introduces an adaptive fusion weight ratio module to complete the feature fusion process. The fusion formula is as follows:

[0130] P all =ω gra P gra +ω dep P dep +ω cur P cur +ω nor P nor (10)

[0131] Among them, P all For the total marginal probability map, P gra P is a grayscale edge probability map. dep For the depth edge probability map, P cur For the curvature marginal probability map, P nor For the marginal probability map of the normal vector, ω k Let be the weights, k∈{gra,dep,cur,nor}.

[0132] The adaptive fusion weight ratio module assigns dynamic weights ω to each feature modality. k The calculation formula is as follows:

[0133]

[0134] weight ω k The value range is [0,1], and the sum of all modal weights is 1, ensuring the rationality and stability of the fusion process.

[0135] Significance factor α k (x)=exp(λ k Var Ω(x) (P k Var is used to measure the response strength of mode k within a local window Ω(x). Ω(x) The variance of the marginal probability map quantifies the significance of the marginal response of this mode in the local region, and the mode correlation gain coefficient λ. k Control the degree of influence of significance factors on weights.

[0136] By using an exponential function, the significance factor effectively amplifies the differences in response intensity of different modes in local regions, allowing regions with significant margins and modes with high variance to receive higher weights, thereby enhancing their contribution to the fusion results.

[0137] β k (x)=1-exp(-γ k μk (x)) is the reliability factor for the noise level of the mode of interest, μ k (x) represents the signal-to-noise ratio of the mode in the local region, γ k β is the modal attenuation coefficient. When the signal-to-noise ratio of the modes is low, β k Decreasing the value of (x) reduces the weight of that mode in the fusion, effectively suppressing noise interference, and vice versa.

[0138] In the example image of this embodiment, ω is set to... gra =0.52, ω dep =0.17, ω cur =0.11, ω nor =0.20.

[0139] Step S3 involves targeted screening, continuity detection, connectivity detection, convex hull construction, and labeling of the total edge probability map to obtain the skin detachment damage area.

[0140] Probabilistic graph growth clustering is used to obtain multiple edge sets, and regions are divided. Continuity and connectivity filtering are applied to obtain multiple valid sets of pixel edges for damaged regions. After constructing the convex hull, all pixels within the convex hull constitute the set of all pixels in the damaged region.

[0141] The first step is to selectively filter the total marginal probability map, removing clusters of non-marginal regions and noisy regions, thus obtaining the clustered total marginal probability map.

[0142] The second step is to perform continuity detection on the total edge probability map of the cluster. Continuity is defined as the sum of the spatial distances between each point and its three neighboring points. If the sum is within a set threshold, a continuous total edge probability map is obtained.

[0143] The third step involves connectivity detection on the continuous total edge probability map. Connectivity is defined as the sum of the differences in gradient directions θ between each point and its three neighboring points. If the sum is within a set threshold, the resulting set of edge pixels for each damaged region is obtained (1, 2, ..., n). The gradient directions are then extracted from the gradient values ​​in the grayscale image.

[0144]

[0145] The fourth step involves using the gradient direction θ to perform non-maximum suppression on the pixel set at the edge of the damaged region, sharpening the edge, and constructing a two-dimensional convex hull for the sharpened edge region to obtain the pixel set 1,2...n of the damaged region.

[0146] Step 5: Label the pixel set of each damaged area. The skin peeling damage area is now delineated. See the final result. Figure 10 .

[0147] Step S4 involves using a data processing system to calculate the segmentation accuracy of the skin detachment damage area. After multiple tests, the accuracy reached 0.02 mm, enabling sub-millimeter-level segmentation of aircraft skin detachment damage.

[0148] This application aims to segment aircraft skin detachment damage using a proposed high-precision damage region segmentation method based on adaptive multimodal fusion. The resulting damage segmentation is more accurate and highly flexible, effectively supporting aircraft health monitoring, maintenance and operation, and aircraft design optimization. By improving the accuracy and efficiency of damage detection, it contributes to promoting the transformation of the aviation industry from "planned maintenance" to "predictive maintenance".

Claims

1. A method for segmenting damaged regions on the surface of an aircraft based on adaptive multimodal fusion, characterized in that, Includes the following steps: Step S1: Collect aircraft skin damage data: Use a surface structured light acquisition system to collect damage data on the damaged skin of the aircraft surface, and obtain grayscale map, depth map and point cloud map of the damaged skin of the aircraft surface. Step S2, extract the total edge probability map of skin damage: The first step is synthesis: using a high dynamic range imaging (HDR) synthesis algorithm, the RGB grayscale image, depth map, and point cloud image of the damaged skin on the aircraft surface are synthesized to obtain an HDR image; The second step is to preprocess the HDR image: gamma correction is performed on the synthesized grayscale image and the synthesized depth image respectively to obtain the preprocessed grayscale image and the preprocessed depth image. The synthesized point cloud map is voxelized to remove invalid parts of the coordinate data, resulting in an effective synthesized point cloud map. The geometric features of the effective synthesized point cloud map are extracted, and the least squares algorithm is used to smooth the effective synthesized point cloud map to obtain a preprocessed point cloud map. The third step is feature extraction: feature extraction is performed on the preprocessed grayscale image and the preprocessed depth image to obtain grayscale gradient features and depth gradient features. Feature extraction is performed on the preprocessed point cloud image to obtain curvature features and normal vector features; The fourth step is feature transformation: feature transformation is performed on gray-level gradient features, depth gradient features, curvature features and normal vector features respectively, that is, normalized to [0,1] and nonlinearized to obtain gray-level edge probability features, depth edge probability features, curvature edge probability features and normal vector edge probability features. Step 5: Image display and projection into two-dimensional space: The image displays grayscale edge probability features and depth edge probability features to obtain grayscale edge probability maps and depth edge probability maps; the curvature edge probability features and normal vector edge probability features are projected into two-dimensional space to obtain curvature edge probability maps and normal vector edge probability maps. Step 6, weighted fusion: Adaptive weighted fusion method is used to weightedly fuse the grayscale edge probability map, depth edge probability map, curvature edge probability map and normal vector edge probability map to obtain the total edge probability map; The grayscale edge probability map, depth edge probability map, curvature edge probability map, and normal vector edge probability map are weighted, and an adaptive fusion weight ratio module is introduced to complete the feature fusion process. The fusion formula is as follows: (10)； in, This is the total marginal probability map. This is a grayscale edge probability map. This is a depth edge probability map. This is a probability map of the curvature edge. This is the marginal probability map of the normal vector. As weight, ; The adaptive fusion weight ratio module assigns dynamic weights to each feature modality. The calculation formula is as follows: (11)； Each feature mode is assigned a dynamic weight. The value range is [0,1], and the sum of all modal weights is 1; Significance factor Used to measure all modes In local window The response intensity within; The variance of the marginal probability map quantifies the significance of the marginal response of this mode in a local region, and the mode correlation gain coefficient is used. Control the degree of influence of significance factors on weights; To focus on the reliability factor of the noise level of the mode, The signal-to-noise ratio of the mode in the local region. This is the modal attenuation coefficient. When the signal-to-noise ratio of the modes is low, Decreasing the value of reduces the weight of that mode in the fusion, effectively suppressing noise interference, and vice versa; Step S3, Region Segmentation: The total edge probability map is subjected to targeted screening, continuity detection, connectivity detection, convex hull construction, and labeling to delineate the skin detachment damage area.

2. The method for segmenting damaged areas on the surface of an aircraft according to claim 1, characterized in that, This includes step S4, calculating the accuracy of damage segmentation; A data processing system was used to calculate the segmentation accuracy of the skin detachment damage area.

3. The method for segmenting damaged areas on the surface of an aircraft according to claim 1, characterized in that, The structured light acquisition system includes a structured light camera and a data processing system.

4. The method for segmenting damaged areas on the surface of an aircraft according to claim 1, characterized in that, Step S2 further includes: In the first step: Select at least 5 RGB grayscale images, depth images and point cloud images with different exposure times, and use HDR synthesis algorithms to capture the details of the highlights, midtones and shadows respectively to obtain HDR images, namely synthesized grayscale images, synthesized depth images and synthesized point cloud images; In the third step: Feature extraction is performed on the preprocessed grayscale image and the preprocessed depth image. The Soble operator is used to extract the gradient value of each individual pixel in the x-direction. and in the y direction : (1)； (2)； in, These are the pixel values ​​of the input image, i.e., the pixel values ​​of the preprocessed grayscale image or the preprocessed depth image. For the pixel index in the x direction, For the pixel index in the y-direction, and These are the weight matrices of the Sobel operator in the horizontal and vertical directions, respectively, as follows: (3)； (4)； and It can capture the magnitude of pixel grayscale changes on a 3x3 grid in the x and y directions, and thus represent gradient changes; according to and The gradient magnitude G is extracted and calculated as follows: (5)； Feature extraction is performed on the preprocessed point cloud image, using operators to extract the curvature and normal vector features; the point cloud in the preprocessed point cloud image is defined as... Construct a kdtree to search for its neighborhood points; For each point Calculate its position vector relative to the target point p. : (6)； Calculate the position vector covariance matrix : (7)； For covariance matrix Perform eigenvalue decomposition to obtain eigenvalues. The relationship is , Curvature features of the preprocessed point cloud map Defined as: (8)； It is the smallest eigenvalue; Principal component analysis is used to obtain the normal vector characteristics of the preprocessed point cloud image; let the point cloud in the preprocessed point cloud image be... There exists any point on and its adjacent points their respective minimum eigenvalues The corresponding feature vector is , After normalizing the eigenvectors, the corresponding normal vector eigenvectors are obtained. , : (9)； In the fourth step: First, feature transformation is performed on the grayscale gradient features and depth gradient features, specifically as follows: Normalize to the range of [0,1], crop and process outliers to obtain gray-level gradient features and depth gradient features with low outliers; normalize the gray-level gradient features and depth gradient features with low outliers to the range of [0,1], and scale them non-linearly to obtain gray-level edge probability features and depth edge probability features. Secondly, feature transformation is performed on the curvature features to obtain curvature edge probability features; specifically: Normalize to the range [0,1], remove and process outliers to obtain curvature gradient features with low outliers; normalize the curvature gradient features with low outliers to the range [0,1], and perform nonlinear scaling to obtain curvature edge probability features; Finally, feature transformation is performed on the normal vector features to obtain the marginal probability features of the normal vector; specifically: For each point in the preprocessed point cloud map, calculate the sum of the angles between its normal vector and the normal vectors of its surrounding neighboring points; normalize to the range of [0,1], crop and process outliers to obtain the normal vector gradient features with low outliers; normalize the normal vector gradient features with low outliers to the range of [0,1], and nonlinearly scale to obtain the normal vector edge probability features. In step five: Using the intrinsic parameters of a surface structured light camera, the point cloud image is projected back into a two-dimensional depth map. Then, the positions of pixels in the two-dimensional depth map image are projected onto the point cloud image. The correspondence between points in the point cloud image and two-dimensional pixels in the depth map is found, resulting in the curvature edge probability map and the normal vector edge probability map.

5. The method for segmenting damaged areas on the surface of an aircraft according to claim 1, characterized in that, In step S2, the sixth step... Weights set to , , , .

6. The method for segmenting damaged areas on the surface of an aircraft according to claim 1, characterized in that, Step S3 further includes: The first step is to selectively filter and remove non-marginal region clusters and noisy region clusters from the total marginal probability map to obtain the total marginal probability map of the clusters. The second step is to perform continuity detection on the total edge probability map of the cluster, and define continuity as the sum of the spatial distances between each point and its three neighboring points within a set threshold, so as to obtain the continuous total edge probability map of each point. The third step involves performing connectivity detection on the continuous total edge probability map. Connectivity is defined as the sum of the differences in gradient directions between each point and its three neighboring points, ensuring that the summation value is within a set threshold. This yields the edge pixel set 1, 2...n for each damaged region. The gradient direction is then extracted from the gradient values ​​in the grayscale image. gradient direction The calculation formula is as follows: (12)； In the above formula, Gy is the gradient value of a single pixel in the y direction, and Gx is the gradient value of a single pixel in the x direction. Fourth step, construct the convex hull; utilize the gradient direction. Non-maximum suppression is applied to the pixel set at the edge of the damaged region to sharpen the edge. A two-dimensional convex hull is constructed for the sharpened edge region to obtain the pixel set 1,2...n of the damaged region. The fifth step is to label the pixel set of each damaged area to delineate the skin detachment damage area.

Images