Stealth wave-absorbing material defect segmentation method based on multi-source image fusion

Abstract

The invention provides a stealth wave-absorbing material defect segmentation method based on multi-source image fusion, and belongs to the technical field of stealth wave-absorbing material defect identification. Visible light and infrared images of a defect area are synchronously acquired through an industrial camera and an infrared thermal imaging device, after image registration and data enhancement, features are extracted by using parallel ResNet18, and the defect of the defect area is identified by using a multi-source image fusion method. And cross-modal feature fusion is realized based on a conditional generative adversarial network, and finally, a DeepLabv3 + network is adopted to complete pixel-level segmentation and classification of defects. According to the invention, texture and temperature information are effectively fused, accurate identification of four defects of broken holes, corrosion, perforation and scratches is realized, and the detection efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention belongs to the field of stealth absorbing material defect identification technology, specifically involving a method for segmenting stealth absorbing material defects based on multi-source image fusion. Background Technology

[0002] Stealth aircraft employ methods such as shape stealth, material stealth technology, and electronic jamming to reduce their radar reflection, thereby evading detection systems. Among these, material stealth technology is widely used due to its convenience, typically achieved by applying a stealthy radar-absorbing coating to the aircraft's surface. Since stealth radar-absorbing materials are only one layer, often applied across the entire aircraft surface, covering a large area, even a tiny scratch can significantly impact their performance. Therefore, ensuring the integrity of the radar-absorbing material is of paramount importance.

[0003] Currently, traditional methods for identifying defects in stealth absorbing materials include visual inspection, eddy current analysis, ultrasonic testing, and penetration testing. The primary method currently used is visual inspection of the absorbing material surface by ground personnel. However, this method is labor-intensive and prone to significant errors due to human eye fatigue and limited individual perception, resulting in extremely low efficiency. Penetration testing causes significant contamination of the object, making it difficult to clean. Eddy current analysis detects defects in the absorbing material through eddy current fields; the presence of defects causes local disturbances in the eddy current field, changing the impedance of the detection coil. However, this technique cannot detect non-ferromagnetic objects and polymers, presenting certain limitations. Ultrasonic testing identifies defects in absorbing materials using ultrasonic waves. It utilizes the propagation characteristics of ultrasound between objects, emitting ultrasonic waves towards the object and determining the presence of defects based on the echo. This method is a contact-based identification method and suffers from low inspection efficiency.

[0004] While eddy current and ultrasonic methods can identify defects on the surface of aircraft stealth absorbing materials, they all require contact with the aircraft surface, which can easily damage the material. Furthermore, they can only detect the approximate location of defects, not their precise segmentation. Therefore, deep learning-based defect identification methods for aircraft stealth absorbing materials have attracted increasing attention due to their speed, accuracy, non-contact capability, and ability to accurately segment defects. However, most existing deep learning-based methods only perform target detection at the defect location. Further research is needed on how to effectively integrate feature information from different modalities to improve recognition accuracy and achieve precise defect segmentation. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for defect segmentation of stealth absorbing materials based on multi-source image fusion. It employs infrared thermal imaging and visible light image acquisition techniques to obtain visible light and infrared images of the stealth absorbing material, respectively. Then, a feature fusion network based on a conditional generative adversarial network (GAN) is constructed to integrate the visible light and infrared images. Simultaneously, complementary feature information from the visible light and infrared images of the aircraft's stealth absorbing material is extracted, and a feature-fused image is generated. Finally, a DeepLabv3+ segmentation network is constructed to efficiently identify, accurately segment, and classify defect locations in the fused image of the stealth absorbing material. This invention effectively improves segmentation accuracy by fusing feature information from images of different modalities.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for segmenting defects in stealth absorbing materials based on multi-source image fusion includes the following steps:

[0008] S10. Create four types of defects on the surface of a flat metal sample coated with a microwave absorbing coating: holes, corrosion, perforation, and scratches. Collect visible light images of the defect areas on the surface of the flat metal sample, and then use a light-excited infrared thermal imaging device to simultaneously collect infrared images of the same area.

[0009] S20. Align the visible light image with the infrared image in size, perform pseudo-color mapping on the infrared image after bilinear interpolation, and obtain a three-channel infrared image with the same resolution as the visible light image.

[0010] S30. Based on the scale-invariant feature transformation algorithm and the random sample consensus algorithm, cross-modal registration of visible light images and infrared images is performed to form registered visible light images and infrared images;

[0011] S40. Simultaneously perform random rotation, cropping, scaling, brightness and contrast adjustment and enhancement on the registered visible light image and infrared image, then normalize and divide them into training set and test set.

[0012] S50. Use the parallel pre-trained ResNet18 feature extraction network to extract visible light feature maps and infrared feature maps respectively.

[0013] S60. After unifying the scale of the visible light feature map and the infrared feature map through the dimension transformation layer, input them into the conditional generative adversarial network. The generator network and the discriminator network are trained adversarially to output the fused feature map.

[0014] S70. Input the fused feature map into the DeepLabv3+ segmentation network. After encoding-decoding and pixel-level classification, a defect category mask is obtained, which realizes accurate segmentation of defects in the absorbing material.

[0015] Beneficial effects:

[0016] 1. This invention designs a multimodal feature fusion layer based on conditional generative adversarial network (CGAN). Through adversarial training between the generator network and the discriminator network, it achieves dynamic optimization fusion of texture detail information of visible light image and temperature distribution information of infrared image, effectively overcoming the problem of easy loss of infrared thermal information in traditional feature fusion methods.

[0017] 2. Compared with existing infrared thermal imaging detection technology, this invention retains its original advantages and can accurately segment the defect location and output the defect type. It realizes the accurate segmentation and classification of four typical defects: holes, corrosion, perforation and scratches, providing front-line maintenance personnel with intuitive defect type and location information. Attached Figure Description

[0018] Figure 1 This is a flowchart of the stealth absorbing material defect segmentation method based on multi-source image fusion according to the present invention;

[0019] Figure 2 This is a diagram of the feature extraction network structure.

[0020] Figure 3 A schematic diagram of the basic convolutional block (CRB) component;

[0021] Figure 4 This is a schematic diagram of the residual stage;

[0022] Figure 5 This is a schematic diagram of a multimodal feature fusion layer;

[0023] Figure 6 This is a schematic diagram of the Deeplabv3+ segmentation layer network framework. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] like Figure 1 As shown, the method for segmenting defects in stealth absorbing materials based on multi-source image fusion of the present invention includes the following steps:

[0026] S10: Several typical defects were artificially created on the surface of a flat metal material sample coated with a microwave absorbing coating, and visible light and infrared images of the defect areas of the microwave absorbing material were collected respectively.

[0027] S20: Align the visible light image and infrared image obtained in S10. The resolution of the visible light image is (640, 640, 3), and the resolution of the infrared image is (384, 288, 1). Adjust the resolution of the infrared image to (640, 640, 1) through bilinear interpolation, and perform pseudo-color mapping on the interpolated single-channel infrared image to generate a three-channel infrared image, retaining the temperature distribution information, so that its resolution becomes (640, 640, 3).

[0028] S30: Cross-modal spatial registration of the visible light and infrared images obtained in S20 is performed based on the Scale-invariant Feature Transform (SIFT) algorithm and the Random Sample Consensus (RANSAC) algorithm. The visible light image is used as the reference image, and the infrared image as the observation image. SIFT feature points and descriptors are extracted from both images. The Euclidean distance between descriptors is calculated, and possible matching points for the feature points of the observation image are found in the reference image based on the nearest neighbor ratio criterion. Then, the optimal transformation matrix is ​​estimated using the RANSAC algorithm and applied to the observation image, ultimately generating the registered visible light and infrared images.

[0029] S40: Perform image preprocessing on the visible light and infrared images obtained in S30. Simultaneously perform simple data augmentation on both images using the same random transformation parameters. Then, perform advanced augmentation operations using multimodal image pairs as basic units. After normalization, divide the data samples into training and test sets in an 8:2 ratio.

[0030] S50: Two parallel pre-trained ResNet18 networks are used as ResNet18 feature extraction networks. The visible light image and infrared image obtained in S40 are respectively fed into the parallel pre-trained ResNet18 feature extraction networks for feature extraction, and the corresponding feature maps are output. The feature map output by the visible light branch is V, and the feature map output by the infrared branch is I, such as... Figure 2 The diagram shown is a network structure diagram of ResNet18 feature extraction network;

[0031] S60: Construct a multimodal feature fusion layer based on a Conditional Generative Adversarial Network (CGAN), and train the discriminator network and generator network using sample data from visible light and infrared images. The trained multimodal feature fusion layer will derive a multimodal fused feature map that incorporates more target feature information based on the actual input visible light and infrared feature maps; for example... Figure 5The image shows the training model of the generative adversarial network in the multimodal feature fusion layer;

[0032] S70: Input the multimodal fused feature map output from the multimodal feature fusion layer into the DeepLabv3+ segmentation layer for image segmentation, and output the segmentation result, such as... Figure 6 The diagram shows the network framework of the DeepLabv3+ segmentation layer.

[0033] Furthermore, S10 specifically includes the following steps:

[0034] S11: Four types of defects, namely perforation, corrosion, scratches and holes, are artificially created on the surface of a flat metal sample coated with a microwave absorbing coating.

[0035] S12: Under stable ambient light conditions, use an industrial camera to acquire a high-resolution visible light image of the defect area of ​​the absorbing material from a distance of 1 meter from the center normal direction of the area under test;

[0036] S13: Set up a light-excited infrared thermal imaging device in the laboratory, using a halogen lamp as the excitation light source, and fix the infrared thermal imager on the light-excited infrared thermal imaging device and aim it at the area to be measured.

[0037] S14: Place the optically excited infrared thermal imaging device about 1 meter away from the center normal of the area to be measured; set the excitation current to 9A and use a 1Hz sinusoidal modulation signal to thermally excite the defect area of ​​the absorbing material.

[0038] S15: During and after thermal excitation, use an infrared thermal imager to collect infrared thermal image sequences of the defect area of ​​the absorbing material, and select the frame corresponding to the moment of maximum temperature difference from the infrared thermal image sequence as representative input;

[0039] S16: Simultaneously store the visible light and infrared images of the defective areas of the absorbing material.

[0040] Furthermore, S30 specifically includes the following steps:

[0041] S31: Keypoints and feature descriptors are extracted from the reference image and the observation image based on the SIFT algorithm, where the observation image represents an infrared image and the reference image represents a visible light image. Gaussian scale spaces are constructed for the reference image and the observation image, and keypoints at different scales are found in the reference image and the observation image using the difference of Gaussian function.

[0042] Constructing Gaussian scaling space functions As shown below:

[0043] = ;

[0044] in, Represents a two-dimensional Gaussian kernel function. , The input image represents the original image, while in this step, the reference image and the observation image represent the original image and the observation image, respectively. σ represents the scale parameter. This represents the coordinates of a pixel relative to the center of the kernel. This indicates a convolution operation.

[0045] Calculate the gradient direction and magnitude of pixels within the neighborhood of each keypoint, and assign a principal direction to each keypoint. Then, take a gradient centered on the keypoint. The neighborhood is divided into 16 groups. The sub-regions are calculated, and gradient direction histograms in 8 directions are calculated in each sub-region, finally outputting a 128-dimensional feature descriptor.

[0046] S32: Calculate the Euclidean distance between each feature descriptor in the observed image and all feature descriptors in the reference image. Compare the ratio of the two closest distances with a set threshold, preferably 0.8. If the ratio is greater than the threshold, exclude the matching points and obtain an initial set of matching point pairs between the reference image and the observed image.

[0047] S33: The optimal spatial transformation model from the observed image to the reference image coordinate system is estimated from the initial set of matching point pairs obtained from S32 based on the RANSAC algorithm.

[0048] The RANSAC algorithm mainly includes the following steps: First, randomly select 4 pairs of matching points from the initial set of matching points, and calculate a temporary homography matrix for these 4 pairs of matching points through a direct linear transformation. Through a temporary homography matrix All initial matching points are checked, and the reprojection error between the transformed coordinates of each infrared image point and its matching point is calculated. If the reprojection error is less than a threshold, the matching point is considered an interior point. Preferably, the threshold is set to 0.6. Preferably, this step is repeated 1000 times, and the temporary matrix with the highest number of interior points is finally selected. .

[0049] The least squares optimization algorithm is used, utilizing the temporary matrix of all matching point pairs considered as interior points. A re-estimation is performed to obtain the final optimal transformation matrix. .

[0050] The homography matrix is ​​calculated as follows:

[0051] ;

[0052] in, The coordinates of a point in an infrared image, ( , () represents the coordinates of the corresponding point in the original image, through Calculate the coordinates of the corresponding point in the visible light image coordinate system. H refers to the homography matrix to be solved. The scaling factor represents the homogeneous coordinates.

[0053] S34: The optimal transformation matrix obtained in S33 The image is applied to the original observation image, and finally a registration result aligned with the reference image space is generated, resulting in a registered image.

[0054] For each pixel coordinate in the final output registered image The inverse matrix of the optimal transformation matrix Calculate its corresponding source coordinates in the original image. The calculation formula is as follows:

[0055] ;

[0056] If the calculated source coordinates If the value is a floating-point number, a weighted average of the neighboring pixels is calculated using bilinear interpolation. This weighted average is then used to determine the pixel value at that location. Finally, the interpolated pixel value is assigned to the target location in the registered image. After traversing all target pixels in the observed image, the final registered infrared image is obtained.

[0057] Furthermore, S40 specifically includes the following steps:

[0058] S41: Perform simple data augmentation operations on visible light and infrared images simultaneously using the same random parameters, including random rotation, random cropping and random scaling, while adjusting the brightness, contrast and saturation of the visible light image.

[0059] S42: Perform advanced enhancement operations using multimodal image pairs as basic units, including mosaic enhancement and mixup enhancement.

[0060] S43: After data augmentation, normalization is performed, and then the data samples are divided into training and test sets in an 8:2 ratio.

[0061] Furthermore, in S50, such as Figure 2 As shown, the ResNet18 feature extraction network consists of one initial convolutional layer, one max pooling layer, and four residual stages (i.e., ...). Figure 2 The residual stages 1 through 4). The initial convolutional layer consists of basic convolutional blocks (CRBs), and a schematic diagram of the CRB components is shown below. Figure 3 As shown, it includes convolutional layers, batch normalization layers, and the ReLU activation function.

[0062] Each residual stage consists of several stacked residual blocks. The residual blocks adopt a short-circuit connection design and contain the main path and shortcut paths, with the structure as follows: Figure 4 As shown. The main path consists of two CRB components (i.e. Figure 4 The shortcut path is constructed by concatenating CRB1 and CRB2. When the input and output dimensions match, an identity mapping is performed. When the dimensions do not match, 1×1 convolutions and batch normalization layers are used for adjustment. The outputs of the two paths are added together by tensors and then output after passing through the ReLU activation function.

[0063] Furthermore, the two branches of the parallel ResNet18 network will output extracted feature maps in the last residual module, and the feature map output by the visible light branch is defined as V, and the feature map output by the infrared branch is defined as I.

[0064] Further, the generation of the multimodal fusion feature map in S60 includes the following steps:

[0065] S61: As Figure 5 As shown, the multimodal feature fusion layer includes a dimension transformation layer, a stitching layer, a generator network, and a discriminator network. First, considering the different data dimensions of some visible light and infrared images, a dimension transformation layer is designed within the multimodal feature fusion layer. The visible light and infrared images output by the ResNet18 feature extraction network are first fed into the dimension transformation layer for processing, transforming them to the same dimension. Specifically, the visible light image, after processing by the dimension transformation layer, is weighted and passed to the stitching layer for fusion. Simultaneously, the dimension transformation layer performs dimension transformation on the infrared image, then performs two operations: one is weighted processing and passed to the stitching layer; the other is directly passed to the discriminator network as the feature input of the real samples, passing through 4 convolutional layers, linear layers, 4 batch normalization layers, and 4 activation functions for judgment.

[0066] S62: The stitching layer stitches the weighted visible light feature map and the infrared light feature map to generate a preliminary multimodal fusion feature map. The formula for generating the formula is:

[0067] ;

[0068] in, It is the output function of the dimension transformation layer. , These are learnable fusion weight parameters; the feature map output by the visible light branch is V, and the feature map output by the infrared branch is I.

[0069] S63: During the training phase, preliminary multimodal fusion feature maps will be generated. The generator network is enhanced by adding a Gaussian-distributed, same-dimensional noise tensor to the input perturbation. The noisy feature map is then fed into the generator network. The generator network G (comprising 5 convolutional layers, 4 batch normalization layers, and 4 activation functions) generates a fused feature map based on the noisy feature map, which is then used as a pseudo-sample input to the discriminator network D. The discriminator network receives two types of input simultaneously: the first is a feature map from the infrared light modality processed by a dimension transformation layer as the real sample input, and the second is the fused feature map generated by the generator network as the pseudo-sample input.

[0070] Furthermore, based on the game theory of Generative Adversarial Networks (CGANs), during adversarial training, the generator network attempts to generate a fused feature map that closely approximates the distribution of real infrared features, while the discriminator attempts to distinguish between real infrared features and the fused feature map generated by the generator. The goal is to make the generator output samples whose probability of judgment by the discriminator is close to 1, i.e., to mislead the discriminator into believing that the generated samples are real, thus achieving the goal of fully fusing the feature maps of the two modalities. The optimization objective function of this CGAN is as follows:

[0071] ;

[0072] in, Represents the mathematical expectation. Refers to the actual distribution of data. This refers to the noise distribution, where D is the discriminator, G is the generator, and z is the noise. The characteristic map I representing the output of the infrared branch follows a data distribution. The mathematical expectation, This indicates that the noise vector z follows a prior distribution. The mathematical expectation, This represents the adversarial objective function of the discriminator D and the generator G. This indicates that the discriminator is performing well in the initial multimodal fusion feature map. The probability of determining the input features. The generator is based on noise z and a preliminary multimodal fusion feature map. The generated fusion feature map.

[0073] After training, the generator network will be fixed. The output fused feature map N will be passed to the subsequent DeepLabv3+ segmentation layer for image segmentation.

[0074] Furthermore, such as Figure 6 As shown, the DeepLabv3+ segmentation layer adopts an encoder-decoder architecture. The encoder mainly includes a backbone network and a Spatial Pyramid Pooling (ASPP) module (i.e.,...). Figure 6 The deep convolutional neural network consists of two main parts: a backbone network and a deep convolutional neural network. The input image passes through the backbone network and produces two outputs. Low-level features are directly passed to the decoder after passing through a 1x1 convolutional layer, while high-level features are input to the dilated spatial pyramid pooling (ASPP) module for further processing. The ASPP module uses four dilated convolutional blocks with different dilation rates (i.e., 1×1 convolutional layer; 3×3 convolutional layer, dilation rate = 6; 3×3 convolutional layer, dilation rate = 12; 3×3 convolutional layer, dilation rate = 18) and a global average pooling block (i.e., a dilated convolutional block with dilation rate = 18). Figure 6 Image pooling is used to process the input high-level feature map, resulting in five sets of feature maps. These are then concatenated and passed through a 1×1 convolutional layer before being fed into the decoder. The decoder receives low-level features from the backbone network and the output from the ASPP module as input. First, a 1×1 convolutional layer is used to reduce the dimensionality of the low-dimensional features, and the high-dimensional features input from the ASPP module are upsampled. The two types of features are then concatenated. Finally, the concatenated feature information is processed through a 3×3 convolutional layer and an upsampling layer to obtain the final output image.

[0075] Example:

[0076] This embodiment proposes a deep learning-based multi-source data segmentation method for stealth absorbing materials to address defect segmentation. The specific process is as follows:

[0077] S10: Several typical defects are artificially created on the surface of a flat metal sample coated with a microwave absorbing coating, and visible light and infrared images of the defect areas of the sample are collected respectively.

[0078] In this embodiment, the experimental data consisted of flat metal samples coated with radar-absorbing and infrared-absorbing coatings provided by a research institute. The artificially created defects included holes, corrosion, perforations, and scratches. A total of 4000 images were obtained, including 2000 visible light images with a resolution of (640, 640, 3) and 2000 infrared images with a resolution of (384, 288, 1). Of the 2000 images in each mode, 1400 contained defects, and 600 were without defects.

[0079] S20: Align the infrared and visible light images obtained in S10. The resolution of the visible light image is (640, 640, 3), and the resolution of the infrared image is (384, 288, 1). Use OpenCV to perform bilinear interpolation to adjust the resolution of the infrared image to (640, 640, 1). Perform pseudo-color mapping on the interpolated single-channel infrared image to generate a three-channel infrared image, retaining the temperature distribution information, and making its resolution (640, 640, 3).

[0080] S30: Cross-modal spatial registration of the visible light and infrared images obtained in S20 is performed based on the Scale-invariant Feature Transform (SIFT) algorithm and the Random Sample Consensus (RANSAC) algorithm. The visible light image is used as the reference image, and the infrared image as the observation image. SIFT feature points and descriptors are extracted from both images. The Euclidean distance between descriptors is calculated, and possible matching points for the feature points of the observation image are found in the reference image based on the nearest neighbor ratio criterion. Then, the optimal transformation matrix is ​​estimated using the RANSAC algorithm and applied to the observation image, ultimately generating the registered visible light and infrared images.

[0081] S40: Image preprocessing is performed on the registered visible light and infrared images obtained in S30. Simple data augmentation operations are simultaneously performed on the visible light and infrared images using the same random transformation parameters. Then, advanced augmentation operations are performed using multimodal image pairs as basic units. After normalization, the data samples are divided into training and test sets in an 8:2 ratio.

[0082] In this embodiment, simple data augmentation of the samples is performed by random image rotation, random cropping, random scaling, and adjusting the image lighting conditions, contrast and saturation. Based on the above methods, Mosaic enhancement and Mixup enhancement algorithms are also used to enhance the data.

[0083] Mosaic data augmentation is performed on a multimodal image pair basis. Four registered visible-infrared image pairs are randomly selected, and the visible and infrared images in each pair are processed separately using the same spatial layout parameters. Finally, they are stitched together to generate a new visible image and a new infrared image, and their corresponding annotation masks are generated simultaneously. This process ensures that the newly generated multimodal image pairs maintain strict spatial alignment after augmentation.

[0084] Mixup is a simple yet effective image data augmentation method that generates new training samples by linearly interpolating different samples, thereby improving the model's generalization ability. Its core formula is:

[0085] ;

[0086] ;

[0087] Where i and j represent the indices of two randomly selected different training samples. This represents the image data of the i-th training sample. Let represent the label of the i-th training sample. This represents the image data of the j-th training sample. This represents the label of the j-th training sample. This represents a parameter. It can be understood as the pixel matrix of an image, carrying the visual information of the image, while These are the category labels corresponding to the images, explicitly indicating the category to which the image belongs. During the process of generating new samples, It is a key parameter, and its value is... The distribution can be flexibly controlled by adjusting its parameters. The range and probability density of the values ​​affect the generation characteristics of new samples. The Mixup data augmentation operation follows a multimodal synchronous mixing strategy. For two randomly selected image pairs, the same [data source] is used. Mixing coefficient of samples from the distribution Linear interpolation is performed on the visible light image and the infrared image respectively to generate new training sample pairs.

[0088] Finally, normalization is performed on the processed image. The pixel mean of the visible light image and the pseudo-color mapped infrared image are calculated separately. and standard deviation ,set up This represents the image pixels before adjustment. Represents the adjusted image pixels, using Normalize and adjust the brightness and contrast of the image:

[0089] ;

[0090] To approximate the standard normal distribution of image pixel values, where a and b represent the pixel coordinates, normalization reduces the training difficulty of the model, improves its generalization ability, and prevents gradient explosion. After normalization, the data samples are divided into training and test sets in an 8:2 ratio.

[0091] S50: Two parallel pre-trained ResNet18 networks are used as ResNet18 feature extraction networks. The visible light image and infrared image obtained in S40 are respectively fed into the parallel pre-trained ResNet18 networks for feature extraction, and the corresponding feature maps are output.

[0092] The minimum network structure component, CRB, contains convolutional layers, batch normalization layers, and a ReLU activation function. Each residual block consists of two network structure components (CRBs) and a tensor addition operation. The presence of these residual blocks allows the feature extraction network to be built deeper, thereby extracting deeper image features from the image.

[0093] Furthermore, the two branches of the parallel ResNet18 feature extraction network will output extracted feature maps in the last residual module. The feature map output by the visible light branch is defined as V, and the feature map output by the infrared branch is defined as I.

[0094] In this embodiment, one of the parallel ResNet18 feature extraction networks first takes a visible light image with dimensions of 640×640×3 as input. After passing through an initial CRB and max pooling layer, it outputs a 160×160×64 feature map. Stage 1 outputs a 160×160×64 feature map, Stage 2 outputs an 80×80×128 feature map, Stage 3 outputs a 40×40×256 feature map, and Stage 4 outputs a 20×20×512 feature map. The multimodal feature fusion layer described in this invention selects the last feature map as input and sets it to V. Similarly, another parallel ResNet18 feature extraction network also takes an infrared image with dimensions of 640×640×3 as input and goes through the same steps to obtain a 20×20×512 feature map, which is set to I.

[0095] S60: Construct a feature fusion layer based on a Conditional Generative Adversarial Network (CGAN), and train the discriminator and generator using sample data from visible light and infrared images. The trained multimodal feature fusion layer will then derive a multimodal fused feature map that incorporates more target feature information based on the actual input visible light and infrared feature maps. See also Figure 5 This is the training model for the generative adversarial network in the feature fusion layer.

[0096] S70: Input the fused feature map output by the multimodal feature fusion layer into the DeepLabv3+ segmentation layer to perform image segmentation tasks, and output the pixel-level segmentation results of the cloaking absorbing material defects in the fused feature map and the category to which the defects belong.

[0097] In this embodiment, the encoder uses a ResNet-50 pre-trained on the ImageNet dataset as the backbone network to extract multi-level features. This backbone network consists of five stages (Stage 1-Stage 5). During the forward propagation process, in addition to outputting the final feature map, a high-resolution, low-level semantic feature map is extracted from the intermediate stage (Stage 3 in this embodiment) and used as low-level feature input to the decoder.

[0098] The core of the encoder is the dilated spatial pyramid pooling module. This module applies multiple dilated convolutional layers with different dilation rates and a global average pooling layer in parallel on the feature map output by the backbone network to effectively capture contextual information of defects at different scales, such as "holes," "erosion," "perforations," and "scratches." The output feature maps of all parallel branches are concatenated and fused, and then passed through a 1×1 convolutional layer for channel dimensionality reduction and feature optimization to generate encoded high-level semantic features.

[0099] In the decoder section, the output of the ASPP module is first upsampled by a factor of 4. Simultaneously, the low-level features extracted from Stage 3 are passed through a 1×1 convolutional layer to adjust the number of channels and reduce the semantic gap between them and the high-level features. Subsequently, the upsampled high-level features are concatenated with the processed low-level features.

[0100] The concatenated fused feature map will be passed sequentially through two 3×3 convolutional layers for non-linear feature fusion and spatial detail optimization. Afterward, it will be upsampled by a factor of 4 to restore the feature map to the same spatial size as the original fused feature map.

[0101] Finally, the decoder output is passed through a 1×1 convolutional layer with 5 output channels to generate the original class value for each pixel. Then, the class probability distribution of each pixel is calculated through the Softmax activation function, thus forming the final segmentation mask.

[0102] Regarding the loss function, this embodiment employs a joint loss function of Cross-Entropy Loss and Dice Loss for end-to-end optimization. Cross-Entropy Loss focuses on optimizing the classification accuracy of each pixel; Dice Loss specifically addresses the class imbalance problem between defects and background, better optimizing the overall shape and contour integrity of the segmented target, and is particularly suitable for the accurate segmentation of small-scale defects such as scratches and perforations. During training, the Adam optimizer is used, with an initial learning rate set to 1×10⁻⁶. -4 It employs a multinomial learning rate decay strategy.

[0103] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A stealth metamaterial defect segmentation method based on multi-source image fusion, characterized in that, It comprises the following steps: S10, four types of defects of holes, corrosion, perforation and scratches are made on the surface of a flat metal sample coated with a wave-absorbing coating; a visible light image of a defect area on the surface of the flat metal sample is collected, and then an infrared image of the same area is synchronously collected by a light-excited infrared thermal imaging device; S20, the visible light image and the infrared image are size-aligned, the infrared image is bilinearly interpolated, and then pseudo-color mapping is performed on the infrared image to obtain a three-channel infrared image with the same resolution as the visible light image; S30, the visible light image and the infrared image are cross-modality registered based on a scale-invariant feature transformation algorithm and a random sample consensus algorithm to form a registered visible light image and an infrared image; S40, the registered visible light image and infrared image are synchronously subjected to random rotation, cropping, scaling, brightness contrast adjustment and enhancement, and then normalized and divided into a training set and a test set; S50, a visible light feature map and an infrared feature map are extracted by using a pre-trained ResNet18 feature extraction network in parallel; S60, the visible light feature map and the infrared feature map are input into a conditional generative adversarial network after being unified in scale by a dimension conversion layer, and a fusion feature map is output by adversarial training of a generator network and a discriminator network; S70, the fusion feature map is input into a DeepLabv3+ segmentation network to obtain a defect class mask through encoding-decoding and pixel-level classification, thereby realizing accurate segmentation of defects of the wave-absorbing material.

2. The defect segmentation method of the stealth wave-absorbing material based on multi-source image fusion according to claim 1, characterized in that, In the S10, under stable ambient light conditions, a high-resolution visible light image is collected from a center normal direction of a to-be-measured area at a distance of 1 meter by using an industrial camera; an infrared thermal imager is fixed on a light-excited infrared thermal imaging device and aimed at the to-be-measured area by using a halogen lamp as an excitation light source.

3. The defect segmentation method of the stealth wave-absorbing material based on multi-source image fusion according to claim 1, characterized in that, In the S20, the resolution of the infrared image is adjusted to 640×640×1 by bilinear interpolation, and then a three-channel infrared image is generated by pseudo-color mapping on the interpolated single-channel infrared image, so that the resolution of the infrared image becomes 640×640×3.

4. The defect segmentation method for stealthy wave-absorbing material based on multi-source image fusion according to claim 1, characterized in that, In the S30, key points and feature descriptors of the visible light image and the infrared image are extracted based on a scale-invariant feature transformation algorithm; the nearest neighbor distance ratio criterion is used to find matching points in the visible light image for the feature points in the infrared image by calculating the Euclidean distance between the feature descriptors, and initial matching point pairs are obtained; A temporary homography matrix is calculated from 4 matching points randomly extracted from the initial matching point pairs based on a random sample consensus algorithm, and the calculation is repeated several times to select a temporary homography matrix with the largest number of inliers as the final optimal transformation matrix.

5. The defect segmentation method of the stealth wave-absorbing material based on multi-source image fusion according to claim 4, characterized in that, The final optimal transformation matrix is obtained by re-estimating all inliers by a least squares optimization algorithm, and the optimal transformation matrix is applied to the infrared image to realize spatial registration.

6. The defect segmentation method of the stealth wave-absorbing material based on multi-source image fusion according to claim 1, characterized in that, In the S50, the pre-trained ResNet18 feature extraction network in parallel includes one initial convolutional layer, one maximum pooling layer and four residual stages, the initial convolutional layer is composed of basic convolutional blocks, each basic convolutional block includes a convolutional layer, a batch normalization layer and a ReLU activation function; each residual stage is stacked by a plurality of residual blocks, the residual block adopts a short circuit connection design, including a main path and a shortcut path, the main path is composed of two basic convolutional blocks in series, the shortcut path is identity mapping when the input and output dimensions match, and is adjusted by using a 1*1 convolutional layer and a batch normalization layer when the input and output dimensions do not match, the output of the temporary homography matrix is output through a tensor addition and a ReLU activation function.

7. The defect segmentation method of the stealth wave-absorbing material based on multi-source image fusion according to claim 1, characterized in that, In the S60, the dimension conversion layer performs dimension uniform processing on the visible light feature map and the infrared feature map extracted by the ResNet18 feature extraction network, and performs weighted processing on the converted visible light feature map and infrared feature map, respectively. The weighted visible light feature map and infrared feature map are spliced in the splicing layer to generate a preliminary multi-modal fusion feature map.

8. The defect segmentation method of the stealth wave-absorbing material based on multi-source image fusion according to claim 7, characterized in that, In the S60, the training process of the conditional generative adversarial network is as follows: the preliminary multi-modal fusion feature map and the same dimension noise of Gaussian distribution are input into the generator network after tensor addition, the generator network generates a fusion feature map as a pseudo sample according to the noise-added preliminary multi-modal fusion feature map and inputs it into the discriminator network; the discriminator network receives the infrared feature map processed by the dimension conversion layer as a real sample input, and the discriminator network receives the fusion feature map generated by the generator network as a pseudo sample input.

9. The defect segmentation method of the stealth wave-absorbing material based on multi-source image fusion according to claim 8, characterized in that, The generator network includes five convolutional layers, four batch normalization layers and four activation functions; the discriminator includes four convolutional layers, a linear layer, four batch normalization layers and four activation functions.

10. The defect segmentation method of stealthy and wave-absorbing material based on multi-source image fusion according to claim 6, characterized in that, In the S70, the encoder of the DeepLabv3+ segmentation network uses ResNet-50 as the main body, and introduces a skip connection in the third residual stage of the four residual stages, and the decoder performs multi-scale feature fusion through a dilated spatial pyramid pooling module and is up-sampled to the original image resolution; the output result of the DeepLabv3+ segmentation network is a hole, corrosion, perforation and scratch.

Images