Nuclear fuel assembly appearance micro-defect detection method based on multi-modal fusion and Transformer
By combining multimodal fusion and Transformer methods with defect-guided attention mechanism and multi-loss function optimization, the problem of insufficient accuracy and robustness in micro-defect detection of nuclear fuel assemblies is solved, and high-precision micro-defect segmentation and detection are achieved.
Patent Information
- Application Number
- CN202511636443.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-03-20
AI Technical Summary
Existing technologies for detecting micro-defects in nuclear fuel assemblies suffer from problems such as low detection accuracy, poor ability to identify weak defects at multiple scales, insufficient robustness to complex backgrounds, and unclear edge segmentation, making it difficult to meet the nuclear industry's requirements for high stability and high precision.
We employ a multimodal fusion and Transformer approach, combining a defect-guided attention mechanism and a multi-loss function optimization strategy. By fusing visible light, infrared, and polarization image information and utilizing Transformer to model the global context, we improve detection accuracy and robustness.
It achieves high-precision segmentation of micro-defects in the appearance of nuclear fuel assemblies, reduces the false detection and false negative rates, improves the robustness of the model and its ability to adapt to complex backgrounds, and is suitable for the detection tasks of various nuclear industry products.
Smart Images
Figure CN121707918A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of automated inspection and artificial intelligence visual recognition technology, specifically relating to a method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer. Background Technology
[0002] As a critical component of a nuclear reactor, the structural integrity and surface quality of nuclear fuel assemblies directly affect the safe operation and service life of the reactor. In particular, micro-defects such as tiny cracks, scratches, and weld abnormalities on the surface of fuel rods, though small in size, can lead to serious consequences such as corrosion, leakage, or breakage during long-term service. Therefore, high-precision, non-destructive testing of these assemblies is of great significance.
[0003] Currently, fuel rod defect detection largely relies on manual visual inspection or traditional image processing algorithms. However, micro-defects often exhibit low contrast, weak edges, and minute dimensions, making them easily obscured by complex backgrounds or uneven lighting. This poses significant challenges to the accuracy, reliability, and consistency of traditional methods, especially when detecting low-visibility defects such as microcracks or minor scratches. These methods frequently suffer from high rates of missed and false detections, heavy reliance on manual inspection, and low levels of automation, failing to meet the nuclear industry's demands for high-stability and high-precision detection.
[0004] In recent years, deep learning technologies, especially convolutional neural networks (CNNs) and visual Transformer models, have demonstrated powerful capabilities in complex pattern recognition and image semantic segmentation tasks. Based on this, industrial visual inspection solutions have begun to be applied in the nuclear fuel field. Although existing research has attempted to classify or segment defects using single-modal images, it still has many shortcomings when facing the high-precision task of detecting micro-defects in nuclear fuel rods, such as blurred boundaries, poor response to weak feature regions, and weak model generalization ability.
[0005] Micro-defects on the surface of nuclear fuel rod assemblies exhibit complex characteristics such as diversity, non-uniformity, and large scale differences, making it difficult to comprehensively characterize them using single-modal information. Therefore, there is an urgent need to develop a deep learning method that integrates multimodal image information and possesses stronger feature modeling and detail segmentation capabilities to improve the automation and intelligence of micro-defect detection, achieving a high-precision, full-coverage, and low-error intelligent detection system. Summary of the Invention
[0006] The purpose of this invention is to provide a method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer. By fusing multimodal image information, using Transformer to model the global context, and combining a defect-guided attention mechanism and a multi-loss function optimization strategy, this method effectively solves the problems of low detection accuracy, poor ability to identify weak defects at multiple scales, insufficient robustness to complex backgrounds, and unclear edge segmentation in existing methods for detecting micro-defects in the appearance of nuclear fuel assemblies. This improves detection accuracy and robustness, and achieves high-precision segmentation of defect areas.
[0007] The technical solution of the present invention is as follows: A method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer, comprising the following steps:
[0008] Step 1: Data preparation and model training, including:
[0009] Step 11: Multimodal image acquisition;
[0010] Step 12: Model building;
[0011] Design a Transformer detection network with a multimodal fusion structure and a defect attention mechanism;
[0012] Step 13: Model training;
[0013] A multi-task loss function is used for joint training of the defect localization and classification model;
[0014] Step 2: Model inference and detection output, including:
[0015] Step 21: Input multimodal images, and use a multimodal feature extractor to extract features from different modal images to obtain multimodal feature maps;
[0016] Step 22: Input the multimodal features into the fusion module for feature alignment and fusion to generate a fused feature map;
[0017] Step 23: Extract deep semantic information through the defect-aware Transformer module and output the defect location, type, and severity score.
[0018] In step 11, multimodal images of nuclear fuel assemblies are acquired, including visible light images, infrared images, and polarization images, and defective regions are marked on the defective images.
[0019] Step 23 includes a defect attention step, which generates a defect heatmap through a defect-guided attention mechanism and performs element-wise weighting with the fused feature map to obtain a defect-perceived feature map.
[0020] Step 23 includes a defect detection step, in which the defect-aware feature map is input into the backbone network based on the Transformer architecture, global context features are extracted, and the segmentation result map of the defect region is output through the segmentation module.
[0021] Step 23 includes a multi-loss optimization step, which constructs a total loss function based on multiple loss functions such as binary cross-entropy loss, Dice loss, and edge-aware loss, and optimizes the model parameters.
[0022] In the feature extraction step, the HRNet network is used to extract feature information of different modalities.
[0023] In the feature fusion step, after extracting the corresponding modal feature maps from the multimodal images, including visible light images, infrared images, and polarization images, using HRNet, the obtained multimodal feature maps are input into the attention-guided fusion module. This module includes the following processing steps: Channel attention mechanism: Calculate the channel importance weights for the feature map of each modality, and extract channel description information using a combination of global average pooling and max pooling. Use fully connected layers and activation functions to generate channel attention weights to enhance the expressive power of key channels; Spatial attention mechanism: Pool the fused preliminary feature map along the channel dimension to generate a spatial attention map, which is used to strengthen the response of key spatial regions in the feature map and suppress irrelevant regions; Intermodal fusion: Superimpose the features of each modality and perform a 1x1 convolution to obtain a unified feature representation after fusion, denoted as the fused feature map.
[0024] In the defect attention step, a visible light image is first used as input and passed through a lightweight auxiliary saliency detection network to extract a preliminary defect heatmap, which represents the probability that each pixel region is a potential defect. Then, this defect heatmap is used as a spatial weight map and is multiplied element-wise with the fusion feature map to enhance the features of potentially defective regions and suppress the features of background regions. Finally, a defect-aware feature map is obtained, which is semantically more focused on the defect region and provides more discriminative input features for the subsequent Transformer backbone.
[0025] In the defect detection step, the defect-aware feature map is input into the backbone network based on the Swing Transformer. First, patch embedding is performed through 4×4 convolution and layer normalization to divide the original feature map into non-overlapping regions of fixed size. Then, it passes through the Transformer encoding module with window multi-head self-attention mechanism in multiple stages to extract local and global contextual semantic information layer by layer. The number of channels in each stage is 96, 192, 384 and 768 respectively. Then, the final output high-dimensional semantic feature map is fed into the simplified UNet decoding structure. The feature map is restored to the original image spatial scale through four bilinear upsampling and 3×3 convolution operations. A single-channel defect segmentation mask map is output through 1×1 convolution. After sigmoid activation, the final defect probability map is obtained, where each pixel value represents the confidence that it is a defect region.
[0026] In the multi-loss optimization step, multiple loss functions tailored to the characteristics of the segmentation task are used to jointly supervise the network, improving the model's ability to perceive different types of defective regions and the segmentation accuracy. These include: binary cross-entropy loss to measure pixel-level classification error, Dice loss to enhance the overall region matching degree, and edge-aware loss to emphasize the accuracy of boundary structures. The edge-aware loss calculates the difference between edges by applying the Sobel edge operator to both the predicted segmentation map and the ground truth mask map to highlight detailed contour information. These loss functions are weighted to form the total loss function, denoted as L. total =λ1·Lbce+λ2·L dice +λ3·L edge By continuously adjusting the model parameters through backpropagation, the network can accurately segment defective regions while possessing good edge recognition capabilities and overall robustness.
[0027] The beneficial effects of this invention are as follows:
[0028] (1) Using multimodal image input improves the ability to identify low-contrast and weak-edge defects and enhances the perceptual robustness of the model.
[0029] (2) Introduce an attention-guided fusion mechanism and a defect-guided attention module to enhance the features of key areas and avoid background interference;
[0030] (3) The backbone network based on Transformer has global modeling capabilities, which improves the contextual understanding of defect areas and is suitable for complex textures and multi-scale defect scenes.
[0031] (4) The multi-loss function collaborative optimization method can improve the edge segmentation quality while maintaining pixel-level accuracy, and significantly reduce the false detection and false detection rates.
[0032] (5) The whole method has good scalability and deployment flexibility, and can be applied to the defect detection tasks of various nuclear industry products such as nuclear fuel rods and fuel assembly grids. It has high practical value and engineering promotion potential. Attached Figure Description
[0033] Figure 1 The flowchart shows a method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer.
[0034] Figure 2 This is a schematic diagram of the network structure for a method of detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer.
[0035] Figure 3 This is a schematic diagram of the network structure for the multimodal fusion step;
[0036] Figure 4 This is a schematic diagram of the network structure of the defect attention guidance module. Detailed Implementation
[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0038] A method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer includes the following:
[0039] (1) Image acquisition steps: Acquire multimodal images of nuclear fuel assemblies, including visible light images, infrared images and polarization images, to obtain more comprehensive surface information;
[0040] (2) Feature extraction step: Use the lightweight convolutional neural network HRNet to extract features from each modality image to obtain the intermediate feature map of the corresponding modality;
[0041] (3) Feature fusion step: Input the multimodal feature map into the attention-guided fusion module. The fusion module includes channel attention mechanism and spatial attention mechanism, which respectively enhance the importance of different channels and spatial regions. The size and semantics of the features of different modalities are aligned to finally generate the fused feature map.
[0042] (4) Defect attention step: Input the visible light image into the defect-guided attention module to generate a defect heat map and weight it element by element with the original fused feature map to obtain a defect perception feature map, thereby enhancing the features of potential defect areas;
[0043] (5) Defect detection steps: Input the defect-aware feature map into the backbone network based on the Swin Transformer architecture, extract global contextual semantic features through the window self-attention mechanism, and then input it into the lightweight UNet decoding module for progressive upsampling, outputting a defect segmentation mask map with the same size as the input image.
[0044] (6) Multi-loss optimization steps: Binary cross-entropy loss, Dice loss and edge-aware loss are used to form a joint loss function to supervise and optimize the network, thereby improving the model’s performance in accurately locating defect boundaries, dealing with complex background interference and handling imbalanced samples.
[0045] The key point of this invention is:
[0046] (1) Innovatively integrate visible light, infrared and polarization images to obtain multimodal feature information and enhance the ability to perceive low contrast and weak edge defects.
[0047] (2) Combining the attention-guided fusion module and the defect-guided attention module enhances the importance of different channels and spatial regions, while also enhancing the characteristics of potential defect regions and reducing background interference.
[0048] (3) The backbone network based on the Swin Transformer architecture is combined with the lightweight UNet decoding module to achieve global context semantic feature extraction and accurate defect segmentation.
[0049] (4) A joint loss function is constructed by using binary cross-entropy loss, Dice loss and edge-aware loss to improve the model’s performance in accurately locating defect boundaries, dealing with complex background interference and handling imbalanced samples.
[0050] A method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer includes the following steps:
[0051] Step 1: Data preparation and model training;
[0052] Step 11: Multimodal image acquisition, acquiring visible light images, infrared images, and polarization images of nuclear fuel assemblies, and marking defect areas in defective images;
[0053] Step 12: Model building, designing a Transformer detection network with a multimodal fusion structure and a defect attention mechanism;
[0054] Step 13: Model training, using a multi-task loss function to jointly train the defect localization and classification models;
[0055] Step 2: Model inference and detection output;
[0056] Step 21: Input multimodal images and extract features using their respective feature extractors;
[0057] Step 22: Input the multimodal features into the fusion module for feature alignment and fusion;
[0058] Step 23: Extract deep semantic information through the defect-aware Transformer module and output the defect location, type, and severity score.
[0059] The image acquisition step includes: acquiring multimodal images of nuclear fuel assemblies, including visible light images, infrared images, and polarization images;
[0060] Feature extraction steps: Use a multimodal feature extractor to extract features from images of different modalities to obtain multimodal feature maps;
[0061] Feature fusion step: The multimodal feature maps are fused using an attention-guided fusion module to generate a fused feature map;
[0062] Defect attention step: A defect heatmap is generated through a defect-guided attention mechanism and then weighted element-wise with the fused feature map to obtain a defect-perceived feature map;
[0063] Defect detection steps: Input the defect-aware feature map into the backbone network based on the Transformer architecture, extract global context features, and output the segmentation result map of the defect region through the segmentation module;
[0064] Multi-loss optimization steps: Construct a total loss function by jointly using multiple loss functions, including binary cross-entropy loss, Dice loss, and edge-aware loss, and optimize the model parameters.
[0065] In the image acquisition step, a multimodal imaging system is used to acquire images synchronously to ensure the spatial and temporal alignment of images of different modalities.
[0066] In the feature extraction step, the HRNet network is used to extract feature information of different modal data. HRNet can learn high-resolution feature representations and can adapt to defect features of different sizes, thus improving the extraction effect of defect features at different scales.
[0067] In the feature fusion step, after extracting the feature maps of the corresponding modalities from the multimodal images (including visible light images, infrared images, and polarization images) using HRNet (High-Resolution Net), the obtained multimodal feature maps are input into the attention-guided fusion module. This module includes the following processing steps: Channel Attention: Calculate the channel importance weights for each modal feature map, using a combination of global average pooling and max pooling to extract channel description information, and use fully connected layers and activation functions to generate channel attention weights, enhancing the expressive power of key channels; Spatial Attention: Pool the fused preliminary feature map along the channel dimension to generate a spatial attention map, which is used to strengthen the response of key spatial regions in the feature map and suppress irrelevant regions; Intermodal Fusion: Superimpose the features of each modality and perform a 1x1 convolution to obtain a unified feature representation after fusion, denoted as the Fusion Feature Map.
[0068] In the defect attention step, a visible light image is first used as input and passed through a lightweight auxiliary saliency detection network to extract a preliminary defect heatmap, which represents the probability that each pixel region is a potential defect. Then, this defect heatmap is used as a spatial weight map and the fusion feature map to perform element-wise multiplication, which enhances the features of potentially defective regions and suppresses the features of background regions. Finally, a defect-aware feature map is obtained, which is more semantically focused on the defect region and provides more discriminative input features for the subsequent Transformer backbone.
[0069] In the defect detection step, the defect-aware feature map is input into the backbone network based on the Swing Transformer. First, patch embedding is performed through 4×4 convolution and layer normalization to divide the original feature map into non-overlapping regions of fixed size. Then, it passes through the Transformer encoding module with window multi-head self-attention mechanism in multiple stages to extract local and global contextual semantic information layer by layer. The number of channels in each stage is 96, 192, 384 and 768 respectively. Then, the final output high-dimensional semantic feature map is fed into the simplified UNet decoding structure. The feature map is restored to the original image spatial scale through four bilinear upsampling and 3×3 convolution operations. A single-channel defect segmentation mask map is output through 1×1 convolution. After sigmoid activation, the final defect probability map is obtained, where each pixel value represents the confidence that it is a defect region.
[0070] In the multi-loss optimization step, multiple loss functions tailored to the characteristics of the segmentation task are used to jointly supervise the network, thereby improving the model's ability to perceive different types of defect regions and the segmentation accuracy. These include: Binary Cross-Entropy Loss to measure pixel-level classification error, Dice Loss to enhance the overall region matching degree, and Edge-aware Loss to emphasize the accuracy of boundary structures. The Edge-aware Loss calculates the difference between edges by applying the Sobel edge operator to both the predicted segmentation map and the ground truth mask map to highlight detailed contour information. These loss functions are weighted to form the total loss function, denoted as L. total =λ1·L bce +λ2·L dice +λ3·L edge , where L total L represents the overall loss function; bce L represents the binary cross-entropy loss, used to measure pixel-level classification error; dice Dice loss is used to improve the overlap between the predicted region and the actual defect region; L edge λ1 represents the edge-aware loss, used to enhance the accuracy of the boundary structure; λ1, λ2, and λ3 are loss weight coefficients, used to balance the contribution ratio of the three losses in the overall optimization process.
[0071] By continuously adjusting the model parameters through backpropagation, the network can accurately segment defective regions while possessing good edge recognition capabilities and overall robustness.
[0072] By continuously adjusting the model parameters through backpropagation, the network can accurately segment defective regions while possessing good edge recognition capabilities and overall robustness.
[0073] This invention can effectively improve the detection accuracy and robustness of surface defects in nuclear fuel assemblies, has strong generalization ability, and can be flexibly adapted to various industrial vision systems according to actual detection needs, so as to achieve efficient and automated detection in complex industrial scenarios.
[0074] Example 1:
[0075] This embodiment provides a deep learning-based method for detecting micro-defects in the appearance of nuclear fuel assemblies. This method can be implemented in a defect detection system equipped with multimodal image acquisition equipment and a GPU-accelerated computing environment. Specifically, it includes the following steps:
[0076] (1) Model training phase
[0077] Step 1: Image Acquisition
[0078] A multimodal vision system was used to image the appearance of nuclear fuel assemblies, acquiring visible light, infrared, and polarized images. The images from the three modalities were aligned using a spatial and temporal synchronization mechanism and uniformly adjusted to a fixed size for use as network input.
[0079] Step 2: Modal Feature Extraction
[0080] The three modal images are respectively input into the corresponding lightweight convolutional neural network HRNet, and the intermediate layer semantic features of each image are extracted. The number of output channels is uniformly 256, forming three modal feature maps.
[0081] Step 3: Feature Fusion
[0082] The three modal feature maps are fed into an attention-guided fusion module. This module includes channel attention and spatial attention mechanisms. Channel attention enhances the complementary information between different modal channels, while spatial attention highlights key regional features. The fused feature map outputs a unified fusion feature map with dimensions maintained at [B, 256, H, W], where B is the batch size, H is the feature height, and W is the feature width.
[0083] Step 4: Defect Attention Guidance
[0084] The fused feature map is input into a lightweight defect attention module, which simultaneously inputs an RGB image into a lightweight saliency detection model to generate a defect heatmap. The heatmap is then weighted pixel-wise with the fused feature map to obtain a defect-aware feature map, thereby enhancing the network's ability to perceive potential defect regions.
[0085] Step 5: Transformer backbone encoding
[0086] The defect-aware feature map is fed into a backbone network built on the Swin Transformer. The network employs a four-stage window attention mechanism for global modeling, with 96, 192, 384, and 768 channels respectively, and uses a sliding window strategy to encode contextual information across regions. The backbone network outputs a high-semantic feature map with dimensions [B, 768, H / 32, W / 32].
[0087] Step 6: Segmentation, Decoding, and Output
[0088] The feature map output from the backbone network is input to the decoding module of the simplified UNet structure. The spatial size is restored by four bilinear upsampling and 3×3 convolutional layers. The number of channels is reduced to 512, 256, 128 and 64 respectively. Finally, a single-channel segmentation mask is output through 1×1 convolution, which represents the probability that each pixel is a defect.
[0089] Step 7: Optimization using multiple loss functions
[0090] During the model training phase, constructing the total loss function consists of three parts:
[0091] 1) Binary Cross Entropy Loss, used for pixel-level classification;
[0092] 2) Dice loss, used to measure the degree of overlap between the predicted region and the actual defect region;
[0093] 3) Edge-aware loss: After extracting edges using the Sobel operator, the L1 difference between the prediction and the annotation is calculated to enhance boundary clarity.
[0094] The total loss function is expressed as follows:
[0095]
[0096] λ1, λ2, and λ3 are hyperparameters, which can be set to 1.0, 1.0, and 0.5 in the experiment.
[0097] (2) Reasoning and Deployment Phase
[0098] Step 1: Model Loading and System Initialization
[0099] Load the trained weight file into the deployment environment and initialize the model structure, image preprocessing module, and defect visualization interface to ensure the system has real-time response capabilities.
[0100] Step 2: Image Acquisition and Preprocessing
[0101] Multi-modal cameras acquire multi-channel image data of the nuclear fuel assembly under test in real time, and perform synchronous registration and standardization processing to ensure consistency with the input during training.
[0102] Step 3: End-to-end inference process
[0103] The preprocessed image is then sequentially processed through modality feature extraction, attention fusion, defect guidance, Transformer backbone encoding and decoder recovery, ultimately generating a defect probability mask.
[0104] Step 4: Defect Region Analysis and Structured Extraction
[0105] The output defect probability map is subjected to thresholding and connected component analysis to extract key parameters such as defect contour, location, area and boundary coordinates, and confidence level can be output according to different thresholds.
[0106] Step 5: Automatic generation and output of test reports
[0107] The system automatically generates defect visualization images and structured data reports, including: defect number, image location, area size, and defect type prediction label (such as scratch, crack, etc.). It supports exporting to PDF or uploading to the quality management system.
[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer, characterized in that, Includes the following steps: Step 1: Data preparation and model training, including: Step 11: Multimodal image acquisition; Step 12: Model building; Design a Transformer detection network with a multimodal fusion structure and a defect attention mechanism; Step 13: Model training; A multi-task loss function is used for joint training of the defect localization and classification model; Step 2: Model inference and detection output, including: Step 21: Input multimodal images, and use a multimodal feature extractor to extract features from different modal images to obtain multimodal feature maps; Step 22: Input the multimodal features into the fusion module for feature alignment and fusion to generate a fused feature map; Step 23: Extract deep semantic information through the defect-aware Transformer module and output the defect location, type, and severity score.
2. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 1, characterized in that: In step 11, multimodal images of nuclear fuel assemblies are acquired, including visible light images, infrared images, and polarization images, and defective regions are marked on the defective images.
3. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 1, characterized in that: Step 23 includes a defect attention step, which generates a defect heatmap through a defect-guided attention mechanism and performs element-wise weighting with the fused feature map to obtain a defect-perceived feature map.
4. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 3, characterized in that: Step 23 includes a defect detection step, in which the defect-aware feature map is input into the backbone network based on the Transformer architecture, global context features are extracted, and the segmentation result map of the defect region is output through the segmentation module.
5. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 4, characterized in that: Step 23 includes a multi-loss optimization step, which constructs a total loss function based on multiple loss functions such as binary cross-entropy loss, Dice loss, and edge-aware loss, and optimizes the model parameters.
6. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 1, characterized in that: In the feature extraction step, the HRNet network is used to extract feature information of different modalities.
7. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 1, characterized in that: In the feature fusion step, after extracting the corresponding modal feature maps from the multimodal images, including visible light images, infrared images, and polarization images, using HRNet, the obtained multimodal feature maps are input into the attention-guided fusion module. This module includes the following processing steps: Channel attention mechanism: Calculate the channel importance weights for the feature map of each modality, and extract channel description information using a combination of global average pooling and max pooling. Use fully connected layers and activation functions to generate channel attention weights to enhance the expressive power of key channels; Spatial attention mechanism: Pool the fused preliminary feature map along the channel dimension to generate a spatial attention map, which is used to strengthen the response of key spatial regions in the feature map and suppress irrelevant regions; Intermodal fusion: Superimpose the features of each modality and perform a 1x1 convolution to obtain a unified feature representation after fusion, denoted as the fused feature map.
8. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 1, characterized in that: In the defect attention step, a visible light image is first used as input and passed through a lightweight auxiliary saliency detection network to extract a preliminary defect heatmap, which represents the probability that each pixel region is a potential defect. Then, this defect heatmap is used as a spatial weight map and is multiplied element-wise with the fusion feature map to enhance the features of potentially defective regions and suppress the features of background regions. Finally, a defect-aware feature map is obtained, which is semantically more focused on the defect region and provides more discriminative input features for the subsequent Transformer backbone.
9. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 1, characterized in that: In the defect detection step, the defect-aware feature map is input into the backbone network based on SwinTransformer. First, patch embedding is performed through 4×4 convolution and layer normalization to divide the original feature map into non-overlapping regions of fixed size. Then, it passes through the Transformer encoding module with window multi-head self-attention mechanism in multiple stages to extract local and global contextual semantic information layer by layer. The number of channels in each stage is 96, 192, 384 and 768 respectively. Then, the final output high-dimensional semantic feature map is fed into the simplified UNet decoding structure. The feature map is restored to the original image spatial scale through four bilinear upsampling and 3×3 convolution operations. A single-channel defect segmentation mask map is output through 1×1 convolution. After sigmoid activation, the final defect probability map is obtained, where each pixel value represents the confidence that it is a defect region.
10. The method for detecting micro-defects in the appearance of nuclear fuel assemblies based on multimodal fusion and Transformer as described in claim 1, characterized in that: In the multi-loss optimization step, multiple loss functions tailored to the characteristics of the segmentation task are used to jointly supervise the network, improving the model's ability to perceive different types of defective regions and the segmentation accuracy. These include: binary cross-entropy loss to measure pixel-level classification error, Dice loss to enhance the overall region matching degree, and edge-aware loss to emphasize the accuracy of boundary structures. The edge-aware loss calculates the difference between edges by applying the Sobel edge operator to both the predicted segmentation map and the ground truth mask map to highlight detailed contour information. These loss functions are weighted to form the total loss function, denoted as L. total =λ1·L bce +λ2·L dice +λ3·L edge By continuously adjusting the model parameters through backpropagation, the network can accurately segment defective regions while possessing good edge recognition capabilities and overall robustness.