Industrial product defect detection method based on multi-granularity feature fusion

Abstract

The invention discloses an industrial product defect detection method based on multi-granularity feature fusion, and the method comprises the steps: carrying out the preprocessing of an original image, inputting the preprocessed image into a multi-model encoder, and generating a multi-granularity feature representation; inputting the multi-granularity feature representation into a CBAM module to generate a weighted feature map; inputting the weighted feature map into a Transform module, and generating a fusion feature map; inputting the fused feature map into a multi-layer decoder to generate a reconstructed image; performing optimization training on the model according to the reconstructed image and the input image by minimizing a multi-scale loss function; and generating a new reconstructed image, and generating a defect thermodynamic diagram through a pixel-level reconstruction error. According to the method, a novel multi-granularity feature extraction and fusion framework is constructed, defect detection requirements of different industrial scenes can be well met, and a new technical thought is provided for recognition of complex anomalies.

Description

Technical Field

[0001] The present invention belongs to the field of industrial product defect detection, and specifically relates to an industrial product defect detection method based on multi-granularity feature fusion. Background Art

[0002] In recent years, the advancement of Industry 4.0 and intelligent manufacturing has driven the global industrial automation market to exceed hundreds of billions of dollars and continues to grow. In modern industrial production, product quality is the core measure of manufacturing capabilities and corporate competitiveness. Industrial products can develop various defects during complex manufacturing processes, affecting performance, durability, and safety, damaging brands, and increasing costs. Therefore, rigorous defect detection is key to ensuring product quality and safeguarding the interests of all parties. As the manufacturing industry pursues higher quality and higher efficiency, traditional inspection methods face numerous challenges in meeting large-scale, high-precision, and diverse demands, as follows: (1) Defect detection method based on traditional image processing: This method uses traditional computer vision methods such as edge detection (such as the Canny operator), threshold segmentation (such as the Otsu algorithm), and texture analysis (such as the Gabor filter) to extract manually designed features and combines them with classifiers (such as SVM) to determine defects. This method has the following disadvantages: Weak feature representation: Hand-designed features struggle to adapt to complex industrial scenarios (such as metal surface reflections and fabric texture variations), and have poor generalization capabilities for minor defects or complex backgrounds. Insufficient robustness: Sensitive to lighting changes and noise interference, frequent parameter adjustments are required, and end-to-end automated detection is difficult. Difficulties in multi-scale defect detection: The lack of a multi-granularity feature fusion mechanism makes it impossible to simultaneously capture macrostructural anomalies and microtexture defects.

[0003] (2) Detection method based on a single depth model: This method uses a single pre-trained model (such as ResNet, VGG) to extract features and implements defect classification or location through a fully connected layer or a simple decoder. This method has the following disadvantages: Feature Simplification: A single model has limited feature extraction capabilities and cannot integrate the strengths of different network architectures (such as ResNet's residual learning and EfficientNet's lightweight design). Ignoring Multi-Grained Information: Shallow features (edges, basic textures) and deep features (semantic information) are not effectively combined, resulting in a high rate of missed detection of small-scale defects. Inadequate Attention Mechanism: The system lacks the ability to dynamically focus on key areas and is easily affected by background interference.

[0004] (3) Fusion method based on simple splicing of multiple models: The technical solution of this method is to perform channel concatenation or weighted averaging of the output features of multiple models and input them into a unified classifier. This method has the following disadvantages: Feature redundancy and conflict: The feature spaces of different models are not aligned, and direct splicing leads to information redundancy and even interference. Lack of adaptive fusion mechanism: Fixed-weight fusion cannot dynamically adjust the contribution of different models / features, making it difficult to address uneven sample distribution. High computational complexity: Parallel inference of multiple models results in high resource consumption, making it difficult to deploy on edge devices (such as industrial cameras).

[0005] (4) Improved method based on single attention mechanism: This method introduces spatial attention (such as SENet) or channel attention modules into the encoder or decoder to enhance feature expression capabilities. This method has the following disadvantages: Single-dimensional attention: Focusing only on spatial or channel dimensions, it fails to achieve cross-modal joint optimization (e.g., CBAM, which simultaneously integrates spatial and channel attention). Lack of global information: Traditional attention modules lack the ability to model long-range dependencies, resulting in poor detection of continuity across large defects (e.g., carpet scratches). Failure to incorporate multi-scale features: Attention weights are not dynamically allocated based on features of different granularities, leading to confusion between shallow noise and deeper semantic information.

[0006] In summary, there is an urgent need to provide an industrial product defect detection method to solve the technical problems of incomplete feature extraction, low fusion efficiency, insufficient application of attention mechanism and single reconstruction loss in existing anomaly detection methods, so as to effectively realize accurate anomaly detection of industrial product images. Summary of the Invention

[0007] In response to the above-mentioned deficiencies in the prior art, the present invention provides an industrial product defect detection method based on multi-granularity feature fusion, which solves the technical problems of incomplete feature extraction, low fusion efficiency, insufficient application of attention mechanism and single reconstruction loss in existing anomaly detection methods.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: an industrial product defect detection method based on multi-granularity feature fusion, comprising the following steps: S1. Preprocess the original image and input the preprocessed image into the multi-model encoder to generate multi-granularity feature representation; S2, input the multi-granularity feature representation into the CBAM module to generate a weighted feature map; S3, input the weighted feature map into the Transformer module to generate a fused feature map; S4, input the fused feature map into the multi-layer decoder to generate a reconstructed image; S5. Optimize and train the model based on the reconstructed image and the input image by minimizing the multi-scale loss function; S6. Based on the trained model, execute S1 to S4 to generate a new reconstructed image, and generate a defect heat map through pixel-level reconstruction error.

[0009] Furthermore: in S1, the method for preprocessing the original image includes: Format conversion: Convert the format of the original image to a format suitable for deep learning framework; Resize: resize the original image to a fixed height and width; Tensor conversion and normalization: reorder the image dimensions from height, width, and channels to channels, height, and width, and linearly scale pixel values ​​from the integer range of [0, 255] to the floating-point range of [0.0, 1.0].

[0010] Furthermore: In S1, the multi-model encoder includes ResNet50, EfficientNet-B0 and ConvNeXt-Tiny network architectures, and shallow, mid-level and deep features are obtained and fused through ResNet50, EfficientNet-B0 and ConvNeXt-Tiny respectively to form a multi-granularity feature representation; Among them, the expression for generating shallow features is specifically: Where, is the input image, is the initial 7x7 convolutional layer operation in the ResNet50 model, is the batch normalization operation, is the ReLU activation function, is the maximum pooling operation, is the shallow feature extracted by ResNet50, and its output size is ; Where, It is the initial backbone network processing module of EfficientNet-B0. is the first MBConv block sequence in EfficientNet-B0, is the shallow feature extracted by EfficientNet-B0, and its output size is ; Where, The image patch embedding module for ConvNeXt-Tiny, is the first ConvNeXt block sequence in ConvNeXt-Tiny, is the shallow feature extracted by ConvNeXt-Tiny, and its output size is ; The specific expression for generating mid-level features is: Where, and are the first and second residual block sequences in ResNet50, The middle layer features extracted by ResNet50 have an output size of ; Where, 、 and are the second, third, and fourth MBConv block sequences in EfficientNet-B0, respectively. The middle-level features extracted by EfficientNet-B0 have an output size of ; Where, It is the backbone network module used for the first downsampling in ConvNeXt-Tiny. is the second ConvNeXt block sequence, is the middle-level feature extracted by ConvNeXt-Tiny, and its output size is ; The specific expression for generating deep features is: Where, and are the third and fourth residual block sequences in ResNet50, The deep features extracted by ResNet50 have an output size of ; Where, 、 and These are the fifth, sixth, and seventh MBConv block sequences in EfficientNet-B0, This is the last convolution and activation module of the EfficientNet-B0 feature extraction part. is the deep feature extracted by EfficientNet-B0, and its output size is ; Where, It is the backbone network module used for the second downsampling in ConvNeXt-Tiny. is the third ConvNeXt block sequence, It is the backbone network module used for the third downsampling in ConvNeXt-Tiny. is the fourth ConvNeXt block sequence, is the deep feature extracted by ConvNeXt-Tiny, and its output size is .

[0011] Furthermore, in S1, the method for generating multi-granularity feature representation is specifically as follows: The shallow, mid-level, and deep features extracted by ResNet50, EfficientNet-B0, and ConvNeXt-Tiny are adjusted to the same spatial size, and the features of the same level extracted by different architectures are spliced ​​along the channel dimension, and the spliced ​​feature maps are used as multi-granularity feature representations.

[0012] The beneficial effect of the above further scheme is that multiple architectures extract features at different levels in parallel and effectively integrate them, capturing comprehensive and diverse visual information, laying the foundation for subsequent detection.

[0013] Furthermore, in S2, the CBAM module includes a channel attention submodule and a spatial attention submodule connected to each other. S2 includes the following substeps: S21. Input the multi-granularity feature representation into the channel attention submodule to generate a first weighted feature map; S22, inputting the first weighted feature map into the spatial attention submodule to generate a second weighted feature map; S23. Perform bottleneck compression on the second weighted feature map to generate a weighted feature map.

[0014] Further: S21 is specifically: performing global average pooling and global maximum pooling on the multi-granularity feature representation to obtain the first channel descriptor and the second channel descriptor ; Where, F is a multi-granularity feature representation, C is the number of channels of the feature, H is the height of the feature, W is the width of the feature, is the global average pooling, is the global maximum pooling; Input the first channel descriptor and the second channel descriptor into the shared multi-layer perceptron to obtain the channel attention weight ; Where, Processed by a multi-layer perceptron, is the Sigmoid activation function; Multiply the channel attention weight with the multi-granularity feature representation to obtain the first weighted feature map ; Where, is element-wise multiplication; S22 is specifically: performing average pooling and maximum pooling on the first weighted feature map along the channel dimension to generate a first feature descriptor and the second feature descriptor ; Where, is the average pooling along the channel dimension, is the maximum pooling along the channel dimension; The first feature descriptor and the second feature descriptor are spliced ​​along the channel dimension. The spliced ​​feature map is input into the convolution layer to capture a large range of spatial context and the channel is compressed back to 1. The sigmoid activation function is used to generate a spatial attention weight map. ; Where, for Convolutional layer processing of convolution kernel, is the concatenated feature map; Multiply the spatial attention weight map with the first weighted feature map to obtain the second weighted feature map ; Where, ; S23 specifically: Use Convolution compresses the number of channels of the second weighted feature map to generate a weighted feature map ; Where, It is a 1×1 standard convolution layer, and the number of input channels is , the number of output channels is , is the target channel number after compression, for shallow features, , for mid-level features, , for deep features, .

[0015] The beneficial effect of the above further scheme is: by integrating features through dual attention weighting and bottleneck compression, the expressiveness and efficiency of subsequent fusion features are guaranteed.

[0016] Furthermore, in S3, the Transformer module includes interconnected multi-head self-attention submodules and a feedforward network. Specifically, S3 is: S31, flattening and permuting the weighted feature map to obtain the input features of the Transformer module; S32: Input the input features of the Transformer module into the multi-head self-attention submodule to obtain the multi-head attention weight features; S33, input the multi-head attention weight features into the feedforward network to obtain a further multi-granularity feature map; S34, performing weighted fusion on the further multi-granularity feature maps to obtain a fused feature map; S31 specifically includes: The spatial dimension of is flattened to a sequence length , get the input features of the Transformer module ; Where, is a function that changes the order of tensor dimensions, To convert a multi-dimensional tensor into a one-dimensional tensor, ; S32 is specifically: input feature Input multi-head self-attention submodule, through the linear projection matrix 、 and Calculate queries separately ,key Sum , query ,key Sum Split into An independent attention head, for the The output of an attention head , which is calculated by the scaled dot product attention mechanism as follows: Where, is the Softmax function, is the transpose, , is the key vector dimension, , For the Query of attention heads, For the The key of an attention head, For the The value of an attention head; Will The output of an attention head Concatenate in the last dimension and get , and pass an output linear projection matrix Transform to obtain the final weighted feature representation ; Where, Indicates that The output of an attention head Splicing along the last feature dimension to obtain a dimension of Tensor ; The weighted features are represented as Input residual connection and layer normalization layer to obtain multi-head attention weight features ; Where, Normalize the feature vector at each sequence position; S33 is specifically as follows: the multi-head attention weight feature is input into the feedforward network, and the feature of each position is independently transformed to obtain the transformed feature ; Where, is the first linear layer processing, Processing for the second linear layer; The transformed features are input into the residual connection and layer normalization layer to obtain the sequence ; will sequence Shaped into the shape of the feature map, generating further multi-granularity feature maps , which includes shallow, middle and deep feature maps; Where, Processed by the reshape function, the original spatial structure is restored by adjusting the dimension order and shape; S34 is specifically: weighted splicing of shallow, middle and deep feature maps to obtain a fused feature map ; Where, For the splicing operation along the channel dimension, is the shallow feature map, is the feature map of the middle layer, is the deep feature map, is the shallow weight, is the middle weight, is the deep weight.

[0017] The beneficial effect of the above further scheme is: through the Transformer module, the features are deeply processed and cross-granularity weighted fusion is achieved, thereby enhancing the representation ability of the final fused features.

[0018] Furthermore, in S4, the multi-layer decoder includes first to fifth decoding layers connected in sequence, and S4 is specifically: The fused feature maps are input into the first to fifth decoding layers in sequence, and the number of channels is adjusted by transposed convolution to generate the output of the fifth decoding layer. ; Where, For transposed convolution layer processing, is the batch normalization operation, is the output of the fourth decoding layer; The number of channels of the output of the fifth decoding layer is adjusted through the convolution layer, and the output value is mapped to the [0,1] interval using the Sigmoid function to obtain the reconstructed image. ; Where, It is a 3x3 standard convolution layer with 64 input channels and 3 output channels.

[0019] The beneficial effect of the above further solution is that a high-quality image that is highly consistent with the input image is reconstructed by gradually upsampling and refining features through a multi-layer decoder.

[0020] Furthermore: In S5, the method for optimizing the model training is specifically as follows: S51, loading an original image belonging to a specific category, preprocessing the original image, obtaining an input image as training data, and initializing training parameters; S52, inputting the input image into the multi-model encoder, the CBAM module, the Transformer module and the multi-layer decoder in sequence to generate a reconstructed image; S53. Calculate the MSE loss at each scale based on the input image and the reconstructed image, and calculate the total weighted loss of the current round based on the dynamic weight and MSE loss of the current round; S54. Calculate the gradient of the total weighted loss of the current round with respect to the parameters, and use the optimizer to update the parameters of the model, where the model includes a multi-model encoder, a CBAM module, a Transformer module, and a multi-layer decoder; S55. Determine whether the current round has reached the maximum number of rounds. If so, save the final trained model parameters. If not, update the dynamic weights according to the predetermined period. The updated dynamic weights are used as the dynamic weights for the next round. Add 1 to the current round number and return to S52. In S53, calculate the MSE loss at each scale The specific expression is: Where, is the mean square error, For scale The input image is For scale The reconstructed image under It's a scale The total number of pixels in the image below, For scale The input image Pixels, For scale The reconstructed image Pixels; Calculate the The total weighted loss of the round The specific expression is: Where, is the scale set, For the Round downscaling The corresponding dynamic weight, For the Round downscaling The calculated MSE loss; In S55, the method for updating the dynamic weight is specifically as follows: In the current round At the end of the round, normalize the MSE loss of the round, calculate the training progress factor, and update the original weights of the current round based on the two ; Where, For the Round downscaling The normalized MSE loss of is the current round number, For the The training progress factor of the round; Where, is the maximum number of rounds, For the Round downscaling The calculated MSE loss; According to The original weight calculated in the round Update dynamic weight, updated dynamic weight As the dynamic weight actually used in the next round; Where, For the current The scale calculated by the round The original weight of .

[0021] The beneficial effects of the above further scheme are: using multi-scale MSE loss, dynamically adjusting weights, and adaptive optimization based on the training stage and error distribution, making the loss function more flexible and effective at different stages.

[0022] Furthermore, in S6, the method for generating a defect heat map by pixel-level reconstruction error is specifically as follows: S61. Based on the trained model, generate a reconstruction result of the test image, and screen out abnormal images based on the comparison between the image-level abnormality score and the abnormality determination threshold; S62. Calculate a pixel-level error map for the abnormal image, normalize the pixel-level error map to generate a pixel-level abnormality score map, input the pixel-level abnormality score map into a color mapping table to generate a pseudo-color image, and use the pseudo-color image as a defect heat map; In S61, the method for screening out abnormal images according to the image-level abnormality score is specifically as follows: setting an abnormality determination threshold, if the image-level abnormality scores of the input image and the reconstructed image are greater than the abnormality determination threshold, then the original image corresponding to the input image is classified as an abnormal image, otherwise the original image corresponding to the input image is classified as a normal image; the method for setting the abnormality determination threshold is specifically as follows: Pre-calculate the image-level anomaly scores of all samples in the training set (all normal samples) and get their mean and standard deviation ; Where, To calculate the image-level anomaly score, , is the input image of the training set sample after preprocessing, is the reconstructed image of the training set sample, To calculate the mean of the selected samples, To calculate the standard deviation of the selected sample; According to the mean and standard deviation Set the anomaly determination threshold ; Where, is a preset constant; In S62, for each pixel point (h, w) at each spatial position in each channel c in the image, the input image of the test set sample after preprocessing is calculated. Corresponding reconstructed image The absolute value of the pixel difference at these points; these absolute differences of all channels at the same spatial position are averaged to obtain the final error at each spatial position, thus generating an overall pixel-level error map , its specific expression is: Where, for The pixel value at channel c, height h, width w, for The pixel value at channel c, height h, width w, is the number of channels, Pixel-level error map At any spatial location The error value at ; Calculate pixel-level anomaly score map The specific expression is: Where, Represents the pixel-level error map Perform Min-Max normalization, To get the minimum value, To obtain the maximum value, To prevent division by zero for small constants, is the pixel-level anomaly score map At any spatial location The score value at Generate pseudo-color images The specific expression is: Where, is a color mapping function used to convert the pixel-level anomaly score map At any spatial location The score value at that position is converted into the color information of the pseudo-color image at that position. Pseudo-color image At any spatial location The color information at.

[0023] The beneficial effects of the above further solution are: abnormal conditions are determined by image-level errors, and defect heat maps are mapped by pixel-level errors, thus achieving effective detection from the whole to the part.

[0024] The beneficial effects of the present invention are as follows: the present invention provides an industrial product defect detection method based on multi-granularity feature fusion. By combining multiple pre-trained convolutional neural network models, multi-granularity feature extraction capabilities, attention mechanism, Transformer fusion module and multi-scale reconstruction loss, it effectively realizes accurate anomaly detection in industrial product images. Compared with existing anomaly detection methods, it has the following effects: (1) The model structure is novel and flexible, which improves detection accuracy: By integrating three pre-trained models, ResNet50, EfficientNetB0 and ConvNeXt-Tiny, and combining the attention mechanism of the CBAM module and the Transformer module, a novel multi-granularity feature extraction and fusion framework is constructed to improve the feature expression and generalization capabilities of the model, and solve the problems of incomplete feature extraction and low fusion efficiency in existing anomaly detection methods. It can better adapt to the defect detection needs of different industrial scenarios and provide new technical ideas for the identification of complex anomalies.

[0025] (2) Improved attention mechanism: The CBAM module is combined with the spatial-channel attention mechanism, combined with the long-range dependency capability of the Transformer model, to enhance the focusing ability of the defect area, thereby improving the model's sensitivity to key areas and solving the problem of insufficient application of the attention mechanism in existing anomaly detection methods.

[0026] (3) Better cope with the challenges of multi-scale anomalies and data imbalance: By adopting the multi-scale MSE loss function and adaptive weight adjustment mechanism, we explored the adaptive processing methods for anomalies of different scales, which to some extent alleviated the limitations of traditional methods such as the scarcity of abnormal data or the need for artificially defined features, solved the problem of single reconstruction loss in existing anomaly detection methods, and provided a flexible solution for unsupervised defect detection.

[0027] (4) Automated processing and resource utilization have potential: By optimizing the feature processing process through adaptive average pooling, bottleneck compression and weighted fusion, combined with unsupervised training design, it reduces excessive dependence on computing resources and has high automated processing capabilities, which helps to reduce operational complexity and lays the foundation for efficient deployment in industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of an industrial product defect detection method based on multi-granularity feature fusion. DETAILED DESCRIPTION

[0029] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0030] The technical problem to be solved by this invention is to provide an industrial product defect detection method based on multi-granularity feature fusion, which solves the technical problems of incomplete feature extraction, low fusion efficiency, insufficient application of attention mechanism, and single reconstruction loss in existing anomaly detection methods, and effectively realizes accurate anomaly detection of industrial product images. Specifically, by combining multiple pre-trained convolutional neural network models, multi-granularity feature extraction capabilities, attention mechanism, Transformer fusion module and multi-scale reconstruction loss, an efficient and accurate anomaly detection system is constructed. The following is the technical solution process: like Figure 1 As shown, in one embodiment of the present invention, a method for industrial product defect detection based on multi-granularity feature fusion includes the following steps: S1. Preprocess the original image and input the preprocessed image into the multi-model encoder to generate multi-granularity feature representation; S2, input the multi-granularity feature representation into the CBAM module to generate a weighted feature map; S3, input the weighted feature map into the Transformer module to generate a fused feature map; S4, input the fused feature map into the multi-layer decoder to generate a reconstructed image; S5. Optimize and train the model based on the reconstructed image and the input image by minimizing the multi-scale loss function; S6. Based on the trained model, execute S1 to S4 to generate a new reconstructed image, and generate a defect heat map through pixel-level reconstruction error.

[0031] In S1, the method for preprocessing the original image includes: Format conversion: Convert the format of the original image to a format suitable for deep learning framework; Resize: resize the original image to a fixed height and width; Tensor conversion and normalization: reorder the image dimensions from height, width, and channel to channel, height, and width, and linearly scale the pixel values ​​from the integer range of [0, 255] to the floating-point range of [0.0, 1.0]; The preprocessed image is represented as: Where, is the original image, ToPILImage is the format conversion, Resize is the size adjustment, and ToTensor is the tensor conversion and normalization. is the input image, , B is the batch size, which is 8 in the training phase and 1 in the test phase, 3 is the number of channels, and , training phase represent , testing phase represent .

[0032] In this example, the original images are images of products to be inspected, captured on an industrial production line. These images are typically RGB color images, such as those for specific categories like bottles and metal nuts in the MVTec AD dataset. The original images may have varying sizes but typically contain rich surface detail information.

[0033] In S1, the multi-model encoder includes ResNet50, EfficientNet-B0 and ConvNeXt-Tiny network architectures. ResNet50, EfficientNet-B0 and ConvNeXt-Tiny obtain shallow, mid-level and deep features respectively and fuse them to form a multi-granularity feature representation; In this embodiment, the present invention utilizes three convolutional neural network architectures pre-trained on a large image dataset as powerful feature extractors. These models, due to their distinct architectural design philosophies, are able to capture complementary feature information in images. Feature maps are extracted from different depth levels of each model, forming a multi-layered, multi-granular feature representation.

[0034] In this embodiment, the shallow feature extraction stage selects the relatively early layers of each model to capture the underlying visual elements of the image, such as edges, corners, colors, and basic textures. The expression for generating shallow features is specifically: Where, is the initial 7x7 convolutional layer operation in the ResNet50 model, is the batch normalization operation, is the ReLU activation function, is the maximum pooling operation, is the shallow feature extracted by ResNet50, and its output size is ; Where, It is the initial backbone network processing module of EfficientNet-B0, including convolution, batch normalization and activation. is the first MBConv block sequence in EfficientNet-B0, is the shallow feature extracted by EfficientNet-B0, and its output size is ; Where, The image block embedding module of ConvNeXt-Tiny contains a 4x4 convolution layer and layer normalization. is the first ConvNeXt block sequence in ConvNeXt-Tiny, is the shallow feature extracted by ConvNeXt-Tiny, and its output size is ; In this embodiment, the mid-level feature extraction stage extracts features from deeper locations in the model, aiming to capture more abstract and complex patterns than those in the shallow layers, such as object parts, shape contours, etc. The expression for generating mid-level features is specifically: Where, and are the first and second residual block sequences in ResNet50, The middle layer features extracted by ResNet50 have an output size of ; Where, 、 and are the second, third, and fourth MBConv block sequences in EfficientNet-B0, respectively. The middle-level features extracted by EfficientNet-B0 have an output size of ; Where, It is the backbone network module for the first downsampling in ConvNeXt-Tiny, including layer normalization and 2x2 convolution layer. is the second ConvNeXt block sequence, is the middle-level feature extracted by ConvNeXt-Tiny, and its output size is ; In this embodiment, the deep feature extraction stage uses the complete encoder backbone of each model (usually the part before the global average pooling layer or classification head) to capture the high-level semantic information and global context representation of the image. The expression for generating deep features is specifically: Where, and are the third and fourth residual block sequences in ResNet50, The deep features extracted by ResNet50; Where, 、 and These are the fifth, sixth, and seventh MBConv block sequences in EfficientNet-B0, This is the last convolution and activation module of the EfficientNet-B0 feature extraction part. is the deep feature extracted by EfficientNet-B0, and its output size is ; Where, It is the backbone network module used for the second downsampling in ConvNeXt-Tiny. is the third ConvNeXt block sequence, It is the backbone network module used for the third downsampling in ConvNeXt-Tiny. is the fourth ConvNeXt block sequence, is the deep feature extracted by ConvNeXt-Tiny, and its output size is .

[0035] In S1, the method for generating multi-granularity feature representation is as follows: The shallow, mid-level, and deep features extracted by ResNet50, EfficientNet-B0, and ConvNeXt-Tiny are adjusted to the same spatial size, and the features of the same level extracted by different architectures are spliced ​​along the channel dimension, and the spliced ​​feature maps are used as multi-granularity feature representations.

[0036] In this embodiment, in order to facilitate subsequent fusion, these feature maps of different sizes are adjusted to a unified size through adaptive average pooling. Spatial resolution.

[0037] Where, Indicates that the spatial dimension of the input feature map is adjusted to Adaptive average pooling operation, represents the resized feature map, , , Indicates shallow layer, Represents the middle layer, Indicates deep.

[0038] At each granularity level, the images from the three models that have been adjusted to the same spatial size are Feature map Stitching is performed along the channel dimension.

[0039] Where, represents the splicing operation along the channel dimension, It is the concatenated feature map; Among them, the number of channels of shallow features , the number of channels of the middle-level features , the number of channels of deep features .

[0040] In S2, the CBAM module includes a channel attention submodule and a spatial attention submodule that are connected to each other. S2 includes the following substeps: S21. Input the multi-granularity feature representation into the channel attention submodule to generate a first weighted feature map; S22, inputting the first weighted feature map into the spatial attention submodule to generate a second weighted feature map; S23. Perform bottleneck compression on the second weighted feature map to generate a weighted feature map.

[0041] S21 is specifically as follows: perform global average pooling and global maximum pooling on the multi-granularity feature representation to obtain the first channel descriptor and the second channel descriptor ; Where, F is a multi-granularity feature representation, , , C is the number of channels of the feature, H is the height of the feature, W is the width of the feature, in this embodiment , is the global average pooling, is the global maximum pooling; Input the first channel descriptor and the second channel descriptor into the shared multi-layer perceptron to obtain the channel attention weight ; Where, Processed by a multi-layer perceptron, Sigmoid activation function; multilayer perceptron The specific expression is: Where, , and is the weight of the MLP, which is equivalent to using 1x1 convolution. The above formula is implemented as: , Indicates 1×1 standard convolution layer processing, the number of input channels is C, and the number of output channels is C / r, Represents 1×1 standard convolutional layer processing, the number of input channels is C / r, and the number of output channels is C.

[0042] Multiply the channel attention weight with the multi-granularity feature representation to obtain the first weighted feature map ; Where, For element-by-element multiplication, when calculating, The spatial dimension 1×1 will be automatically broadcast to To match spatial dimension; S22 is specifically: performing average pooling and maximum pooling on the first weighted feature map along the channel dimension to generate a first feature descriptor and the second feature descriptor ; Where, is the average pooling along the channel dimension, is the maximum pooling along the channel dimension, , ; By formula The first feature descriptor and the second feature descriptor are spliced ​​along the channel dimension. The spliced ​​feature map is input into the convolution layer to capture a large range of spatial context and the channel is compressed back to 1. The sigmoid activation function is used to generate a spatial attention weight map. ; Where, for Convolutional layer processing of convolution kernel, is the concatenated feature map, ; Multiply the spatial attention weight map with the first weighted feature map to obtain the second weighted feature map ; Where, For element-by-element multiplication, when calculating The channel dimension 1 is automatically broadcast to To match The channel dimension, ; S23 specifically: Use Convolution compresses the number of channels of the second weighted feature map to generate a weighted feature map ; Where, It is a 1×1 standard convolution layer, and the number of input channels is , the number of output channels is , is the target number of channels after compression, , for shallow features, , for mid-level features, , for deep features, .

[0043] In S3, the Transformer module includes interconnected multi-head self-attention submodules and feedforward networks. Specifically, S3 is: S31, flattening and permuting the weighted feature map to obtain the input features of the Transformer module; S32: Input the input features of the Transformer module into the multi-head self-attention submodule to obtain the multi-head attention weight features; S33, input the multi-head attention weight features into the feedforward network to obtain a further multi-granularity feature map; S34, performing weighted fusion on the further multi-granularity feature maps to obtain a fused feature map; In this embodiment, the Transformer module is introduced to capture the long-distance dependencies between different spatial positions in the feature map, which is very helpful for understanding the global structure and context information.

[0044] S31 specifically includes: The spatial dimension of is flattened to a sequence length , get the input features of the Transformer module ; Where, , is a function that changes the order of tensor dimensions, To convert a multi-dimensional tensor into a one-dimensional tensor, ; S32 is specifically: input feature Input multi-head self-attention submodule, through the linear projection matrix 、 and Calculate queries separately ,key Sum , the specific expression is: In the formula, query ,key Sum The dimensions of , , , ; The query ,key Sum Split into In this embodiment, h The value of is 8, for The output of an attention head , which is calculated by the scaled dot product attention mechanism as follows: Where, is the Softmax function, is the transpose, , , is the key vector dimension, , For the Query of attention heads, For the The key of an attention head, For the The value of an attention head, 、 and The dimensions of ; Will The output of an attention head Concatenate in the last dimension and get , and pass an output linear projection matrix Transform to obtain the final weighted feature representation ; Where, Indicates that The output of an attention head Splicing along the last feature dimension to obtain a dimension of Tensor , , ; The weighted features are represented as Input residual connection and layer normalization layer to obtain multi-head attention weight features ; Where, Normalize the feature vector at each sequence position; S33 is specifically as follows: the multi-head attention weight feature is input into the feedforward network, and the feature of each position is independently transformed to obtain the transformed feature ; Where, is the first linear layer processing, Processing for the second linear layer; The transformed features are input into the residual connection and layer normalization layer to obtain the sequence ; will sequence Shaped into the shape of the feature map, generating further multi-granularity feature maps , which includes shallow, middle and deep feature maps; Where, Processed by the reshape function, the original spatial structure is restored by adjusting the dimension order and shape; S34 is specifically: weighted splicing of shallow, middle and deep feature maps to obtain a fused feature map ; Where, For the splicing operation along the channel dimension, is the shallow feature map, is the feature map of the middle layer, is the deep feature map, is the shallow weight, is the middle weight, is the deep weight. In this embodiment, , , , used to adjust the contribution of different level features to the final representation. Fusion feature map Size ,in, The expression is: Fusion feature map It is the final output of the entire encoding process, which contains rich multi-scale and multi-source feature information and will serve as the input of the decoder.

[0045] In S4, the goal of the multi-layer decoder is to fuse the feature maps Gradually upsample and restore to the same size as the input image ( ) and the reconstructed image with the same number of channels (3) .

[0046] The multi-layer decoder includes the first to fifth decoding layers connected in sequence, and S4 is specifically: The fused feature maps are input into the first to fifth decoding layers in sequence, and the number of channels is adjusted by transposed convolution to generate the output of the fifth decoding layer. ; Where, , For transposed convolution layer processing, is the batch normalization operation, is the output of the fourth decoding layer; In this embodiment, each decoding layer includes a transposed convolution layer and a batch normalization layer, and the formula for calculating the output is the same.

[0047] The output of the first decoding layer , the number of output channels of the transposed convolution layer is 1024, and the stride is 2.

[0048] The output of the second decoding layer , the number of output channels of the transposed convolution layer is 512, and the stride is 2.

[0049] The output of the third decoding layer , the number of output channels of the transposed convolution layer is 256, and the stride is 2.

[0050] Output of the fourth decoding layer , the number of output channels of the transposed convolution layer is 128, and the stride is 2.

[0051] Output of the fifth decoding layer , the number of output channels of the transposed convolution layer is 64, and the stride is 2.

[0052] The number of channels of the output of the fifth decoding layer is adjusted through the convolution layer, and the output value is mapped to the [0,1] interval using the Sigmoid function to obtain the reconstructed image. ; Where, , It is a 3x3 standard convolution layer with 64 input channels and 3 output channels.

[0053] In S5, the method for optimizing the model training is as follows: S51, loading an original image belonging to a specific category, preprocessing the original image, obtaining an input image as training data, and initializing training parameters; S52, inputting the input image into the multi-model encoder, the CBAM module, the Transformer module and the multi-layer decoder in sequence to generate a reconstructed image; S53. Calculate the MSE loss at each scale based on the input image and the reconstructed image, and calculate the total weighted loss of the current round based on the dynamic weight and MSE loss of the current round; S54. Calculate the gradient of the total weighted loss of the current round with respect to the parameters, and use the optimizer to update the parameters of the model, where the model includes a multi-model encoder, a CBAM module, a Transformer module, and a multi-layer decoder; S55. Determine whether the current round has reached the maximum number of rounds. If so, save the final trained model parameters. If not, update the dynamic weights according to the predetermined period. The updated dynamic weights are used as the dynamic weights for the next round. Add 1 to the current round number and return to S52. In this embodiment, the goal of training is to optimize the parameters of the entire model. , so that it can reconstruct the normal samples in the training set with high quality, that is, minimize the difference between the input image and the reconstructed image. This example is performed on a computing device equipped with an NVIDIA GeForce RTX 4060 GPU during the training phase, and the learning rate is set to , the batch size is set to 8, the maximum number of rounds is set to 250, the scheduled period for adaptive adjustment of dynamic weights at each scale is set to 10 training rounds, the optimizer uses Adam, and mixed precision training (AMP) is enabled to improve training efficiency.

[0054] In S53, calculate the MSE loss at each scale The specific expression is: Where, is the mean square error, For scale The input image is For scale The reconstructed image under It's a scale The total number of pixels in the image below, For scale The input image Pixels, For scale The reconstructed image Pixels; In this example, the MSE loss is calculated at each scale. Before, yes and Apply average pooling to downsample to obtain image versions of different scales. Let the scale set be , corresponding to 0, 1, and 2 average pooling with a step size of 2.

[0055] Where, Indicates application Average pooling with a step size of 2. hour, , which is the original image size.

[0056] To ensure that the model can reconstruct the image well both in terms of overall appearance and details at different scales, this embodiment adopts a multi-scale mean squared error loss strategy, calculates the mean squared error of the input image and the reconstructed image at multiple resolution scales, and weights and sums these errors according to dynamic weights to calculate the total weighted loss.

[0057] Calculate the The total weighted loss of the round The specific expression is: Where, is the scale set, For the Round downscaling The corresponding dynamic weight, For the Round downscaling The calculated MSE loss; In S55, the dynamic weights are updated according to a predetermined period, for example, every 10 epochs, to adapt to the learning progress of the model at different scales. The method for updating the dynamic weights is as follows: First, in the current round At the end of the round, normalize the MSE loss of the round, calculate the training progress factor, and update the original weights of the current round based on the two ; Where, For the Round downscaling The normalized MSE loss of is the current round number, For the The training progress factor of the round; Where, is the maximum number of rounds, For the Round downscaling The calculated MSE loss; Then, according to The original weight calculated in the round Update dynamic weight, updated dynamic weight As the next round ( The dynamic weight actually used in each round); Where, For the current The scale calculated by the round The original weight of .

[0058] In S6, the method for generating a defect heat map through pixel-level reconstruction error is as follows: S61. Based on the trained model, generate a reconstruction result of the test image, and screen out abnormal images based on the comparison between the image-level abnormality score and the abnormality determination threshold; S62. Calculate a pixel-level error map for the abnormal image, normalize the pixel-level error map to generate a pixel-level abnormality score map, input the pixel-level abnormality score map into a color mapping table to generate a pseudo-color image, and use the pseudo-color image as a defect heat map; In this embodiment, the present invention uses a model trained on a specific class of normal samples to evaluate new test samples. The core idea is that normal samples should be well reconstructed by the model (low reconstruction error), while abnormal samples (containing defects) are difficult to accurately reconstruct (high reconstruction error).

[0059] In S61, the method for screening out abnormal images according to the image-level abnormality score is specifically as follows: setting an abnormality determination threshold, if the image-level abnormality scores of the input image and the reconstructed image are greater than the abnormality determination threshold, then the original image corresponding to the input image is classified as an abnormal image, otherwise the original image corresponding to the input image is classified as a normal image; the method for setting the abnormality determination threshold is specifically as follows: Pre-calculate the image-level anomaly scores of all samples in the training set (all normal samples) and get their mean and standard deviation ; Where, To calculate the image-level anomaly score, this embodiment uses MSE as the image-level anomaly score. , is the input image of the training set sample after preprocessing, is the reconstructed image of the training set sample, To calculate the mean of the selected samples, To calculate the standard deviation of the selected sample; According to the mean and standard deviation Set the anomaly determination threshold ; Where, is a preset constant, and its value in this embodiment is 2; For the preprocessed input image of the test set sample First, we use the trained model to get the reconstructed image, and then use the same method to calculate the image-level anomaly score. ,like , then the corresponding original image is classified as an abnormal image, otherwise it is classified as a normal image.

[0060] In S62, for each pixel point (h, w) at each spatial position in each channel c in the image (i.e., considering each specific pixel data point (c, h, w)), the input image of the test set sample after preprocessing is calculated. Corresponding reconstructed image The absolute value of the pixel difference at these points; these absolute differences of all channels at the same spatial position are averaged to obtain the final error at each spatial position, thus generating an overall pixel-level error map , its specific expression is: Where, for The pixel value at channel c, height h, width w, for The pixel value at channel c, height h, width w, is the number of channels, is the pixel-level error map At any spatial location The error value at ; Calculate pixel-level anomaly score map The specific expression is: Where, Represents the pixel-level error map Perform Min-Max normalization, To get the minimum value, To take the maximum value (that is, take the entire ), the minimum and maximum error values ​​in To prevent the small constant from dividing by zero, the value in this embodiment is , is the pixel-level anomaly score map At any spatial location The score value at .

[0061] Generate pseudo-color images The specific expression is: Where, is a color mapping function used to convert the pixel-level anomaly score map At any spatial location The score value at that position is converted into the color information of the pseudo-color image at that position. Pseudo-color image At any spatial location In this embodiment, the high-score area of ​​the pseudo-color image is represented by a striking color, and the high-score area refers to a high error or defect.

[0062] This example uses the area under the receiver operating characteristic curve (AUROC) to quantify the model's ability to distinguish between normal and abnormal conditions.

[0063] Image level The specific expression is: Where, is a set containing the true labels of all images in the test set, where each label value is 0 or 1 (0 represents a normal sample and 1 represents an abnormal sample). It is a The set of image-level anomaly scores corresponding to each image in , Represents the standard function for calculating the area under the receiver operating characteristic curve.

[0064] Pixel level The specific expression is: Where, It is a set of all pixel true labels of all images in the test set (from the binary Ground Truth Mask corresponding to each image), where each label value is 0 or 1 (0 represents a normal pixel and 1 represents a defective pixel). It is a The set of pixel-level anomaly scores corresponding to each pixel in .

[0065] The closer the AUROC value is to 1, the better the discrimination performance of the model.

[0066] This method extracts multi-granular features through a multi-model encoder, performs feature weighting through a CBAM module, and performs deep fusion through a Transformer module. It then optimizes and trains the model using a multi-scale MSE loss. Image reconstruction is then achieved through a multi-layer decoder. Finally, anomaly detection is performed based on the reconstruction error. This solution aims to overcome the technical issues of existing methods, such as incomplete feature extraction, inefficient fusion, insufficient application of the attention mechanism, and a single reconstruction loss, making it suitable for accurate defect detection in industrial scenarios. The performance of this method on the MVTec AD dataset is shown in Table 1.

[0067] Table 1 Performance on the MVTec AD dataset (unit: %) In the description of the present invention, it should be understood that the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of such features.

Claims

1. A method for industrial product defect detection based on multi-granularity feature fusion, characterized in that: The following steps are involved: S1. Preprocess the original image and input the preprocessed image into the multi-model encoder to generate multi-granularity feature representation; S2, input the multi-granularity feature representation into the CBAM module to generate a weighted feature map; S3, input the weighted feature map into the Transformer module to generate a fused feature map; S4, input the fused feature map into the multi-layer decoder to generate a reconstructed image; S5. Optimize and train the model based on the reconstructed image and the input image by minimizing the multi-scale loss function; S6. Based on the trained model, execute S1 to S4 to generate a new reconstructed image, and generate a defect heat map through pixel-level reconstruction error.

2. The industrial product defect detection method based on multi-granularity feature fusion according to claim 1 is characterized in that: In S1, the method for preprocessing the original image includes: Format conversion: Convert the format of the original image to a format suitable for deep learning framework; Resize: resize the original image to a fixed height and width; Tensor conversion and normalization: reorder the image dimensions from height, width, and channels to channels, height, and width, and linearly scale pixel values ​​from the integer range of [0, 255] to the floating-point range of [0.0, 1.0].

3. The industrial product defect detection method based on multi-granularity feature fusion according to claim 1 is characterized in that: In S1, the multi-model encoder includes ResNet50, EfficientNet-B0 and ConvNeXt-Tiny network architectures. ResNet50, EfficientNet-B0 and ConvNeXt-Tiny obtain shallow, mid-level and deep features respectively and fuse them to form a multi-granularity feature representation; Among them, the expression for generating shallow features is specifically: Where, is the input image, is the initial 7x7 convolutional layer operation in the ResNet50 model, is the batch normalization operation, is the ReLU activation function, is the maximum pooling operation, The shallow features extracted by ResNet50; Where, It is the initial backbone network processing module of EfficientNet-B0. is the first MBConv block sequence in EfficientNet-B0, It is the shallow features extracted by EfficientNet-B0; Where, The image patch embedding module for ConvNeXt-Tiny, is the first ConvNeXt block sequence in ConvNeXt-Tiny, The shallow features extracted by ConvNeXt-Tiny; The specific expression for generating mid-level features is: Where, and are the first and second residual block sequences in ResNet50, The middle-level features extracted by ResNet50; Where, 、 and are the second, third, and fourth MBConv block sequences in EfficientNet-B0, respectively. The middle-level features extracted by EfficientNet-B0; Where, It is the backbone network module used for the first downsampling in ConvNeXt-Tiny. is the second ConvNeXt block sequence, The middle-level features extracted by ConvNeXt-Tiny; The expression for generating deep features is: Where, and are the third and fourth residual block sequences in ResNet50, The deep features extracted by ResNet50; Where, 、 and These are the fifth, sixth, and seventh MBConv block sequences in EfficientNet-B0, This is the last convolution and activation module of the EfficientNet-B0 feature extraction part. The deep features extracted by EfficientNet-B0; Where, It is the backbone network module used for the second downsampling in ConvNeXt-Tiny. is the third ConvNeXt block sequence, It is the backbone network module used for the third downsampling in ConvNeXt-Tiny. is the fourth ConvNeXt block sequence, The deep features extracted by ConvNeXt-Tiny.

4. The industrial product defect detection method based on multi-granularity feature fusion according to claim 3 is characterized in that: In S1, the method for generating multi-granularity feature representation is as follows: The shallow, mid-level, and deep features extracted by ResNet50, EfficientNet-B0, and ConvNeXt-Tiny are adjusted to the same spatial size, and the features of the same level extracted by different architectures are spliced ​​along the channel dimension, and the spliced ​​feature maps are used as multi-granularity feature representations.

5. The industrial product defect detection method based on multi-granularity feature fusion according to claim 4 is characterized in that: In S2, the CBAM module includes a channel attention submodule and a spatial attention submodule that are connected to each other. S2 includes the following substeps: S21. Input the multi-granularity feature representation into the channel attention submodule to generate a first weighted feature map; S22, inputting the first weighted feature map into the spatial attention submodule to generate a second weighted feature map; S23. Perform bottleneck compression on the second weighted feature map to generate a weighted feature map.

6. The industrial product defect detection method based on multi-granularity feature fusion according to claim 5 is characterized in that: S21 is specifically as follows: perform global average pooling and global maximum pooling on the multi-granularity feature representation to obtain the first channel descriptor and the second channel descriptor ; Where, F is a multi-granularity feature representation, C is the number of channels of the feature, H is the height of the feature, W is the width of the feature, is the global average pooling, It is the global maximum pooling; Input the first channel descriptor and the second channel descriptor into the shared multi-layer perceptron to obtain the channel attention weight ; Where, Processed by a multi-layer perceptron, is the Sigmoid activation function; Multiply the channel attention weight with the multi-granularity feature representation to obtain the first weighted feature map ; Where, is element-wise multiplication; S22 is specifically: performing average pooling and maximum pooling on the first weighted feature map along the channel dimension to generate a first feature descriptor and the second feature descriptor ; Where, is the average pooling along the channel dimension, is the maximum pooling along the channel dimension; The first feature descriptor and the second feature descriptor are spliced ​​along the channel dimension. The spliced ​​feature map is input into the convolution layer to capture a large range of spatial context and the channel is compressed back to 1. The sigmoid activation function is used to generate a spatial attention weight map. ; Where, for Convolutional layer processing of convolution kernel, is the concatenated feature map; Multiply the spatial attention weight map with the first weighted feature map to obtain the second weighted feature map ; Where, ; S23 specifically: Use Convolution compresses the number of channels of the second weighted feature map to generate a weighted feature map ; Where, It is a 1×1 standard convolution layer, and the number of input channels is , the number of output channels is , is the target channel number after compression, for shallow features, , for mid-level features, , for deep features, .

7. The industrial product defect detection method based on multi-granularity feature fusion according to claim 6 is characterized in that: In S3, the Transformer module includes interconnected multi-head self-attention submodules and feedforward networks. Specifically, S3 is: S31, flattening and permuting the weighted feature map to obtain the input features of the Transformer module; S32: Input the input features of the Transformer module into the multi-head self-attention submodule to obtain the multi-head attention weight features; S33, input the multi-head attention weight features into the feedforward network to obtain a further multi-granularity feature map; S34, performing weighted fusion on the further multi-granularity feature maps to obtain a fused feature map; S31 specifically includes: The spatial dimension of is flattened to a sequence length , get the input features of the Transformer module ; Where, is a function that changes the order of tensor dimensions, To convert a multi-dimensional tensor into a one-dimensional tensor, ; S32 is specifically: input feature Input multi-head self-attention submodule, through the linear projection matrix 、 and Calculate queries separately ,key Sum , query ,key Sum Split into An independent attention head, for the The output of an attention head , which is calculated by the scaled dot product attention mechanism as follows: Where, is the Softmax function, is the transpose, , is the key vector dimension, , For the Query of attention heads, For the The key of an attention head, For the The value of an attention head; Will The output of an attention head Concatenate in the last dimension and get , and pass an output linear projection matrix Transform to obtain the final weighted feature representation ; Where, Indicates that The output of an attention head Splicing along the last feature dimension to obtain a dimension of Tensor ; The weighted features are represented as Input residual connection and layer normalization layer to obtain multi-head attention weight features ; Where, Normalize the feature vector at each sequence position; S33 is specifically as follows: the multi-head attention weight feature is input into the feedforward network, and the feature of each position is independently transformed to obtain the transformed feature ; Where, is the first linear layer processing, Processing for the second linear layer; The transformed features are input into the residual connection and layer normalization layer to obtain the sequence ; will sequence Shaped into the shape of the feature map, generating further multi-granularity feature maps , which includes shallow, middle and deep feature maps; Where, Processed by the reshape function, the original spatial structure is restored by adjusting the dimension order and shape; S34 is specifically: weighted splicing of shallow, middle and deep feature maps to obtain a fused feature map ; Where, For the splicing operation along the channel dimension, is the shallow feature map, is the feature map of the middle layer, is the deep feature map, is the shallow weight, is the middle weight, is the deep weight.

8. The industrial product defect detection method based on multi-granularity feature fusion according to claim 7 is characterized in that: In S4, the multi-layer decoder includes the first to fifth decoding layers connected in sequence. Specifically, S4 is: The fused feature maps are input into the first to fifth decoding layers in sequence, and the number of channels is adjusted by transposed convolution to generate the output of the fifth decoding layer. ; Where, For transposed convolution layer processing, is the batch normalization operation, is the output of the fourth decoding layer; The number of channels of the output of the fifth decoding layer is adjusted through the convolution layer, and the output value is mapped to the [0,1] interval using the Sigmoid function to obtain the reconstructed image. ; Where, It is a 3x3 standard convolution layer with 64 input channels and 3 output channels.

9. The industrial product defect detection method based on multi-granularity feature fusion according to claim 8, characterized in that: In S5, the method for optimizing the model training is as follows: S51, loading an original image belonging to a specific category, preprocessing the original image, obtaining an input image as training data, and initializing training parameters; S52, inputting the input image into the multi-model encoder, the CBAM module, the Transformer module and the multi-layer decoder in sequence to generate a reconstructed image; S53. Calculate the MSE loss at each scale based on the input image and the reconstructed image, and calculate the total weighted loss of the current round based on the dynamic weight and MSE loss of the current round; S54. Calculate the gradient of the total weighted loss of the current round with respect to the parameters, and use the optimizer to update the parameters of the model, where the model includes a multi-model encoder, a CBAM module, a Transformer module, and a multi-layer decoder; S55. Determine whether the current round has reached the maximum number of rounds. If so, save the final trained model parameters. If not, update the dynamic weights according to the predetermined period. The updated dynamic weights are used as the dynamic weights for the next round. Add 1 to the current round number and return to S52. In S53, calculate the MSE loss at each scale The specific expression is: Where, is the mean square error, For scale The input image is For scale The reconstructed image under It's a scale The total number of pixels in the image below, For scale The input image Pixels, For scale The reconstructed image Pixels; Calculate the Total weighted loss for the round The specific expression is: Where, is the scale set, For the Round downscaling The corresponding dynamic weight, For the Round downscaling The calculated MSE loss; In S55, the method for updating the dynamic weight is specifically as follows: In the current round At the end of the round, normalize the MSE loss of the round, calculate the training progress factor, and update the original weights of the current round based on the two ; Where, For the Round downscaling The normalized MSE loss of is the current round number, For the The training progress factor of the round; Where, is the maximum number of rounds, For the Round downscaling The calculated MSE loss; According to The original weight calculated in the round Update dynamic weight, updated dynamic weight As the dynamic weight actually used in the next round; Where, For the current The scale calculated by the round The original weight of .

10. The industrial product defect detection method based on multi-granularity feature fusion according to claim 9 is characterized in that: In S6, the method for generating a defect heat map through pixel-level reconstruction error is as follows: S61. Based on the trained model, generate a reconstruction result of the test image, and screen out abnormal images based on the comparison between the image-level abnormality score and the abnormality determination threshold; S62. Calculate a pixel-level error map for the abnormal image, normalize the pixel-level error map to generate a pixel-level abnormality score map, input the pixel-level abnormality score map into a color mapping table to generate a pseudo-color image, and use the pseudo-color image as a defect heat map; In S61, the method for screening out abnormal images according to the image-level abnormality score is specifically as follows: setting an abnormality determination threshold, if the image-level abnormality scores of the input image and the reconstructed image are greater than the abnormality determination threshold, then the original image corresponding to the input image is classified as an abnormal image, otherwise the original image corresponding to the input image is classified as a normal image; the method for setting the abnormality determination threshold is specifically as follows: Pre-calculate the image-level anomaly scores of all samples in the training set (all normal samples) and get their mean and standard deviation ; Where, To calculate the image-level anomaly score, , is the input image of the training set sample after preprocessing, is the reconstructed image of the training set sample, To calculate the mean of the selected samples, To calculate the standard deviation of the selected sample; According to the mean and standard deviation Set the anomaly determination threshold ; Where, is a preset constant; In S62, for each pixel point (h, w) at each spatial position in each channel c in the image, the input image of the test set sample after preprocessing is calculated. And the corresponding reconstructed image The absolute value of the pixel difference at these points; these absolute differences of all channels at the same spatial position are averaged to obtain the final error at each spatial position, thus generating an overall pixel-level error map , its specific expression is: Where, for The pixel value at channel c, height h, width w, for The pixel value at channel c, height h, width w, is the number of channels, is the pixel-level error map At any spatial location The error value at ; Calculate pixel-level anomaly score map The specific expression is: Where, Represents the pixel-level error map Perform Min-Max normalization, To get the minimum value, To obtain the maximum value, To prevent division by zero for small constants, is the pixel-level anomaly score map At any spatial location The score value at Generate pseudo-color images The specific expression is: Where, is a color mapping function used to convert the pixel-level anomaly score map At any spatial location The score value at that position is converted into the color information of the pseudo-color image at that position. Pseudo-color image At any spatial location The color information at.

Images