An industrial defect detection method and system based on SAM prior guidance

Abstract

The present invention proposes an industrial defect detection method and system based on SAM prior guidance, which relates to the field of industrial computer vision. An industrial surface defect image to be detected is input into a trained detection model, and feature extraction, feature aggregation and defect detection are performed in sequence to obtain a detection result, including a defect category and a defect position. The feature extraction of the detection model first extracts a priori features and local global features based on a SAM large model and a convolution operation respectively, and then embeds the priori features into the local global features to obtain multi-level industrial features, and generates a detection result guided by the SAM priori based on the multi-level industrial features. The present invention develops a more intelligent and adaptive defect detection model through the priori guidance of the SAM visual large model, so that the model can understand and utilize prior knowledge, and has the ability of self-learning and self-optimization to better adapt to the ever-changing data distribution and industrial needs.

Description

Technical Field

[0001] The present invention belongs to the field of industrial computer vision, and in particular relates to an industrial defect detection method and system based on SAM prior guidance. Background Art

[0002] In the wave of intelligent manufacturing, industrial defect detection plays a vital role; it not only ensures the quality standards of products before leaving the factory and guarantees the safety of products, but also greatly improves production efficiency; industrial defect detection can help companies cope with the complexity and diversity of industrial scenarios, solve the challenge of sample scarcity, promote technological innovation, and support the comprehensive implementation of intelligent manufacturing; more importantly, it helps promote the sustainable development of the manufacturing industry, and is the key force in promoting the development of the manufacturing industry towards a more efficient, smarter, safer and more environmentally friendly direction.

[0003] In the current industrial scenario, traditional defect detection methods are inefficient and poorly generalized, making it difficult to adapt to the needs of large-scale industrial quality inspection. Faced with a variety of industrial products and production lines, customized defect detection models are usually required to cope with different inspection tasks. Currently, many industrial defect detection models are adjusted by general visual inspection technology, combined with specific prior knowledge or designed specific modules to meet specific needs. However, this method is often unable to cope with defect detection tasks with large differences in data distribution or scarce samples. The rigidity of prior knowledge limits the model's ability to adapt to new data distributions, making it difficult to achieve dynamic adjustment and optimization. Especially in complex industrial environments, this limitation is more obvious, seriously affecting the model's detection performance and generalization ability.

[0004] Therefore, the shortcomings of the prior art can be summarized as follows:

[0005] (1) The complexity of industrial scenarios and the diversity of industrial defects reduce the generalization and detection performance of existing defect detection models.

[0006] (2) Industrial defect images contain large amounts of noise information, and existing models tend to ignore edge texture information during the multi-level feature fusion process.

[0007] (3) The scales of different industrial defects vary dramatically, and the semantic information of defect foreground is weak. It is difficult for existing models to achieve local and global simultaneous modeling. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an industrial defect detection method and system based on SAM prior guidance, which not only significantly improves the detection efficiency and generalization ability, promotes the innovation of industrial defect detection technology, but also supports the development of the manufacturing industry towards a more efficient, intelligent, safe and environmentally friendly direction, and provides key technical support for the comprehensive implementation of intelligent manufacturing, while ensuring product quality and safety and promoting sustainable development.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] A first aspect of the present invention provides an industrial defect detection method based on SAM prior guidance.

[0011] An industrial defect detection method based on SAM prior guidance, comprising:

[0012] Acquire an image of industrial surface defects to be detected;

[0013] The industrial surface defect image is input into the trained detection model, and then undergoes feature extraction, feature aggregation and defect detection to obtain the detection results, including defect category and defect location.

[0014] Among them, the feature extraction of the detection model first extracts prior features and local global features based on the SAM large model and convolution operation respectively, and then embeds the prior features into the local global features to obtain multi-level industrial features. Based on the multi-level industrial features, they are sent to the multi-level feature aggregation network, and after the dual-domain feature aggregation module, the SAM prior-guided detection results are generated.

[0015] Furthermore, the detection model includes a feature extraction network, a multi-level feature aggregation network and a defect detection network;

[0016] The feature extraction network takes the industrial surface defect image as input, performs expert-guided feature extraction, and outputs multi-level industrial features;

[0017] The multi-level feature aggregation network takes multi-level industrial features as input, performs multi-level feature aggregation, and outputs aggregated features;

[0018] The defect detection network uses the aggregated features as input, performs target detection of defects, and outputs defect categories and defect locations.

[0019] Furthermore, the feature extraction network includes an expert prior encoder based on the SAM large model, a feature extraction module based on MRF-Mamba, and a prior embedding module;

[0020] The expert prior encoder is used to extract and provide domain prior knowledge with significant industrial defect characteristics to obtain prior features;

[0021] The feature extraction module is used to capture and model the global semantic information and local detail features of industrial defects to obtain local global features;

[0022] The prior embedding module is used to embed the prior features from the expert prior encoder into the local global features to obtain multi-level industrial features.

[0023] Furthermore, the specific steps of the expert a priori encoder are:

[0024] First, the defect image is processed into Send to Mobile SAM encoder;

[0025] Secondly, the modules in the encoder are adapted and adjusted. After the multi-head self-attention global modeling of each layer of the Mobile SAM encoder, a priori adapter consisting of fully connected layers is introduced for the first adaptation.

[0026] Then, the prior after the first adaptation With original features Obtained through residual connection fusion ,Will The feedback layer of the frozen weights is sent to refine the features, and finally the refined prior and Fusion To ensure the robustness of the model; at the same time, It is fed into the second prior adaptation module and then through the learnable adaptation factor Adjust the contribution of the second prior and inject , complete the a priori second adaptation;

[0027] Finally, the features after secondary adaptation are sent to two layers of learnable convolutional layers to further refine the activation features. This allows the provided prior knowledge to retain the broad visual representation from the large visual model and inject the domain expression from the industrial defect features, ultimately obtaining the prior features. , the specific process is shown in the formula:

[0028]

[0029]

[0030] in, Indicates the encoder module, Represents the value obtained by the expert prior encoder Feature Block, represents the LayerNorm regularization function, represents the adaptation factor, represents the convolutional layer; represents the fully connected layer, It stands for Prior Adapter and consists of two fully connected layers and GELU activation function.

[0031] Furthermore, the feature extraction module is composed of an MRF-Mamba module and a spatial pooling module;

[0032] The MRF-Mamba module effectively models the global semantic information of industrial defects and accurately captures local detail features through a multi-scale receptive field scanning mechanism and a local attention mechanism. Specifically:

[0033] First, the features After being sent to MRF-Mamba, they are sent to 1×1 convolution modules to adjust the feature dimensions and obtain and ; At this time, the normalization operation will be performed The feature is sent to the weight-sharing linear layer for channel projection; then it is split from the channel into a channel modeling branch and a spatial modeling branch through the Split module to obtain two features and , the specific process is shown in the formula:

[0034]

[0035] The two features and They are input into the multi-scale receptive field scanning mechanism and the local attention mechanism respectively to obtain global semantic information and local detail features, specifically:

[0036] (1) Multi-scale receptive field scanning mechanism, The features are sent to the cascaded 1×1 and 3×3 convolution layers respectively to fully extract the local texture features of the blur defects. ; Then select the void ratio Different multiple convolutions will The features are divided into multiple local windows with unique receptive fields. , the local windows of different receptive fields Through the 2D scanning mechanism, the state space of texture defects can be obtained by scanning the local window of the small receptive field. , the state space of the defect edge is obtained by scanning the local window of the medium receptive field , obtain the semantic state of the complete defect by scanning the local window of the large receptive field Finally, the state space with different information is spliced ​​and aligned, and sent to the convolutional layer to learn the complex relationship between different features, and then added to the residual feature and normalized to obtain , the specific process is shown in the formula:

[0037]

[0038]

[0039]

[0040] in, represents the adaptation factor, Represents 1×1 convolution layer, regularization layer BN and SILU activation function, represents element-wise multiplication, represents the fully connected layer, Represents the GELU activation function;

[0041] (2) Local attention mechanism: Compress the spatial information of each channel into 1 in the spatial dimension, so that Next, Through the ReLU activation function, a nonlinear function is introduced to facilitate better learning; then, the activated Then, through a fully connected layer, the blurred foreground feature weights of each channel and the noise weights that discard the background redundancy are learned; finally, the learned foreground feature weights are processed nonlinearly and combined with the initial Feature multiplication to obtain the enhanced features after cleaning , the specific process is shown in the formula:

[0042]

[0043]

[0044] in, represents the spatial position of each channel, represents the RELU activation function, and represents the learnable parameters in the linear function, represents element-wise multiplication, Represents the splicing of features on the channel;

[0045] The spatial pooling module performs spatial pooling on the global semantic information and local detail features to obtain local global features, which can be expressed as:

[0046]

[0047] in, represents the spatial position of each channel, represents the RELU activation function, and represents the learnable parameters in the linear function, represents element-wise multiplication, Represents the concatenation of features on channels.

[0048] Furthermore, the specific process of the prior embedding module is:

[0049] First, the prior features and local global features are sent to the convolutional layer for feature perception learning; then the weighted function Calculate the weighted proportion of learnable parameters at each position of the parameter matrix, and multiply the corresponding updated weighted matrix with the two features respectively; splice and fuse the features with refined priors and rich representations on the channel; finally, integrate the spliced ​​features through convolution to express the rich features; the specific process is shown in the formula:

[0050]

[0051]

[0052] in, Learnable parameters, is the number of pixel positions in the feature map, is a non-zero parameter, Indicates that features are concatenated on channels.

[0053] Furthermore, the multi-level feature aggregation network includes a recursive split mapping module and a dual-domain feature aggregation module;

[0054] The recursive split mapping module splits the multi-level industrial features according to their importance to obtain hierarchical features with different channel contributions;

[0055] The dual-domain feature aggregation module aggregates multi-level industrial features and grade features in the frequency domain and the spatial domain, and uses an adaptive frequency domain filtering method based on the frequency domain feature judgment mechanism to perform feature noise reduction and edge enhancement processing to obtain aggregated features.

[0056] Furthermore, the specific process of the dual-domain feature aggregation module is as follows:

[0057] Features and hierarchical features from three different recursive splitting mechanisms , , First, the regularization function is used to make the dual-domain feature aggregation module better adapt to the deep and shallow features of different distributions; then, the deep and shallow features from the cross-layer are subjected to 2D FFT to obtain the corresponding frequency domain representation. ; Then, the frequency domain representation of the defect feature map is mapped on the frequency spectrum from low-frequency features to high-frequency features; Then, the learnable gating layer parameters are used To control the weights of different frequency features, and finally obtain the aggregated features. It is worth noting that the parameters of the gating layer can be updated using the back propagation of the network. The specific process is shown in the formula:

[0058]

[0059] .

[0060] Furthermore, the adaptive frequency domain filtering method based on the frequency domain feature judgment mechanism is specifically:

[0061] The frequency domain feature judgment mechanism is to analyze and judge in the frequency domain whether the feature after dual domain aggregation is an unclear edge foreground or too much foreground noise;

[0062] The adaptive frequency domain filtering method adaptively selects a corresponding filtering method to perform noise reduction and edge enhancement processing based on the frequency domain feature judgment result, and finally converts it to the spatial domain.

[0063] Furthermore, during the training of the detection model, the real box and the predicted box are modeled as a new scaling distribution function, which is used to dynamically adjust the CIoU loss as the network is trained through learnable dynamic parameters according to changes in the image data input. and mean square error function The proportion of the bounding box regression loss function is constructed, which is expressed as follows:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] in, and are the widths of the real box and the predicted box respectively, and are the heights of the real box and the predicted box, respectively, ( , )and( , ) represent the center coordinates of the real box and the predicted box respectively, is a learnable hyperparameter, Indicates defect data, Indicates the corresponding label, It is a bounding box regression loss function.

[0070] A second aspect of the present invention provides an industrial defect detection system based on SAM prior guidance.

[0071] An industrial defect detection system based on SAM prior guidance includes an acquisition module and a detection module:

[0072] The acquisition module is configured to: acquire an image of industrial surface defects to be detected;

[0073] The detection module is configured to: input the industrial surface defect image into the trained detection model, and sequentially perform feature extraction, feature aggregation and defect detection to obtain the detection result, including defect category and defect location;

[0074] Among them, the feature extraction of the detection model first extracts prior features and local global features based on the SAM large model and convolution operation respectively, and then embeds the prior features into the local global features to obtain multi-level industrial features. Based on the multi-level industrial features, the detection results guided by the SAM prior are generated.

[0075] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of an industrial defect detection method based on SAM prior guidance as described in the first aspect of the present invention.

[0076] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of an industrial defect detection method based on SAM prior guidance as described in the first aspect of the present invention are implemented.

[0077] One or more of the above technical solutions have the following beneficial effects:

[0078] The present invention develops a more intelligent and adaptive defect detection model through the prior guidance of the SAM visual large model, so that it can understand and utilize prior knowledge, and at the same time has the ability of self-learning and self-optimization, which significantly improves the detection efficiency and generalization ability. It can maintain high accuracy even when the data distribution difference is large or the samples are scarce, and can better adapt to the ever-changing data distribution and industrial needs.

[0079] The present invention fully integrates multi-level and cross-level prior features by designing a recursive split mapping mechanism and a dual-domain feature aggregation module, decouples the redundant relationship between shallow and deep features through the contribution of different feature channels to shallow texture and deep semantics, and improves the model's sensitivity to the edges of multi-scale defective textures.

[0080] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0082] Figure 1 This is a flow chart of the method of the first embodiment.

[0083] Figure 2 This is a structural diagram of the detection model of the first embodiment.

[0084] Figure 3 This is a structural diagram of the expert priori encoder of the first embodiment.

[0085] Figure 4 This is the MRF-Mamba module structure diagram of the first embodiment.

[0086] Figure 5 This is a structural diagram of the prior embedding module of the first embodiment.

[0087] Figure 6 This is a structural diagram of the recursive split mapping mechanism of the first embodiment.

[0088] Figure 7 This is a structural diagram of the dual-domain feature aggregation module of the first embodiment. DETAILED DESCRIPTION

[0089] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0090] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0091] Embodiment 1

[0092] In one embodiment of the present disclosure, a method for industrial defect detection based on SAM prior guidance is provided, comprising the following steps:

[0093] Step 1: Obtain the industrial surface defect image to be detected;

[0094] Step 2: Input the industrial surface defect image into the trained detection model, and then go through feature extraction, feature aggregation and defect detection to obtain the detection results, including defect category and defect location;

[0095] Among them, the feature extraction of the detection model first extracts prior features and local global features based on the SAM large model and convolution operation respectively, and then embeds the prior features into the local global features to obtain multi-level industrial features. Based on the multi-level industrial features, they are sent to the multi-level feature aggregation network, and after the dual-domain feature aggregation module, the SAM prior-guided detection results are generated.

[0096] From the above solution, we can see that how to build a detection model is the key point. The following describes in detail the implementation process of an industrial defect detection method based on SAM prior guidance from the perspective of model construction and training.

[0097] As an example, a specific implementation of an industrial defect detection method based on SAM prior guidance is as follows: Figure 1 As shown, the specific steps are:

[0098] Step S1: Collect industrial surface defect images in different scenarios, perform preprocessing operations on the images, and divide the images and label files into training sets and tests required for model training. The specific implementation process is as follows:

[0099] In order to fully improve the generalization ability and detection performance of the constructed model, 8 industrial surface defect images involving various industrial categories were used.

[0100] Specifically, it involves surface defect images of aluminum in the aluminum industry, surface defect images of PCB boards in the electronics manufacturing industry, surface defect images of liquor bottles in the glass manufacturing industry, surface defect images of copper plates in the metal industry, surface defect images of railroad tracks in the railway industry, surface defect images of auto parts in the automobile manufacturing industry, surface defect images of wood, and surface defect images of cement boards in the building materials industry.

[0101] Step S2: Build an expert prior encoder based on the SAM large model in the expert branch, dedicated to adapting the feature distribution of industrial images to the general visual corpus.

[0102] The feature extraction network takes the industrial surface defect image as input, performs expert-guided feature extraction, and outputs multi-level industrial features; in the feature extraction network, the industrial surface defect image is input into two branches: an expert branch and a detection branch, the expert branch is used to extract prior features, and the detection branch is used to extract local global features, wherein the expert branch is specifically:

[0103] like Figure 2 As shown, the expert branch belongs to one of the prior branches of the expert-guided feature extraction network, which extracts and provides domain prior knowledge with significant industrial defect characteristics, namely, prior features, through an expert prior encoder.

[0104] like Figure 3 As shown in Figure 1, the expert prior encoder consists of three parts: data preprocessing operation, pre-trained Mobile SAM encoder after distillation, and convolution layer. Among them, the prior adapter and convolution layer in the MobileSAM encoder module are learnable, and the weights of other parts in the MobileSAM encoder are frozen. The specific structure of the expert prior encoder is:

[0105] First, the defect image is processed into This step ensures that pixel-level semantic and texture information can be extracted based on the general visual corpus prior.

[0106] Secondly, the modules in the encoder are adapted and adjusted to better adapt to the information of industrial defect characteristics.

[0107] Specifically, after the global modeling of multi-head self-attention in each layer of Mobile SAM encoder, such as Figure 3 As shown in the figure, a priori adapter consisting of fully connected layers is introduced for the first adaptation, which can adjust the position information and ensure that the model not only understands the global structure of the image, but also pays attention to the approximate location of local defects.

[0108] Then, the prior after the first adaptation With original features Obtained through residual connection fusion ,Will The feedback layer of the frozen weights is sent to refine the features, and finally the refined prior and Fusion To ensure the robustness of the model. It is fed into the second prior adaptation module and then through the learnable adaptation factor Adjust the contribution of the second prior and inject , completing the second adaptation of the prior; this aims to perform a learnable adaptation of the preliminary prior features extracted from the Mobile SAM encoder to better capture details and irregularities in industrial images.

[0109] Finally, the features after secondary adaptation are sent to two layers of learnable convolutional layers to further refine the activation features. This allows the provided prior knowledge to retain the broad visual representation from the large visual model and inject the domain expression from the industrial defect features, ultimately obtaining the prior features. , the specific process is shown in formula (4) (5):

[0110] (4)

[0111] (5)

[0112] in, Indicates the encoder module, Represents the value obtained by the expert prior encoder Feature Block, represents the LayerNorm regularization function, represents the adaptation factor, represents the convolutional layer; represents the fully connected layer, It stands for Prior Adapter and consists of two fully connected layers and GELU activation function.

[0113] Step S3: Build the MRF-Mamba feature extraction module in the detection branch, aiming to more accurately capture and model the global semantic information and local detail features of industrial defects and obtain local global features. The specific implementation process is as follows:

[0114] like Figure 2 As shown in the figure, the detection branch belongs to another feature extraction branch in the expert-guided feature extraction network, which is composed of convolution operations, MRF-Mamba modules and spatial pooling modules. The image input to the detection branch is first processed by several layers of convolution operations, and then sent to the MRF-Mamba module and the spatial pooling module in turn for further feature extraction, which is used to extract subtle defects that are difficult to identify and subtle semantic defects.

[0115] The structure of the MRF-Mamba module is as follows: Figure 4 As shown in the figure, the effective modeling of the global semantic information of industrial defects and the accurate capture of local detail features are achieved mainly through the multi-scale receptive field scanning mechanism and the local attention mechanism.

[0116] Specifically, first of all, the characteristics After being sent to MRF-Mamba, they are sent to 1×1 convolution modules to adjust the feature dimensions and obtain and At this time, the normalization operation will be performed The feature is sent to the weight-sharing linear layer for channel projection to add richer spatial and channel information; then it is split from the channel into a channel modeling branch and a spatial modeling branch through the Split module to obtain two features and , the specific process is shown in formula (6):

[0117] (6)

[0118] The two features and They are input into the multi-scale receptive field scanning mechanism and the local attention mechanism respectively to obtain global semantic information and local detail features, specifically:

[0119] (1) Multi-scale receptive field scanning mechanism, The features are sent to the cascaded 1×1 and 3×3 convolution layers respectively to fully extract the local texture features of the blur defects. ; Then select the void ratio Different multiple convolutions will The features are divided into multiple local windows with unique receptive fields. , the local windows of different receptive fields Through the 2D scanning mechanism, the state space of texture defects can be obtained by scanning the local window of the small receptive field. , the state space of the defect edge is obtained by scanning the local window of the medium receptive field , obtain the semantic state of the complete defect by scanning the local window of the large receptive field Finally, the state space with different information is spliced ​​and aligned, and sent to the convolutional layer to learn the complex relationship between different features, and then added to the residual feature and normalized to obtain ; This greatly improves the compact arrangement of local subtle feature markers while retaining the global semantic features after state space scanning. The specific process is shown in formulas (7), (8), and (9):

[0120] (7)

[0121] (8)

[0122] (9)

[0123] in, represents the adaptation factor, Represents 1×1 convolution layer, regularization layer BN and SILU activation function, represents element-wise multiplication, represents the fully connected layer, Represents the GELU activation function.

[0124] (2) Local attention mechanism to model the characteristics of the channel branch output , a channel-by-channel local attention is designed to enhance the fuzzy features while filtering out redundant background noise.

[0125] Specifically, Compress the spatial information of each channel into 1 in the spatial dimension, so that Next, Through the ReLU activation function, a nonlinear function is introduced to facilitate better learning; then, the activated Then, through a fully connected layer, the blurred foreground feature weights of each channel and the noise weights that discard the background redundancy are learned; finally, the learned foreground feature weights are processed nonlinearly and combined with the initial Feature multiplication to obtain the enhanced features after cleaning , the specific process is shown in formula (10) (11):

[0126] (10)

[0127] (11)

[0128] in, represents the spatial position of each channel, represents the RELU activation function, and represents the learnable parameters in the linear function, Represents element-wise multiplication.

[0129] Formula (12) shows the complete working process of the MRF-Mamba module:

[0130] (12)

[0131] in, represents the spatial position of each channel, represents the RELU activation function, and represents the learnable parameters in the linear function, represents element-wise multiplication, Represents the concatenation of features on channels.

[0132] Step S4: Build a priori embedding module in the expert-guided feature extraction network, and aim to embed the prior features from the expert prior encoder into each layer of industrial features of the MRF-Mamba feature extraction module to obtain multi-level industrial features. The specific implementation process is as follows:

[0133] like Figure 5 As shown in Figure 1, the prior features and local global features are sent to the convolution layer for feature perception learning. At the same time, the learnable parameter matrix is ​​initialized according to the dimensions of these two features; then the weighted function Calculate the weighted proportion of learnable parameters at each position of the parameter matrix, and multiply the corresponding updated weighted matrix by the two features respectively; splice and fuse the features with refined priors and rich representations on the channel; finally, integrate the spliced ​​features with rich feature expressions through convolution; in this way, the pixel-level position priors can be embedded into the detection branch to improve defect positioning, and the semantically rich feature representations can be passed to the multi-level feature aggregation network; the weighted fusion strategy subtly reflects the guiding effect of the large-scale model prior knowledge on the detection model, giving the model a strong adaptability to the changing industrial environment, thereby significantly improving its robustness in the industrial defect detection task. The specific process is shown in formulas (13) and (14):

[0134] (13)

[0135] (14)

[0136] in, Learnable parameters, is the number of pixel positions in the feature map, is a non-zero parameter, Indicates that features are concatenated on channels.

[0137] Step S5: In the multi-level feature aggregation network, a recursive split mapping mechanism is innovated to obtain the contribution of different feature channels to shallow texture and deep semantics, so as to decouple the redundant relationship between shallow and deep features. The specific implementation process is as follows:

[0138] In order to improve the model's adaptability to defects of different scales and irregular shapes, and thus enhance the model's robustness in diverse industrial product defect detection tasks, the MRF-Mamba module is continued to be used in the multi-level feature aggregation network, and the multi-level and cross-level prior features are fully integrated by designing the recursive split mapping mechanism RS-Mapping and the dual-domain feature aggregation DDFA module; Figure 2 As shown in the figure, the recursive splitting mechanism is to split the deep, shallow and other multi-level industrial features according to their importance (i.e., contribution) in the multi-level feature aggregation network.

[0139] The recursive split mapping mechanism, such as Figure 6 As shown, first, the feature Perform augmentation mapping on the channel and then split it through a recursive splitting function Get split channels of different sizes; select the smallest split channel as the residual feature Keep the remaining channel features as the main features for centralized learning processing .

[0140] Then, the main features Passed separately and Two-path processing, Recursive splitting function is used again in the path Will Split into m split features with different number of channels; In the path, a large kernel 7×7 convolution is first used to extract features from each channel to obtain the contribution of different channels to shallow texture edges and deep semantic features. This can decouple the redundant relationship between shallow and deep features, so that the model can better understand the relationship between defects and background. Then, the recursive splitting function is also used. Will Split into m features with channel contributions.

[0141] Finally, and The split features of the path are multiplied accordingly, and the residual features are Injected into the split feature branch with the same number of channels, the final result is , , Three level features with different channel contributions are selected as the input of the dual-domain feature aggregation module. The specific process is shown in formulas (15), (16), and (17):

[0142] (15)

[0143] (16)

[0144] , , (17)

[0145] Where C represents the number of channels, represents feature multiplication, represents the splitting function.

[0146] Step S6: A dual-domain feature aggregation module is built in the multi-level feature aggregation network, which is dedicated to denoising and edge enhancement of different levels of prior features in the spatial domain and frequency domain, and then aggregating the processed results to improve the model's sensitivity to multi-scale defect texture edges. The specific implementation process is as follows:

[0147] like Figure 2 As shown in the figure, according to the neighboring layer features and the number of features of different levels across layers transmitted from the MRF-Mamba module and the recursive splitting mechanism, the feature aggregation module is divided into a four-input fusion stage I and a three-input fusion stage II.

[0148] like Figure 7 As shown, specifically, in the first stage of fusion, Features and hierarchical features from three different recursive splitting mechanisms , , First, the regularization function is used to make the dual-domain feature aggregation module better adapt to the deep and shallow features of different distributions; then, the deep and shallow features from the cross-layer are subjected to 2D FFT (Fourier transform) to obtain the corresponding frequency domain representation. Then, the frequency domain representation of the defect feature map is mapped on the frequency spectrum from low-frequency features to high-frequency features; then, the learnable gating layer parameters are used To control the weight of different frequency features, the dual-domain feature aggregation module can be properly adjusted to capture the defect edge and size change law of shallow features. It is worth noting that the parameters of the gate layer can be updated using the back propagation of the network, which is expressed as:

[0149] (18)

[0150] (19)

[0151] In addition, in order to design a strong generalization detection model that needs to accurately identify any industrial product defects, it is used to filter the features after dual-domain aggregation to improve the accuracy of the features. For this purpose, a frequency domain feature judgment mechanism and adaptive frequency domain filtering are created. The frequency domain feature judgment mechanism is used to analyze and judge whether the image features are unclear edge foreground or excessive foreground noise in the frequency domain. Based on the frequency domain feature judgment results, the adaptive frequency domain filtering method is used to adaptively select the corresponding filtering method for noise reduction and edge enhancement processing, specifically:

[0152] First, the energy spectrum represented in the frequency domain is calculated;

[0153] Then, the statistical characteristics of the frequency domain image energy spectrum are calculated and ;

[0154] Then, set the learnable threshold parameter and As a benchmark for judging the energy spectrum, as shown in formula (22), the model will adaptively select different filtering methods for processing based on the judgment result of the threshold. and When it is larger, it is judged that there are more high-frequency components, that is, the foreground noise in the image is larger. and When it is small, it is judged that there are many low-frequency components, that is, the edge foreground in the image is not obvious.

[0155] Then, adaptive frequency domain filtering is used to filter the frequency domain features. The amplitude spectrum and phase spectrum are filtered separately, by hindering the intervention of high-frequency filtering of the amplitude spectrum and low-frequency filtering features of the phase spectrum, aiming to reduce the background noise in the industrial defect image and enhance the edge texture of the blurred features.

[0156] The formula is:

[0157] (20)

[0158] (twenty one)

[0159] (twenty two)

[0160] (twenty three)

[0161] Finally, the 2D IFFT (inverse Fourier transform) layer is used to map the frequency domain features to the spatial domain features; the features after frequency domain adaptation and noise reduction The adaptive average pooling layer is then fused and adapted with the original spatial domain features, and the output is obtained after the convolution layer. This ensures the progressive fusion of fine-grained information by performing noise reduction and edge enhancement on the cross-layer features in the dual domains, which can be expressed as:

[0162] (twenty four)

[0163] in, and are the width and height of the image, corresponding to the number of pixels in the spatial domain; and are the horizontal and vertical frequency components in the frequency domain, and Standard deviation parameter of the filter; controls the frequency range of the filter; Indicates pixel value multiplication, and represents a frequency variable, and and Related, is an imaginary unit, and are the spatial coordinates of the image respectively.

[0164] Step S7: Complete the model building process of the expert-guided feature extraction network, feature aggregation network and detector. Use the designed scaling distribution function in the model training process, and finally select the model with the best parameter weight for industrial product surface defect detection. The specific implementation process is as follows:

[0165] Considering that tiny target datasets and datasets with large scale variations are common in complex industrial scenarios, a scaling distribution function is designed to ensure that the model can adaptively and dynamically identify defective objects of various sizes.

[0166] It is mainly reflected in the calculation method of the distance between the two boxes. Specifically, when the defect size in the input image does not change much, the network often does not pay much attention to the fine-grained similarity between the objects. At this time, using the IoU value to measure the distance between the two boxes is very effective; when the defect size in the image involved is small and these small objects vary greatly in scale, by focusing on using the normalized mean square error function To measure the distribution similarity of two boxes.

[0167] Specifically, the true box and the predicted box are modeled as a new scaling distribution function , which is used to adjust the image data according to the changes in the above two image data input conditions through learnable dynamic parameters To dynamically adjust as the network is trained and The proportion of the bounding box regression loss function is constructed , to better adapt to different defect samples, specifically:

[0168] First, define the aspect ratio of the true box and the predicted box to introduce a scaling factor into the distribution function. and , so that it has good balance for boxes of different sizes, aiming to gradually reduce the weight of important pixels in the defined box from the center to the boundary of the bounding box, so that the network can be more sensitive to the distribution of changes in small defects and not affected by scale changes, which can be expressed as:

[0169] (25)

[0170] (26)

[0171] (27)

[0172] (28)

[0173] (29)

[0174] in, and are the widths of the real box and the predicted box respectively, and are the heights of the real box and the predicted box respectively, ( , )and( , ) represent the center coordinates of the real box and the predicted box respectively, is a learnable hyperparameter, Indicates defect data, Indicates the corresponding label, It is a bounding box regression loss function.

[0175] Finally, the model can be trained in an end-to-end manner by jointly optimizing all relevant losses, and the overall loss function is shown in formula (31):

[0176] (30)

[0177] (31)

[0178] in, Indicates defect data, Indicates the corresponding label, is the cross entropy loss function, Indicates category information.

[0179] After the model has been trained for a preset number of rounds, the best performing model is evaluated through the test set and its corresponding weight file is retained to ensure that the model can accurately identify various industrial defects in actual industrial production line applications.

[0180] Embodiment 2

[0181] In one embodiment of the present disclosure, an industrial defect detection system based on SAM prior guidance is provided, including an acquisition module and a detection module:

[0182] The acquisition module is configured to: acquire an image of industrial surface defects to be detected;

[0183] The detection module is configured to: input the industrial surface defect image into the trained detection model, and sequentially perform feature extraction, feature aggregation and defect detection to obtain the detection result, including defect category and defect location;

[0184] Among them, the feature extraction of the detection model first extracts prior features and local global features based on the SAM large model and convolution operation respectively, and then embeds the prior features into the local global features to obtain multi-level industrial features. Based on the multi-level industrial features, the detection results guided by the SAM prior are generated.

[0185] Embodiment 3

[0186] The purpose of this embodiment is to provide a computer-readable storage medium.

[0187] A computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of an industrial defect detection method based on SAM prior guidance as described in the first embodiment of the present disclosure.

[0188] Embodiment 4

[0189] The purpose of this embodiment is to provide an electronic device.

[0190] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of an industrial defect detection method based on SAM prior guidance as described in the first embodiment of the present disclosure are implemented.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An industrial defect detection method based on SAM prior guidance, characterized in that: include: Acquire an image of industrial surface defects to be detected; The industrial surface defect image is input into the trained detection model, and then undergoes feature extraction, feature aggregation and defect detection to obtain the detection results, including defect category and defect location. Among them, the feature extraction of the detection model first extracts prior features and local global features based on the SAM large model and convolution operation respectively, and then embeds the prior features into the local global features to obtain multi-level industrial features. Based on the multi-level industrial features, they are sent to the multi-level feature aggregation network, and after the dual-domain feature aggregation module, the detection results guided by the SAM prior are generated; The detection model includes a feature extraction network, a multi-level feature aggregation network and a defect detection network; The feature extraction network takes the industrial surface defect image as input, performs expert-guided feature extraction, and outputs multi-level industrial features; The multi-level feature aggregation network takes multi-level industrial features as input, performs multi-level feature aggregation, and outputs aggregated features; The defect detection network uses the aggregated features as input, performs target detection of defects, and outputs defect categories and defect locations; The feature extraction network includes an expert prior encoder based on the SAM large model, a feature extraction module based on MRF-Mamba, and a prior embedding module; The expert prior encoder is used to extract and provide domain prior knowledge with significant industrial defect characteristics to obtain prior features; The feature extraction module is used to capture and model the global semantic information and local detail features of industrial defects to obtain local global features; The prior embedding module is used to embed the prior features from the expert prior encoder into the local global features to obtain multi-level industrial features; The multi-level feature aggregation network includes a recursive split mapping module and a dual-domain feature aggregation module; The recursive split mapping module splits the multi-level industrial features according to their importance to obtain hierarchical features with different channel contributions; The dual-domain feature aggregation module aggregates multi-level industrial features and grade features in the frequency domain and the spatial domain, and uses an adaptive frequency domain filtering method based on the frequency domain feature judgment mechanism to perform feature noise reduction and edge enhancement processing to obtain aggregated features.

2. The industrial defect detection method based on SAM prior guidance as claimed in claim 1, characterized in that: The specific steps of the expert a priori encoder are: First, the defect image is processed into Sent to the MobileSAM encoder, where C represents the number of channels; Secondly, the modules in the encoder are adapted and adjusted. After the multi-head self-attention global modeling of each layer of the Mobile SAM encoder, a priori adapter consisting of fully connected layers is introduced for the first adaptation. Then, the prior after the first adaptation With original features Obtained through residual connection fusion ,Will The feedback layer of the frozen weights is sent to refine the features, and finally the refined prior and Fusion To ensure the robustness of the model; at the same time, It is fed into the second prior adaptation module and then through the learnable adaptation factor Adjust the contribution of the second prior and inject , complete the a priori second adaptation; Finally, the features after secondary adaptation are sent to two layers of learnable convolutional layers to further refine the activation features. This allows the provided prior knowledge to retain the broad visual representation from the large visual model and inject the domain expression from the industrial defect features, ultimately obtaining the prior features. , the specific process is shown in the formula: in, Indicates the encoder module, Represents the value obtained by the expert prior encoder Feature Block, represents the LayerNorm regularization function, represents the adaptation factor, represents the convolutional layer; represents the fully connected layer, It stands for Prior Adapter and consists of two fully connected layers and GELU activation function.

3. The industrial defect detection method based on SAM prior guidance as claimed in claim 1, characterized in that: The feature extraction module is composed of an MRF-Mamba module and a spatial pooling module; The MRF-Mamba module effectively models the global semantic information of industrial defects and accurately captures local detail features through a multi-scale receptive field scanning mechanism and a local attention mechanism. Specifically: First, the features After being sent to MRF-Mamba, they are sent to 1×1 convolution modules to adjust the feature dimensions and obtain and ; At this time, the normalization operation will be performed The feature is sent to the weight-sharing linear layer for channel projection; then it is split from the channel into a channel modeling branch and a spatial modeling branch through the Split module to obtain two features and , the specific process is shown in the formula: The two features and They are input into the multi-scale receptive field scanning mechanism and the local attention mechanism respectively to obtain global semantic information and local detail features, specifically: (1) Multi-scale receptive field scanning mechanism, The features are sent to the cascaded 1×1 and 3×3 convolution layers respectively to fully extract the local texture features of the blur defects. ; Then select the void ratio Different multiple convolutions will The features are divided into multiple local windows with unique receptive fields. , the local windows of different receptive fields Through the 2D scanning mechanism, the state space of texture defects is obtained by scanning the local window of the small receptive field. , the state space of the defect edge is obtained by scanning the local window of the medium receptive field , obtain the semantic state of the complete defect by scanning the local window of the large receptive field Finally, the state space with different information is spliced ​​and aligned, and sent to the convolutional layer to learn the complex relationship between different features, and then added to the residual feature and normalized to obtain , the specific process is shown in the formula: in, represents the void ratio; (2) Local attention mechanism: Compress the spatial information of each channel into 1 in the spatial dimension, so that Next, Through the ReLU activation function, a nonlinear function is introduced to facilitate better learning; then, the activated Then, through a fully connected layer, the blurred foreground feature weights of each channel and the noise weights that discard the background redundancy are learned; finally, the learned foreground feature weights are processed nonlinearly and combined with the initial Feature multiplication to obtain the enhanced features after cleaning , the specific process is shown in the formula: in, represents the spatial position of each channel, represents the RELU activation function, and represents the learnable parameters in the linear function, represents element-wise multiplication, Represents the splicing of features on the channel; The spatial pooling module performs spatial pooling on the global semantic information and the local detail features to obtain the local global features, which can be expressed as: 。 4. The industrial defect detection method based on SAM prior guidance as claimed in claim 1, characterized in that: The specific process of the prior embedding module is: First, the prior features and local global features are sent to the convolutional layer for feature perception learning; then the weighted function Calculate the weighted proportion of learnable parameters at each position in the parameter matrix, and multiply the corresponding updated weighted matrix with the two features respectively; concatenate and fuse the features with refined priors and rich representations on the channel; finally, convolve the concatenated features to integrate the rich feature expressions; The specific process is shown in the formula: in, Learnable parameters, is the number of pixel positions in the feature map, is a non-zero parameter, Indicates that features are concatenated on channels.

5. The industrial defect detection method based on SAM prior guidance as claimed in claim 1, characterized in that: The specific process of the dual-domain feature aggregation module is as follows: Features and hierarchical features from three different recursive splitting mechanisms , , First, the regularization function is used to make the dual-domain feature aggregation module better adapt to the deep and shallow features of different distributions; then, the deep and shallow features from the cross-layer are subjected to 2D FFT to obtain the corresponding frequency domain representation. ; Then, the frequency domain representation of the defect feature map is mapped on the frequency spectrum from low-frequency features to high-frequency features; Then, the learnable gating layer parameters are used To control the weights of different frequency features, and finally obtain the aggregated features. It is worth noting that the parameters of the gating layer are updated using the back propagation of the network. The specific process is shown in the formula: 。 6. The industrial defect detection method based on SAM prior guidance as claimed in claim 1, characterized in that: The adaptive frequency domain filtering method based on the frequency domain feature judgment mechanism is specifically: The frequency domain feature judgment mechanism is to analyze and judge in the frequency domain whether the feature after dual domain aggregation is an unclear edge foreground or too much foreground noise; The adaptive frequency domain filtering method adaptively selects a corresponding filtering method to perform noise reduction and edge enhancement processing based on the frequency domain feature judgment result, and finally converts it to the spatial domain.

7. The industrial defect detection method based on SAM prior guidance as claimed in claim 1, characterized in that: During the training process of the detection model, the real box and the predicted box are modeled as a new scaling distribution function, which is used to dynamically adjust the CIoU loss as the network is trained through learnable dynamic parameters according to changes in the image data input. and mean square error function The proportion of the bounding box regression loss function is constructed, which is expressed as follows: in, and are the widths of the real box and the predicted box respectively, and are the heights of the real box and the predicted box respectively, ( , )and( , ) represent the center coordinates of the real box and the predicted box respectively; is a learnable hyperparameter, Indicates defect data, Indicates the corresponding label, It is a bounding box regression loss function.

Images