Small sample industrial fault detection method and system based on signal-vision cooperation

Abstract

The invention relates to a data processing technology, in particular to a small sample industrial fault detection method and system based on signal-vision collaboration. The method comprises the following steps: designing a cross-domain alignment module, and introducing a self-supervised visual feature aligner to realize feature alignment of a target domain; constructing a domain self-adaption device, jointly optimizing the domain self-adaption device through cross-domain distribution alignment loss and feature reconstruction loss, obtaining target domain alignment features and source domain self-adaption features, and storing the target domain alignment features and the source domain self-adaption features into a visual feature library; based on a signal-visual fusion feature obtained after the global visual feature and the signal feature are fused, obtaining a text feature and storing the text feature into a text feature library; extracting low-layer, high-layer and global visual features of the to-be-detected image, comparing the low-layer, high-layer and global visual features with the visual feature library to obtain low-layer and high-layer feature scores, and performing weighted average to obtain a visual score; comparing the global visual features with a text feature library to obtain text scores; and fusing the visual score and the text score to realize anomaly detection of the to-be-detected image. According to the invention, the detection robustness is obviously improved.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a small sample industrial fault detection method and system based on signal-vision collaboration. Background Art

[0002] Small-sample anomaly detection has important applications in fields such as industrial quality inspection and medical image analysis. However, its core challenge lies in achieving high-precision detection using only a small number of normal samples and overcoming the problem of feature distribution shift in cross-domain scenarios. Although traditional methods such as PromptAD reduce reliance on abnormal samples through prompt learning, their feature learning mechanisms and cross-modal fusion strategies still have limitations: First, the text feature library relies solely on the concatenation of learnable normal and abnormal features, lacking explicit modeling of the diversity of normal samples; second, current methods are limited to anomaly detection within a single domain, making it difficult to effectively achieve fine-grained distribution matching from the source domain to the target domain; third, the score fusion strategy for visual and text modalities is simple and fails to fully exploit cross-modal correlation information.

[0003] It can be seen that existing small-sample cross-domain anomaly detection technologies often face challenges such as insufficient feature diversity, poor cross-domain adaptability, and limited modal interaction when relying on a small number of normal samples. Summary of the Invention

[0004] In order to solve the problems existing in the existing technology of small sample cross-domain anomaly detection, the present invention proposes a small sample industrial fault detection method and system based on signal-vision collaboration.

[0005] In an embodiment of the present invention, a small sample industrial fault detection method based on signal-vision collaboration is first provided, comprising the following steps:

[0006] S1. Design a cross-domain alignment module and introduce a self-supervised visual feature aligner. By constructing sample pairs on the source domain to train the self-supervised visual feature aligner, we can achieve feature alignment in the target domain.

[0007] S2. Design a two-stage domain adaptation framework, build a domain adaptor, and jointly optimize the domain adaptor by cross-domain distribution alignment loss and feature reconstruction loss to migrate source domain features to the target domain space, achieve feature space alignment, obtain target domain alignment features and source domain adaptation features, and store them in the visual feature library;

[0008] S3. Through multimodal feature fusion of global visual features and signal features, the spatial morphological features and time-frequency domain features of the anomaly are simultaneously captured to obtain signal-visual fusion features. The text feature learning module is trained using the signal-visual fusion features, and text is obtained from the trained text feature learning module. The text is then input into the pre-trained text encoder to obtain text features, and the text features are stored in the text feature library.

[0009] S4. Extract low-level, high-level, and global visual features of the image to be tested. Use the nearest neighbor search method to compare the extracted low-level and high-level visual features with the visual feature library through the previously trained feature aligner to obtain low-level feature scores and high-level feature scores, and then perform weighted average to obtain the visual score; then use the nearest neighbor search method to compare the extracted global visual features with the text feature library to obtain the text score; based on the cross-self-attention mechanism, fuse the visual score and text score to realize anomaly detection of the image to be tested.

[0010] The embodiment of the present invention further provides a small sample industrial fault detection system based on signal-vision collaboration, including the following modules:

[0011] Feature alignment module, which is used to design a cross-domain alignment module and introduce a self-supervised visual feature aligner. By constructing sample pairs on the source domain to train the self-supervised visual feature aligner, it can achieve feature alignment in the target domain.

[0012] The visual feature processing module is used to design a two-stage domain adaptation framework, build a domain adaptor, and jointly optimize the domain adaptor through cross-domain distribution alignment loss and feature reconstruction loss to migrate source domain features to the target domain space, achieve feature space alignment, obtain target domain alignment features and source domain adaptation features, and store them in the visual feature library;

[0013] The fusion module is used to fuse the multimodal features of global visual features and signal features to simultaneously capture the spatial morphological features and time-frequency domain features of the anomaly and obtain signal-visual fusion features. The signal-visual fusion features are used to train the text feature learning module, obtain text from the trained text feature learning module, input it into the pre-trained text encoder to obtain text features, and store the text features in the text feature library.

[0014] The anomaly detection module extracts low-level, high-level, and global visual features of the image to be tested, and compares the extracted low-level and high-level visual features with the visual feature library through the previously trained feature aligner using the nearest neighbor search method to obtain low-level feature scores and high-level feature scores, which are then weighted averaged to obtain a visual score. The extracted global visual features are then compared with the text feature library using the nearest neighbor search method to obtain a text score. The visual score and text score are fused based on the cross-self-attention mechanism to achieve anomaly detection in the image to be tested.

[0015] Compared with the existing technology, the present invention proposes a small sample industrial fault anomaly detection method and system based on signal-vision collaboration, which achieves the following technical effects:

[0016] 1. Design a cross-domain alignment module, introduce a self-supervised visual feature aligner, use source domain samples to train feature maps, and achieve feature alignment in the target domain.

[0017] 2. By jointly optimizing the domain adaptor through cross-domain distribution alignment loss and feature reconstruction loss, the source domain features are forced to migrate to the target domain space, alleviating the distribution differences between domains.

[0018] 3. A signal-visual feature fusion mechanism is introduced into the text learner, which supports the splicing and fusion of signal features extracted from the signal data through a trained multi-scale twin network with visual features, jointly trains text features, optimizes the text feature generation process, and significantly enhances the diversity modeling capabilities of normal samples.

[0019] 4. Design an improved text feature learning module, introduce manual normal features, splice and fuse manual features with learnable features to obtain total normal features, and strengthen the semantic difference between normal features and abnormal features through comparative learning of total normal features and total abnormal features; at the same time, through the dual constraint mechanism of total normal features approaching normal signal-visual fusion features and total abnormal features moving away from normal signal-visual fusion features, improve the discriminative ability of feature expression, and store total normal features and total abnormal features in the feature library to improve the completeness of feature expression.

[0020] 5. A dynamic multimodal decision fusion method is proposed, which adopts a cross-modal attention fusion network and dynamically integrates the abnormal responses of the visual and text feature libraries through the self-attention mechanism to replace the traditional harmonic averaging strategy and linear fusion, significantly improving the detection robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a small sample industrial fault detection method according to an embodiment of the present invention;

[0022] Figure 2 is a training flow chart of a self-supervised visual feature aligner according to an embodiment of the present invention;

[0023] Figure 3 is a flow chart of feature space alignment based on a two-stage domain adaptor in an embodiment of the present invention;

[0024] Figure 4 is a flow chart of multimodal feature fusion of visual features and signal features in an embodiment of the present invention;

[0025] Figure 5 is a flowchart of an implementation of an enhanced text learner in an embodiment of the present invention;

[0026] Figure 6 is a flow chart of anomaly detection in an embodiment of the present invention;

[0027] Figure 7This is a flowchart of the self-attention mechanism fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and embodiments, but the embodiments of the present invention are not limited thereto.

[0029] Example

[0030] This embodiment proposes a small sample industrial fault detection method based on signal-vision collaboration, such as Figure 1 As shown, the specific steps include:

[0031] S1. Design a cross-domain alignment module and introduce a self-supervised visual feature aligner. By constructing sample pairs on the source domain to train the self-supervised visual feature aligner, we can achieve feature alignment in the target domain.

[0032] A cross-domain alignment module is designed, a self-supervised visual feature aligner is introduced, and source domain samples are used to map training features. That is, by constructing sample pairs on the source domain to train the self-supervised visual feature aligner, feature alignment of the target domain is achieved.

[0033] The specific implementation process of this step includes: constructing source domain image pairs, obtaining deep representations through a pre-trained feature extractor, and then calculating the contrastive loss through a feature aligner to update the parameters of the self-supervised visual feature aligner.

[0034] During training, if Figure 2 As shown in the figure, a pair of normal images of the same category are selected from the source domain, and their image features are extracted through the pre-trained backbone network respectively. The extracted image features are then input into the visual feature aligner to calculate the alignment loss to train the self-supervised visual feature aligner.

[0035] In this embodiment, the pre-trained backbone network uses a pre-trained visual encoder. To overcome the cross-level semantic gap of the pre-trained visual encoder, a multi-granularity feature consistency alignment mechanism is proposed. The specific training process includes the following steps:

[0036] S11. Cross-level feature extraction.

[0037] Given a pair of normal images (x1, x2) of the same category from the source domain, multi-level features, i.e. low-level features and high-level features, are extracted through the pre-trained visual encoder:

[0038] The low-level features are: F1 = Encoder low (x1), F3 = Encoder low (x2);

[0039] High-level features are: F2 = Encoder high(x1), F4 = Encoder high (x2).

[0040] S12. Adaptive feature projection.

[0041] We design a self-supervised visual feature aligner PRED(·) to perform adaptive feature projection on multi-level features:

[0042] P3,P4=PRED(F3,F4)

[0043] S13, hierarchical comparison constraints, constructing bidirectional similarity maximization loss L align :

[0044]

[0045] in, Where N is the batch size, the margin parameter m introduces a regularization constraint to prevent overfitting, and cos(·) calculates the cosine similarity.

[0046] S2. Design a two-stage domain adaptation framework, build a domain adaptor, and jointly optimize the domain adaptor by using cross-domain distribution alignment loss (MMD) and feature reconstruction loss (L2 regularization) to migrate source domain features to the target domain space, achieve feature space alignment, alleviate inter-domain distribution differences, obtain target domain alignment features and source domain adaptive features, and store them in the visual feature library.

[0047] In this embodiment, to address the problem of cross-domain (i.e., source domain data → target domain data) feature offset, a two-stage domain adaptation strategy is adopted and a two-stage domain adaptation framework is designed. The maximum mean difference loss and feature reconstruction loss are combined to drive the potential space alignment of the source domain and the target domain.

[0048] Specifically, a normal image is selected from the support set of the source domain and the target domain respectively, and their image features are first extracted through the pre-trained visual encoder. Then, the source domain alignment feature and the target domain alignment feature are obtained through the self-supervised visual feature aligner trained in step S1, and the target domain alignment feature is stored in the visual feature library; the source domain alignment feature is input into the two-stage domain adaptor to obtain the source domain adaptive feature; the adaptive loss of the source domain adaptive feature and the target domain alignment feature is calculated through distribution moment matching (MMD), and the reconstruction loss of the residual feature of the source domain adaptive feature and the source domain alignment feature is calculated, so as to optimize the two-stage domain adaptor and realize the feature space alignment from the source domain to the target domain, and the finally optimized source domain adaptive feature is also stored in the visual feature library.

[0049] In this embodiment, if Figure 3 As shown, this step may specifically include the following steps:

[0050] S21. Obtain source domain alignment features and target domain alignment features through the pre-trained visual encoder and the trained self-supervised visual feature aligner.

[0051] Let F s1 、F s2 is the alignment feature of the source domain (normal sample), extracted by the pre-trained visual encoder and trained by the feature aligner.

[0052] And set F t1 、F t2 The alignment features of the target domain support set (the very few normal samples in the test samples) are extracted by the pre-trained visual encoder and trained by the feature aligner.

[0053] S22. Process the source domain alignment features through the domain adaptor G(·) to obtain source domain adaptive features:

[0054]

[0055] S23. Construct a multi-objective optimization function based on adaptive loss and feature reconstruction loss.

[0056] The adaptive loss is calculated using the maximum mean difference (MMD) loss, and the maximum mean difference (MMD) loss L mmd The maximum mean difference is used to minimize the distribution difference between source domain features and target domain features.

[0057]

[0058] In the formula To calculate the Gaussian kernel function.

[0059] Feature reconstruction loss L recon By constraining the difference between the source domain adaptive features output by the domain adaptor and the original input features, the semantic information and key structures of the source domain samples are preserved during cross-domain transfer. This can be calculated using the mean squared error loss.

[0060]

[0061] The constructed multi-objective optimization function L adapt , that is, the overall optimization goal is:

[0062] L adapt =λ1L mmd +λ2L recon

[0063] Where λ1 and λ2 are weight coefficients.

[0064] S3. Through the multimodal feature fusion of global visual features and signal features, the spatial morphological features and time-frequency domain features of the abnormality are captured simultaneously to obtain the signal-visual fusion features; the text feature learning module is trained through the signal-visual fusion features, and the text is obtained from the trained text feature learning module, and then input into the pre-trained text encoder to obtain text features, and the text features are stored in the text feature library.

[0065] In view of the limitation of the existing anomaly detection methods based on visual features that are not sensitive enough to subtle or dynamically evolving anomaly patterns, this step proposes a signal-visual multimodal feature collaborative modeling mechanism to enhance the ability to capture subtle anomalies and dynamic defects. Specifically, some anomalies in industrial scenes (such as surface microcracks, localized corrosion, etc.) may be difficult to identify in images due to lighting, occlusion or resolution limitations, but they often present specific spectral characteristics or time domain evolution patterns in the signal dimension (such as abnormal harmonic components of vibration signals, gradual trends in temperature signals, etc.). By fusing visual features with signal features, the present invention can simultaneously capture the spatial morphological characteristics and time-frequency domain characteristics of anomalies, and achieve comprehensive detection of multi-scale defects in complex industrial scenes.

[0066] Specifically, the following steps are included:

[0067] S31. Extract sample features of positive and negative sample pairs through multi-scale feature extraction and fusion network.

[0068] First, randomly select two samples from the normal signal samples in the training set to construct a positive sample pair, and use x i ,x j Represented by, and select a sample from the normal sample set and the abnormal sample set to construct a negative sample pair, using x i ,x k Indicates that the multi-scale feature extraction and fusion network is trained. The multi-scale feature extraction and fusion network includes a 1D-CNN, a batch normalization (BN) layer, a linear rectifier unit (ReLU) layer, a 2×1 maximum pooling layer, and a dropout layer.

[0069] like Figure 4 As shown, the specific implementation includes the following steps:

[0070] (1) A dual-branch convolutional layer is used to extract sample features.

[0071] For the convolutional layer with a kernel size of 1×s (s∈{3,7}), the output sample features are:

[0072]

[0073] in is the 1×s convolution kernel parameter of the jth channel of the nth layer, β is the number of channels, fReLU (·) represents the ReLU activation function, is the bias term, x∈{x i ,x j ,x k}.

[0074] (2) Pooling and Dropout regularization processing.

[0075] The sample features output after pooling and regularization are:

[0076]

[0077] (3) Feature Fusion

[0078] The output features after fusion are:

[0079]

[0080] Where ⊕ represents fusion.

[0081] S32. The extracted sample features are fused through different modules through a multi-scale feature extraction and fusion network to obtain multi-scale fusion features.

[0082] In this step, the W-BAM module can be used for multi-scale feature fusion. The W-BAM module mainly includes the channel attention module (CAM) and the spatial attention module (SAM). The channel attention module CAM focuses on learning the importance of different channels, while the spatial attention module SAM highlights the important spatial areas in the feature map. At the same time, different weights are assigned to CAM and SAM respectively, so that the model can dynamically balance the contributions of channel information and spatial information. This design enhances the model's ability to capture channel features and spatial features. By integrating channel attention and spatial attention, the W-BAM module can adaptively focus on key areas and channels, thereby improving the performance of the model in various tasks.

[0083] The output can be expressed as:

[0084] M c (F)=σ(g 1×1 (AP(F))+g 1×1 (MP(F))

[0085]

[0086] Among them, M c (F), M s (F) represents the output of the channel attention module CAM and the spatial attention module SAM, AP(·) represents average pooling, MP(·) represents maximum pooling, g 1×1 and g 1×7Represent the one-dimensional convolution with kernel size of 1×1 and 1×7 respectively, and σ(·) represents the Sigmoid function. Therefore, the multi-scale fusion feature F′ combined with the W-BAM module fusion The calculation method is:

[0087]

[0088] in, represents element-by-element multiplication, F′ is the fusion feature enhanced by channel attention, F″ is the fusion feature enhanced by spatial attention, α and β are the weights of channel attention and spatial attention respectively, F′ fusion It is the final multi-scale fusion feature.

[0089] S33. Perform multi-scale optimization on the multi-scale fusion features.

[0090] Using WBAM to represent the signal sample after the two stages of feature extraction and feature fusion, the output features are:

[0091] z i =WBAM(x i ),z j =WBAM(x j ),z k =WBAM(x k )

[0092] Use Euclidean distance to measure the distance between two samples:

[0093]

[0094] Among them, z a and z b is the feature vector of any two samples (dimension is D); z a,m represents the mth dimension of the ath sample feature, z b,m Represents the mth dimension of the bth sample feature.

[0095] The contrast loss is thus:

[0096]

[0097] Where y is the sample pair label: y = 1 for positive sample pairs and y = 0 for negative sample pairs; α is a hyperparameter, which represents the minimum margin of negative sample pairs, forcing the distance between negative sample pairs to be greater than α.

[0098] According to the obtained contrast loss, the parameters of the multi-scale feature extraction and fusion network are optimized to obtain the optimized multi-scale feature extraction and fusion network.

[0099] S34. Perform multimodal feature fusion based on the signal features and global visual features of the target domain support set to obtain signal-visual fusion features.

[0100] The normal signal of the target domain support set is input into the optimized multi-scale feature extraction and fusion network to obtain the signal features of the target domain support set.

[0101] The normal image of the target domain support set is input into the pre-trained visual encoder to obtain the global visual features of the target domain support set.

[0102] The signal features of the normal signal samples of the target domain support set obtained through multi-scale feature extraction and fusion are:

[0103] Z signal =WBAM(x support )

[0104] The global visual features of the normal image samples of the target domain support set extracted by the pre-trained visual encoder are:

[0105] Z visual =Encoder visual (x support )

[0106] The final multimodal feature fusion is:

[0107]

[0108] Where ⊕ represents fusion.

[0109] In this embodiment, an enhanced text learner is used as the text feature learning module. To address the problem in traditional PromptAD methods where learnable prompt vectors are susceptible to visual feature assimilation, leading to text semantic degradation, this embodiment proposes a hierarchical hybrid prompt learning mechanism that achieves multimodal collaborative optimization by explicitly decoupling semantic preservation and feature alignment.

[0110] The specific design of the text feature learning module is as follows: Figure 5 Shown, including:

[0111] S341. Structured prompt design includes four parts: manual normal prompt, learnable normal prompt, manual abnormal prompt, and learnable abnormal prompt.

[0112] Among them, Manual Normal Prompts (MN) builds a priori semantic preservation templates based on domain knowledge (source domain and / or target domain knowledge):

[0113] s mn =[obj.][without][flaw]

[0114] Where [obj.] is the detected object category label, and normal suffixes are generated from the normal labels of the dataset, such as []without flaw, []without defect, etc.

[0115] Learnable Normal Prompts (NP) introduces parameterized prefixes to enhance cross-modal alignment capabilities:

[0116]

[0117] Among them, P i is the trainable vector, E N Indicates the length of the normal prefix that can be learned, and [obj.] is the name of the detection object.

[0118] In Manual Anomaly Prompts (MAP), anomaly suffix texts are generated from the anomaly labels of the dataset, such as []with color stain, []with crack, etc., and then these texts are connected with NP to obtain Manual Anomaly Prompts (MAP):

[0119]

[0120] Where the prefix is ​​the trainable NP and the suffix is ​​the manual exception text.

[0121] Combine NP with learnable anomaly suffix text to design learnable anomaly prompts (LAP):

[0122]

[0123] Among them E A Indicates the length of the learnable abnormal suffix text.

[0124] S342. Optimization of contrast prompts.

[0125] During training, MN and NP move towards the normal fusion features, while MAP and LAP move away from the normal fusion features. The training loss of the prompt learning is as follows:

[0126]

[0127] in They are the prototypes of all normal features and abnormal prompt features respectively.

[0128] w n ={g(s mn ),g(s np)}: A collection of text embeddings containing both manually generated normal hints (MN) and learnable normal hints (NP).

[0129] w a ={g(s map ),g(s lap )}: A collection of text embeddings containing Manual Anomaly Prompts (MAPs) and Learnable Anomaly Prompts (LAPs).

[0130] Z represents the signal-vision multimodal fusion feature. N mn and N np are the number of manual normal prompts and learnable normal prompts, respectively, g(·) represents the text encoder (such as the text encoder of CLIP), represents the expectation (mean) of the signal-visual multimodal fusion feature Z, and τ is the temperature coefficient.

[0131] The anomaly score is:

[0132]

[0133] where z t is the global image feature of the image to be tested.

[0134] S35. Train the text feature learning module (i.e., text learner) through signal-visual fusion features, then obtain text from the trained text feature learning module, input it into the pre-trained text encoder to obtain text features, and store the text features in the text feature library.

[0135] S4. Extract low-level, high-level and global visual features of the image to be tested, and compare the extracted low-level and high-level visual features with the visual feature library through the previously trained feature aligner by the nearest neighbor search method to obtain low-level feature scores and high-level feature scores, and then perform weighted average to obtain the visual score; then compare the extracted global visual features with the text feature library by the nearest neighbor search method to obtain the text score; fuse the visual score and text score based on the cross self-attention mechanism to realize anomaly detection of the image to be tested.

[0136] In this embodiment, the anomaly detection process is as follows:

[0137] Input the image to be tested into the pre-trained visual encoding network, extract the low-level, high-level and global visual features of the image to be tested, and use the nearest neighbor search method to compare the extracted low-level and high-level visual features with the visual feature library through the previously trained feature aligner to obtain the low-level feature score and the high-level feature score, and then perform weighted average to obtain the visual score; then use the nearest neighbor search method to compare the extracted global visual features with the text feature library to obtain the text score; then, based on the cross-self-attention mechanism, fuse the visual score and the text score to obtain the total anomaly score to achieve anomaly detection. The overall process is as follows: Figure 6 shown.

[0138] In this embodiment, a multimodal fusion of visual scores and text scores is performed based on a cross-self-attention mechanism to achieve complementary enhancement of visual and textual modalities. Specifically, the following steps are included:

[0139] S41. Fractional space mapping.

[0140] Project the scalar anomaly score into semantic space:

[0141] s v =W v s v +b v ,s t =W t s t +b t

[0142] Represents the visual score, which is obtained by searching the nearest neighbor between the visual feature and the visual feature library; represents the text score, which is obtained by searching the nearest neighbor of the global visual feature and the text feature library; W v 、W t are all weight matrices; b v 、b t All are bias terms.

[0143] S42. Cross-attention interaction.

[0144] The interactive relationship between visual and text scores is captured dynamically through cross attention, and the formula is as follows:

[0145] Visual as query, text as key: Q v =W Q s v ,K t =W K s t ,V t =W V s t .

[0146] Text as query, visual as key: Q t =W Q s t ,K v =W K s v ,V v =W V s v .

[0147] W Q 、W K 、W V is the attention weight parameter.

[0148] Visual→Textual Attention:

[0149]

[0150] Visually enhanced text representation is h v =α v→t V t .

[0151] Text → Visual Attention:

[0152]

[0153] The visual representation of text enhancement is h t =α t→v V v .

[0154] S43, residual fusion.

[0155] First, the enhanced feature representation is obtained by performing residual connection and layer normalization:

[0156]

[0157] The enhanced feature representations are then concatenated and mapped into scalar scores through a fully connected layer:

[0158]

[0159] Where || represents the concatenation operation.

[0160] Based on the same inventive concept, this embodiment also provides a small sample industrial fault detection system based on signal-vision collaboration, including the following modules:

[0161] Feature alignment module, which is used to design a cross-domain alignment module and introduce a self-supervised visual feature aligner. By constructing sample pairs on the source domain to train the self-supervised visual feature aligner, it can achieve feature alignment in the target domain.

[0162] The visual feature processing module is used to design a two-stage domain adaptation framework, build a domain adaptor, and jointly optimize the domain adaptor through cross-domain distribution alignment loss and feature reconstruction loss to migrate source domain features to the target domain space, achieve feature space alignment, obtain target domain alignment features and source domain adaptation features, and store them in the visual feature library;

[0163] The fusion module is used to fuse the multimodal features of global visual features and signal features to simultaneously capture the spatial morphological features and time-frequency domain features of the anomaly and obtain signal-visual fusion features. The signal-visual fusion features are used to train the text feature learning module, obtain text from the trained text feature learning module, input it into the pre-trained text encoder to obtain text features, and store the text features in the text feature library.

[0164] The anomaly detection module extracts low-level, high-level, and global visual features of the image to be tested, and compares the extracted low-level and high-level visual features with the visual feature library through the previously trained feature aligner using the nearest neighbor search method to obtain low-level feature scores and high-level feature scores, which are then weighted averaged to obtain a visual score. The extracted global visual features are then compared with the text feature library using the nearest neighbor search method to obtain a text score. The visual score and text score are fused based on the cross-self-attention mechanism to achieve anomaly detection in the image to be tested.

[0165] The above modules are respectively used to implement the steps of the detection method of this embodiment. The detailed implementation process can be found in the above step descriptions.

[0166] In this embodiment, the training process mainly includes:

[0167] (1) First, two normal images are randomly selected from the source domain to train a self-supervised visual feature aligner.

[0168] (2) Then randomly select a normal image from the source domain, train the domain adaptor together with the support set normal image of the target domain, and store the trained source domain adaptive features and target domain alignment features in the visual feature library.

[0169] (3) Then, two signals are randomly selected from the normal signal samples in the training set to train the multi-scale feature extraction and fusion network (multi-scale twin network).

[0170] (4) Finally, a multimodal fusion feature is obtained by fusing the signal features of the target domain support set with the visual features, which is used to train the text learner, and the trained text features are stored in the text feature library.

[0171] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A small sample industrial fault detection method based on signal-vision collaboration, characterized in that: The following steps are involved: S1. Design a cross-domain alignment module and introduce a self-supervised visual feature aligner. By constructing sample pairs on the source domain to train the self-supervised visual feature aligner, we can achieve feature alignment in the target domain. S2. Design a two-stage domain adaptation framework, build a domain adaptor, and jointly optimize the domain adaptor by cross-domain distribution alignment loss and feature reconstruction loss to migrate source domain features to the target domain space, achieve feature space alignment, obtain target domain alignment features and source domain adaptation features, and store them in the visual feature library; S3. Through multimodal feature fusion of global visual features and signal features, the spatial morphological features and time-frequency domain features of the anomaly are simultaneously captured to obtain signal-visual fusion features. The text feature learning module is trained using the signal-visual fusion features, and text is obtained from the trained text feature learning module. The text is then input into the pre-trained text encoder to obtain text features, and the text features are stored in the text feature library. S4. Extract low-level, high-level, and global visual features of the image to be tested. Use the nearest neighbor search method to compare the extracted low-level and high-level visual features with the visual feature library through the previously trained feature aligner to obtain low-level feature scores and high-level feature scores, and then perform weighted average to obtain the visual score; then use the nearest neighbor search method to compare the extracted global visual features with the text feature library to obtain the text score; based on the cross-self-attention mechanism, fuse the visual score and text score to realize anomaly detection of the image to be tested.

2. The small sample industrial fault detection method according to claim 1, characterized in that: The training process of the self-supervised visual feature aligner in step S1 includes: S11, cross-level feature extraction, given a pair of normal images of the same category from the source domain, multi-level features are extracted through the pre-trained visual encoder; S12. Design a self-supervised visual feature aligner to perform adaptive feature projection on multi-level features; S13, hierarchical comparison constraints, constructing bidirectional similarity maximization loss.

3. The small sample industrial fault detection method according to claim 1, characterized in that: Step S2 selects a normal image from the support set of the source domain and the target domain respectively, first extracts their image features through the pre-trained visual encoder, then obtains the source domain alignment feature and the target domain alignment feature through the self-supervised visual feature aligner trained in step S1, and stores the target domain alignment feature in the visual feature library; the source domain alignment feature is input into the two-stage domain adaptor to obtain the source domain adaptive feature; The adaptive loss of the source domain adaptive features and the target domain aligned features is calculated through distribution moment matching, as well as the reconstruction loss of the residual features of the source domain adaptive features and the source domain aligned features. This optimizes the two-stage domain adaptor and realizes the feature space alignment from the source domain to the target domain. The finally optimized source domain adaptive features are also stored in the visual feature library.

4. The small sample industrial fault detection method according to claim 3, characterized in that: Step S2 includes: S21. Obtain source domain alignment features and target domain alignment features through the pre-trained visual encoder and the trained self-supervised visual feature aligner; S22, processing the source domain alignment features through a domain adaptor to obtain source domain adaptive features; S23. Construct a multi-objective optimization function based on adaptive loss and feature reconstruction loss.

5. The small sample industrial fault detection method according to claim 1, characterized in that: Step S3 includes: S31. Extract sample features of positive and negative sample pairs through multi-scale feature extraction and fusion network; S32, fusing the extracted sample features through different modules through a multi-scale feature extraction and fusion network to obtain multi-scale fusion features; S33, perform multi-scale optimization on the multi-scale fusion features, and optimize the multi-scale feature extraction and fusion network; S34. Inputting the normal signal of the target domain support set into the optimized multi-scale feature extraction and fusion network to obtain the signal features of the target domain support set, inputting the normal image of the target domain support set into the pre-trained visual encoder to obtain the global visual features of the target domain support set; performing multimodal feature fusion based on the signal features of the target domain support set and the global visual features to obtain signal-visual fusion features; S35. Train the text feature learning module through signal-visual fusion features, obtain text from the trained text feature learning module, input it into the pre-trained text encoder to obtain text features, and store the text features in the text feature library.

6. The small sample industrial fault detection method according to claim 5, characterized in that: The design process of the text feature learning module in step S34 includes: S341. Structured prompt design, including manual normal prompts, learnable normal prompts, manual abnormal prompts, and learnable abnormal prompts. Manual normal prompts build a priori semantic preservation templates based on domain knowledge, and learnable normal prompts introduce parameterized prefixes to enhance cross-modal alignment capabilities. Manual abnormal prompts generate abnormal suffix text from abnormal labels in the dataset, and then concatenate the abnormal suffix text with the learnable normal prompt to obtain manual abnormal prompts. Learnable abnormal prompts are designed by combining learnable normal prompts with learnable abnormal suffix text. S342. Contrast prompt optimization. During the training process, manual normal prompts and learnable normal prompts move toward the normal fusion feature, while manual abnormal prompts and learnable abnormal prompts move away from the normal fusion feature.

7. The small sample industrial fault detection method according to claim 1, characterized in that: In step S4, the multimodal fusion of visual scores and text scores is performed based on the cross self-attention mechanism to achieve complementary enhancement of visual and textual modalities.

8. The small sample industrial fault detection method according to claim 7, characterized in that: The multimodal fusion of visual scores and text scores based on the cross self-attention mechanism in step S4 includes: S41, score space mapping, projects the scalar anomaly score into the semantic space; S42, dynamically capturing the interaction between visual and text scores through cross-attention; S43, residual fusion, first performs enhancement processing through residual connection and layer normalization to obtain enhanced feature representation; then the enhanced feature representation is spliced ​​and mapped into scalar scores through a fully connected layer.

9. The small sample industrial fault detection method according to claim 4, characterized in that: The multi-objective optimization function constructed in S23 is: L adapt =λ1L mmd +λ2L recon Where λ1 and λ2 are weight coefficients, L mmd is the maximum mean difference loss, L recon is the feature reconstruction loss.

10. A small sample industrial fault detection system based on signal-vision collaboration, characterized in that: Based on the method according to any one of claims 1 to 9, the detection system includes the following modules: Feature alignment module, which is used to design a cross-domain alignment module and introduce a self-supervised visual feature aligner. By constructing sample pairs on the source domain to train the self-supervised visual feature aligner, it can achieve feature alignment in the target domain. The visual feature processing module is used to design a two-stage domain adaptation framework, build a domain adaptor, and jointly optimize the domain adaptor through cross-domain distribution alignment loss and feature reconstruction loss to migrate source domain features to the target domain space, achieve feature space alignment, obtain target domain alignment features and source domain adaptation features, and store them in the visual feature library; The fusion module is used to fuse the multimodal features of visual features and signal features to simultaneously capture the spatial morphological features and time-frequency domain features of the anomaly and obtain signal-visual fusion features. The signal-visual fusion features are used to train the text feature learning module, obtain text from the trained text feature learning module, input it into the pre-trained text encoder to obtain text features, and store the text features in the text feature library. The anomaly detection module extracts low-level, high-level, and global visual features of the image to be tested, and compares the extracted low-level and high-level visual features with the visual feature library through the previously trained feature aligner using the nearest neighbor search method to obtain low-level feature scores and high-level feature scores, which are then weighted averaged to obtain a visual score. The extracted global visual features are then compared with the text feature library using the nearest neighbor search method to obtain a text score. The visual score and text score are fused based on the cross-self-attention mechanism to achieve anomaly detection in the image to be tested.

Images