Multi-modal contrast learning fault diagnosis method for small sample scene

Abstract

The invention discloses a multi-modal contrast learning fault diagnosis method for a small sample scene, and the method comprises the following steps: carrying out the enhancement of a one-dimensional signal and two-dimensional image fused fault data set through physical simulation for the small sample scene with scarce industrial fault data, and constructing a positive and negative sample pair through a plurality of data enhancement strategies; based on heterogeneous multi-modal fault data, designing a double-flow encoder architecture of a time sequence branch and an image branch, extracting depth features and mapping the depth features to a unified feature space through a projection head; performing supervised contrast learning pre-training based on intra-modal and inter-modal dual contrast loss; supervision fine tuning is carried out based on multiple loss functions such as physical guidance, so that accurate diagnosis of equipment faults is realized in a small sample scene. According to the fault diagnosis method under the unbalanced sample and limited labeling conditions, the problem that a traditional data driving model depends on large-scale labeling samples is effectively relieved through supervised comparative learning and cross-modal information alignment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing and equipment fault diagnosis technology, specifically involving a multimodal comparative learning fault diagnosis method for small sample scenarios. Background Technology

[0002] Driven by Industry 4.0 and intelligent manufacturing, industrial equipment fault diagnosis has become a core component in ensuring production quality, optimizing operational efficiency, and preventing catastrophic failures. Equipment fault diagnosis utilizes various inspection and testing methods to identify faults in systems and equipment, further pinpointing the approximate location of the fault and locating it at the level of replaceable products for repair. Among these methods, data-driven approaches based on deep learning, with their superior adaptive feature extraction capabilities, effectively reduce the costs of expert manual feature extraction and physical modeling, gradually becoming the mainstream paradigm. They have achieved end-to-end fault classification across various data types, including vibration signals, temperature curves, and infrared images. Superplastic molding equipment, as advanced forming equipment for manufacturing thin-walled complex components, has extremely high requirements for operational stability and accurate fault identification. Long-term operation at high temperatures makes it prone to various faults such as overheating, short circuits, open circuits, and leakage, severely impacting forming accuracy and equipment safety. Furthermore, the low frequency of fault occurrence and high annotation costs in actual operation make it difficult for traditional deep learning methods to achieve ideal fault identification performance in small-sample scenarios. Summary of the Invention

[0003] To address the aforementioned technical challenges, this invention proposes a multimodal contrastive learning-based fault diagnosis method for scenarios with limited sample sizes. This method enables intelligent fault diagnosis even with a limited number of labeled samples. It utilizes multimodal data generated from physical simulation for input and enhancement, constructing positive and negative sample pairs through various data augmentation strategies. Based on heterogeneous multimodal data, a dual-stream encoder architecture with temporal and image branches is designed. Supervised contrastive learning pre-training is then performed using intra-modal and inter-modal contrastive losses. A proxy task incorporating label information drives the dual-stream encoder to learn discriminative and universal feature representations. During fine-tuning, a physical-guided multiple loss function is introduced to concatenate the feature vectors output by the dual-stream encoder. Finally, a classification head composed of multilayer perceptrons outputs the probability distributions of various fault types, thereby achieving accurate identification of different fault types.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A multimodal contrastive learning-based fault diagnosis method for scenarios with small sample sizes includes the following steps:

[0006] Step S110: For small sample scenarios where industrial fault data is scarce, physical simulation is used to enhance heterogeneous multimodal data fused from one-dimensional signals and two-dimensional images. At the same time, positive and negative sample pairs are constructed through various data augmentation strategies.

[0007] Step S120: Based on heterogeneous multimodal data, design a dual-stream encoder architecture with temporal branch and image branch;

[0008] Step S130: Based on the dual contrast loss of intra-modal and inter-modal modes, supervised contrastive learning pre-training is performed on the two-stream encoder;

[0009] Step S140: Supervised fine-tuning and fault diagnosis based on physical-guided multiple loss functions: Remove the projection head used in pre-training, concatenate the feature vectors output by the dual-stream encoder, introduce multiple loss functions for joint optimization, and finally input a classification head composed of multiple perceptrons to obtain the probability distribution of each type of fault.

[0010] Beneficial effects:

[0011] In the field of industrial fault diagnosis, existing solutions to the few-shot problem mainly focus on improving the model's recognition ability by utilizing data augmentation, transfer learning, or few-shot learning algorithms. However, these methods often optimize only within a single modality or local feature space, making it difficult to fully explore the potential complementary information between different modalities, resulting in limited feature representation capabilities. Furthermore, traditional methods generally rely on large-scale labeled data to support model training, lacking the ability to effectively handle cross-modal semantic differences, and still have shortcomings in generalization performance and robustness.

[0012] The fault diagnosis method based on multimodal contrastive learning proposed in this invention can fully utilize complementary feature information from multi-source heterogeneous data such as time series and images. By constructing a dual contrast mechanism within and between modalities, it achieves feature alignment and fusion between different modalities at the semantic level. This strategy not only improves the separability and discriminability of feature distribution under small sample conditions, but also effectively reduces the model's dependence on large-scale labeled data, enhancing its adaptability in complex environments. By introducing an optimization mechanism that combines physical constraints and supervision signals, this invention achieves higher diagnostic accuracy and stronger cross-modal generalization performance while maintaining model interpretability. Therefore, this invention is of great significance for overcoming the challenges of multimodal feature learning under small sample conditions and improving the reliability and practicality of industrial intelligent diagnostic systems. Attached Figure Description

[0013] Figure 1 This is a flowchart of a multimodal contrastive learning fault diagnosis method for small sample scenarios according to the present invention.

[0014] Figure 2This is a flowchart illustrating the overall process of pre-training and supervised fine-tuning of a multimodal contrastive learning diagnostic algorithm. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0016] Figure 1 This is a flowchart of a multimodal contrastive learning fault diagnosis method for small sample scenarios according to the present invention. Figure 2 This is a flowchart illustrating the overall process of pre-training and supervised fine-tuning of a multimodal contrastive learning diagnostic algorithm. Figure 1 and Figure 2 As shown, the method includes:

[0017] Step S110: Addressing the scarcity of industrial fault data in small-sample scenarios, this step enhances heterogeneous multimodal data fused from one-dimensional signals and two-dimensional images through physical simulation. Specifically, multimodal refers to data containing two different dimensions of physical quantities. Simultaneously, positive and negative sample pairs are constructed using various data augmentation strategies. Step S110 may include: supplementing fault data using physical simulation, and then applying multiple data augmentation methods to both time-series data and image data to generate semantically consistent but visually diverse samples.

[0018] Step S120: Based on heterogeneous multimodal data, design a dual-stream encoder architecture with temporal and image branches. This step may include: constructing a hybrid temporal encoder that integrates convolutional neural networks and Transformers, while the image branch adopts a feature extraction strategy based on transfer learning, using ResNet-34 as the backbone network;

[0019] Step S130: Based on intra-modal and inter-modal dual contrastive loss, supervised contrastive learning pre-training is performed on the two-stream encoder. This step may include: constructing a proxy task based on label information to drive the two-stream encoder to learn discriminative and universal feature representations; and using a dual contrastive learning mechanism to bring similar samples closer together and dissimilar samples further apart in the feature space, thereby achieving single-modal feature extraction and cross-modal semantic alignment.

[0020] Step S140: Supervised fine-tuning based on multiple loss functions such as physical guidance: Remove the projection head used in pre-training, concatenate the feature vectors output by the dual-stream encoder, introduce multiple loss functions for joint optimization, and finally input a classification head composed of multiple perceptrons to obtain the probability distribution of each type of fault.

[0021] Furthermore, step S110 specifically includes:

[0022] Step S110-1: Using the finite element physical simulation strategy, by changing simulation variables such as material parameters, boundary conditions and load states, a multi-dimensional and multi-distributed virtual sample set is generated to supplement the fault dataset.

[0023] Step S110-2: For one-dimensional temperature time series data, enhancement techniques such as dithering, scaling, and random masking are employed. Dithering simulates sensor noise by adding Gaussian noise to the original signal; scaling simulates the overall change in signal amplitude by multiplying each time point of the signal by a random factor sampled by a Gaussian distribution.

[0024] ,in (1)

[0025] ,in (2)

[0026] (3)

[0027] in, It is the signal value of the original one-dimensional temperature time series at time t. It is an additive random noise term that follows a Gaussian distribution. It is a one-dimensional time series signal that has undergone enhancement processing. It is the symbol for the normal distribution. It is the standard deviation of the distribution. It is a multiplicative random scaling factor that follows a specific Gaussian distribution. It is the length of the time window for the random masking operation. It is the random starting index position of the masking window. It is the index of data points in a time series. It is the value of the j-th data point in the original sequence. It is the value of the j-th data point in the sequence after masking.

[0028] Step S110-3, for the processing of the two-dimensional temperature map, combines geometric transformation and photometric transformation. At the geometric level, random affine transformations are applied, including operations such as rotation, translation, scaling, random cropping and resizing, and horizontal flipping; at the photometric level, color dithering, random grayscale conversion, and random erasing techniques are used to simulate changes in imaging conditions and local occlusion phenomena, prompting a focus on more essential image features rather than relying on color or local details.

[0029] Furthermore, step S120 specifically includes:

[0030] Step S120-1: For one-dimensional temperature time series signals, a hybrid encoder fusing convolutional neural networks and Transformers is constructed to simultaneously capture local fluctuation patterns and global long-term dependencies. The convolutional neural network branch consists of stacked one-dimensional residual convolutional modules, each containing two convolutional layers and a short-circuit connection. The other branch uses standard Transformer encoder layers to model long-range dependencies between signals, effectively capturing macroscopic fault correlations and trend changes over the time span through a self-attention mechanism.

[0031] Step S120-2: The local fluctuation features extracted by the convolutional neural network branch and the global dependency features extracted by the Transformer branch are concatenated along the feature dimension to form a combined feature vector. This combined feature vector is then input into a fusion module composed of fully connected layers for nonlinear fusion processing, ultimately generating a unified temporal feature vector.

[0032] Further, step S130 specifically includes:

[0033] Step S130-1: Within the time series mode, the signals from the same original signal are... Construct positive sample pairs, and treat the features of all other time series samples within the batch as negative samples; within the image modality, features from the same original image are considered as negative samples. Positive sample pairs are constructed, and the features of all other image samples in the batch are treated as negative samples. Intramodal contrast loss is calculated, and the similarity of positive sample pairs is maximized and the similarity of negative samples is minimized by optimization to learn robust features that are invariant to disturbances such as noise and occlusion.

[0034] Among them, z' t , z'' t These represent the first and second types of enhanced view features in the time series modality, respectively, z' i ,z'' i These represent the first and second types of enhanced view features in the image modality. Specifically, for the temporal modality, the first enhancement refers to random time window cropping and amplitude scaling, while the second enhancement refers to Gaussian white noise injection; for two-dimensional images, the first enhancement refers to random rotation or horizontal flipping, while the second enhancement refers to Gaussian blurring or color dithering.

[0035] Step S130-2 involves combining heterogeneous features originating from the same fault category, such as... and Construct cross-modal positive sample pairs; treat all other sample features in the batch that do not belong to this event as negative samples; calculate cross-modal contrast loss, and optimize to make the representations of cross-modal positive sample pairs in the shared feature space closer to each other, thereby achieving semantic alignment.

[0036] Step S130-3: Average the two intramodal losses from step S130-1 and the two symmetrical crossmodal losses from step S130-2 to form an overall contrastive loss function; train the two-stream encoder by minimizing the overall contrastive loss function until the model converges.

[0037] Both intra-modal contrastive loss and cross-modal contrastive loss can use the InfoNCE loss function, as shown below:

[0038] (4)

[0039] in, Represents a sample anchor point. Indicates a positive sample. Indicates a negative sample. It is the temperature coefficient. This represents the mathematical expectation operation with respect to the sample anchor point, the corresponding positive sample, and the set of negative samples.

[0040] The total loss function L during the entire pre-training phase pretrain Defined as intramodal contrast loss L intra Intermodal contrast loss L inter Weighted sum:

[0041] (5)

[0042] in, These are the weighting coefficients of the intra-modal contrast loss term. These are the weighting coefficients of the intermodal contrast loss term.

[0043] Furthermore, step S140 specifically includes:

[0044] Step S140-1, construct the supervised fine-tuning framework: based on the pre-trained dual-stream encoder obtained in step S130, remove the projection head used in its pre-training stage; add a main classification head at the output of the dual-stream encoder to output the probability of the fault category.

[0045] Step S140-2: Design the Focal Loss function as the basic loss function for the classification task; construct the triplet loss based on the feature embedding of the encoder output. The triplet loss and the focal loss are combined to form the overall objective function through weighted summation, and the model parameters are optimized collaboratively in backpropagation; Since the actual industrial fault data has a long tail distribution, that is, there are far more normal working condition samples than fault samples, and there are also huge differences in the number of samples of different fault categories. The standard cross-entropy loss function treats all samples equally, which will cause the model training to be mainly affected by the majority category, and the ability to distinguish rare fault categories is insufficient. To overcome this problem, the Focal Loss function is used as the basic loss function for the classification task. As shown in Equation (6), it introduces a modulation factor (1-p) t ) γ The contribution of easily classified samples to the total loss is dynamically reduced. Secondly, based on the feature embedding of the encoder output, the triplet loss is constructed as shown in Equation (7) to increase the distance between features of different categories and enhance the distinguishability of features.

[0046] (6)

[0047] (7)

[0048] in, Represents probability. It is the class balance coefficient, used to balance positive and negative samples. This indicates the focusing parameter; for triplet equations, This is the margin hyperparameter, used to ensure the minimum distance between positive and negative samples. For anchor point samples, These represent positive and negative samples, respectively.

[0049] Step S140-3 introduces physical guided regularization into the supervised fine-tuning framework to constrain the framework to learn cross-modal relationships with physical consistency; an additional lightweight auxiliary classifier is added to the output of the temporal branch of the two-stream encoder. Since a purely data-driven model may produce diagnoses that violate physical laws in regions with insufficient training data, leading to decreased generalization ability, a physical guided regularization term L is introduced into the model. PG The mean square error between the predicted features and the true features is calculated as shown in Equation (8), which serves as a physical regularization term to constrain the model to learn cross-modal relationships with physical consistency. In addition, since the model will over-rely on the easier-to-learn modality, i.e. the image modality, a lightweight auxiliary classifier is added to the output of the temporal branch (time encoder) of the dual-stream encoder as shown in Equation (9) to ensure that the branch can independently learn discriminative information and prevent representation collapse.

[0050] (8)

[0051] (9)

[0052] in, These are the feature vectors of the source domain and the target domain, respectively. To predict the probability distribution, aux is used as an auxiliary label. It is a feature mapping function from temporal sequence modes to image modes. It is a feature mapping function from image mode to temporal sequence mode.

[0053] Step S140-4: The focus loss, triplet loss, physical guided regularization, and lightweight auxiliary classifier calculated in steps S140-2 and S140-3 are applied using preset weight coefficients λ. i We perform a weighted summation to obtain the overall fine-tuning loss L. finetune as follows:

[0054] (10)

[0055] By minimizing the Gradient updates are performed on the dual-stream encoder and all related classification heads and projection heads until the supervised fine-tuning framework converges.

[0056] In summary, this invention provides a multimodal contrastive learning-based fault diagnosis method for scenarios with limited sample sizes. This method supplements multimodal data through physical simulation and data augmentation, constructs a temporal and image dual-stream encoder structure, and performs supervised pre-training based on dual contrastive losses within and between modes. Furthermore, multiple loss functions are introduced to jointly optimize the model, thereby effectively improving the discriminative and generalizable properties of the features. This invention can achieve accurate fault identification under conditions of limited sample sizes and complex operating conditions, which is of great significance for improving the reliability and diagnostic efficiency of intelligent manufacturing systems.

[0057] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0058] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered to fall within the protection scope of the present invention.

Claims

1. A multimodal contrastive learning-based fault diagnosis method for small sample scenarios, characterized in that, Includes the following steps: Step S110: For small sample scenarios where industrial fault data is scarce, physical simulation is used to enhance heterogeneous multimodal data fused from one-dimensional signals and two-dimensional images. At the same time, positive and negative sample pairs are constructed through various data augmentation strategies. Step S120: Based on heterogeneous multimodal data, design a dual-stream encoder architecture with temporal branch and image branch; Step S130: Based on the dual contrast loss of intra-modal and inter-modal modes, supervised contrastive learning pre-training is performed on the two-stream encoder; Step S140: Supervised fine-tuning and fault diagnosis based on physical-guided multiple loss functions: Remove the projection head used in pre-training, concatenate the feature vectors output by the dual-stream encoder, introduce multiple loss functions for joint optimization, and finally input a classification head composed of multiple perceptrons to obtain the probability distribution of each type of fault.

2. The multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 1, characterized in that, Step S110 specifically includes: Step S110-1: Using the finite element physical simulation strategy, by changing material parameters, boundary conditions and load states, a multi-dimensional and multi-distributed virtual sample set is generated to supplement the fault dataset. Step S110-2: For one-dimensional temperature time series data, dithering, scaling and random masking enhancement techniques are used: dithering simulates sensor noise by adding Gaussian noise to the original signal; scaling simulates the overall change in signal amplitude by multiplying each time point of the signal by a random factor of Gaussian distribution sampling. Step S110-3, for the processing of the two-dimensional temperature map, combines geometric transformation and photometric transformation: at the geometric level, random affine transformation is applied, including rotation, translation, scaling, random cropping and size adjustment, and horizontal flipping operations; at the photometric level, color dithering, random grayscale conversion and random erasure techniques are used to simulate changes in imaging conditions and local occlusion phenomena, prompting the focus to be on more essential image features rather than relying on color or local details.

3. The multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 2, characterized in that, Step S120 includes: constructing a hybrid temporal encoder that integrates convolutional neural networks and Transformers, while the image branch adopts a feature extraction strategy based on transfer learning, and ResNet-34 is selected as the backbone network.

4. The multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 3, characterized in that, Step S120 specifically includes: Step S120-1: For one-dimensional temperature time series signals, a hybrid encoder integrating convolutional neural networks and Transformers is constructed to simultaneously capture local fluctuation patterns and global long-term dependencies. The convolutional neural network branch consists of stacked one-dimensional residual convolutional modules, each containing two convolutional layers and a short-circuit connection. The other branch uses standard Transformer encoder layers to model long-range dependencies between signals, effectively capturing macroscopic fault correlations and trend changes over the time span through a self-attention mechanism. Step S120-2: The local fluctuation features extracted by the convolutional neural network branch and the global dependency features extracted by the Transformer branch are concatenated in the feature dimension to form a combined feature vector. The combined feature vector is then input into a fusion module composed of fully connected layers for nonlinear fusion processing, ultimately generating a unified time feature vector.

5. The multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 1, characterized in that, The specific steps of S130 include: constructing a proxy task based on label information, driving a dual-stream encoder to learn a feature representation with discriminative and universal characteristics, and using a dual contrastive learning mechanism to make similar samples closer together and dissimilar samples farther apart in the feature space, thereby achieving single-modal feature extraction and cross-modal semantic alignment.

6. The multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 5, characterized in that, Step S130 specifically includes: Step S130-1: Within the time series mode, the signals from the same original signal are... Construct positive sample pairs, and treat the features of all other time series samples within the batch as negative samples; within the image modality, features from the same original image are considered as negative samples. Positive sample pairs are constructed, and the features of all other image samples in the batch are treated as negative samples. The intramodal contrast loss is calculated, and the similarity of positive sample pairs is maximized and the similarity of negative samples is minimized by optimization in order to learn robust features that are invariant to disturbances. in, These represent the first and second types of enhanced view features in the time series modality, respectively. The first and second types of enhanced view features are represented in the image modality, respectively. For the time series modality, the first enhancement refers to random time window cropping and amplitude scaling, and the second enhancement refers to Gaussian white noise injection. For the image modality, the first enhancement refers to random rotation or horizontal flipping, and the second enhancement refers to Gaussian blurring or color dithering. Step S130-2: Construct cross-modal positive sample pairs from heterogeneous features originating from the same fault category; treat all other sample features in the batch that do not belong to this event as negative samples; calculate cross-modal contrast loss, and optimize to make the representations of cross-modal positive sample pairs in the shared feature space closer to each other, thereby achieving semantic alignment. Step S130-3: Average the two intramodal losses from step S130-1 and the two symmetrical crossmodal losses from step S130-2 to form an overall contrastive loss function; train the two-stream encoder by minimizing the overall contrastive loss function until the model converges.

7. The multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 6, characterized in that, Both intra-modal contrast loss and cross-modal contrast loss use the InfoNCE loss function, as shown below: （4） in, Represents a sample anchor point. Indicates a positive sample. Indicates a negative sample. It is the temperature coefficient. This represents the mathematical expectation operation with respect to the sample anchor point, the corresponding positive sample, and the set of negative samples; The total loss function L during the entire pre-training phase pretrain Defined as intramodal contrast loss L intra Intermodal contrast loss L inter Weighted sum: （5） in, These are the weighting coefficients of the intra-modal contrast loss term. These are the weighting coefficients of the intermodal contrast loss term.

8. The multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 7, characterized in that, Step S140 specifically includes: Step S140-1: Construct a supervised fine-tuning framework. Based on the pre-trained dual-stream encoder obtained in step S130, remove the projection head used in the pre-training stage. Add a main classification head to the output of the dual-stream encoder to output the probability of the fault category. Step S140-2: Design the focus loss function as the basic loss function for the classification task; based on the feature embedding of the encoder output, construct the triplet loss to enhance the aggregation of similar samples and widen the gap between samples of different categories in the feature space. Step S140-3: Introduce physical guided regularization into the supervised fine-tuning framework to constrain the framework to learn cross-modal relationships with physical consistency; add an additional lightweight auxiliary classifier to the output of the temporal branch of the two-stream encoder. Step S140-4: The focus loss, triplet loss, physical guided regularization, and lightweight auxiliary classifier calculated in steps S140-2 and S140-3 are applied using preset weight coefficients λ. i We perform a weighted summation to obtain the overall fine-tuning loss L. finetune ; By minimizing L finetune Gradient updates are performed on the dual-stream encoder and all related classification heads and projection heads until the supervised fine-tuning framework converges.

9. A multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 8, characterized in that, The focus loss function is shown in equation (6); the constructed triplet loss is shown in equation (7): （6） （7） in, Represents probability. It is the class balance coefficient, used to balance positive and negative samples. This indicates the focusing parameter; for triplet equations, This is the margin hyperparameter, used to ensure the minimum distance between positive and negative samples. For anchor point samples, These represent positive and negative samples, respectively.

10. A multimodal contrastive learning fault diagnosis method for small sample scenarios according to claim 9, characterized in that, Step S140-3 includes: Physical guidance regularization L PG The mean squared error between the predicted features and the true features is calculated as shown in Equation (8), and used as a physical regularization term to constrain the model to learn cross-modal relationships with physical consistency; a lightweight auxiliary classifier is shown in Equation (9) to ensure that this branch can independently learn discriminative information and prevent representation collapse. （8） （9） in, These are the feature vectors of the source domain and the target domain, respectively. To predict the probability distribution, aux is used as an auxiliary label. It is a feature mapping function from temporal sequence modes to image modes. It is a feature mapping function from image modes to time series modes.

Images