Cross-domain mechanical fault diagnosis method and device based on unbalanced multi-modal heterogeneous data

Abstract

The invention discloses a cross-domain mechanical fault diagnosis method and device based on unbalanced multi-modal heterogeneous data, and the method comprises the steps: obtaining a multi-modal heterogeneous signal of to-be-diagnosed mechanical equipment, the multi-modal heterogeneous signal comprising a one-dimensional vibration signal and a two-dimensional infrared thermal imaging image signal; inputting the multi-modal heterogeneous signal into a pre-trained cross-domain mechanical fault diagnosis model, and outputting a diagnosis result; the cross-domain mechanical fault diagnosis model is obtained by training a double-branch heterogeneous feature fusion network by adopting training data, and the double-branch heterogeneous feature fusion network comprises a double-branch heterogeneous feature extractor, a feature splicing layer, a feature classifier and a domain discriminator based on an adversarial network, the double-branch heterogeneous feature extractor comprises a vibration signal feature extractor and an image signal feature extractor which are arranged in parallel; a total loss function in the training process comprises focus loss of enhanced inter-class imbalance and intra-class difficult sample imbalance, and joint depth domain adaptive loss and adversarial network domain confusion loss. The invention aims to solve the problem of low utilization rate of multi-source heterogeneous data through a multi-modal feature fusion network.

Description

Technical Field

[0001] The present invention belongs to the technical field of mechanical equipment fault diagnosis, and specifically relates to a cross-domain mechanical fault diagnosis method and device based on unbalanced multimodal heterogeneous data. Background Art

[0002] Equipment health management (PHM), as a core technology for intelligent operation and maintenance in modern industry, holds irreplaceable strategic value in the full lifecycle management of critical equipment, such as aircraft engines and gas turbines. Data-driven intelligent diagnostic methods, with their powerful feature extraction capabilities, have become the most promising technology in the field of equipment health management. These methods not only automatically extract fault features through end-to-end learning, avoiding the subjectivity and limitations of traditional manual feature extraction, but also possess the ability to characterize complex, nonlinear fault modes. However, the single-sensor monitoring solutions currently used in industrial sites suffer from significant limitations in the perception dimension. Firstly, key fault features are prone to information loss when sensors are restricted by installation locations or encounter environmental interference. Secondly, single-dimensional perception data cannot fully capture the evolution of equipment states under complex operating conditions. This perception blind spot severely constrains modern industry's demand for high-precision early warning of faults. The widespread adoption of multi-sensor networks in the era of industrial big data offers new opportunities to overcome this dilemma. Multi-source sensor networks enable the simultaneous acquisition of multimodal data, such as vibration, acoustic emission, and thermal imaging, significantly improving condition awareness through the complementary nature of these modalities. Compared with traditional single-channel sensing methods, multi-source sensing shows significant advantages in signal diversity, robustness and comprehensive fault characterization.

[0003] However, current fault diagnosis methods based on multi-source sensing still face many key challenges: First, multimodal data such as acoustics, vibration, and temperature collected by different sensors often have significant differences in sampling frequency, signal-to-noise ratio, and feature distribution. Traditional methods find it difficult to fully tap into their collaborative representation value, resulting in low utilization of multi-source heterogeneous data. Second, due to the sporadic nature of mechanical equipment failures, the time series signals collected by industrial monitoring systems often exhibit extreme class imbalance. This data characteristic makes traditional diagnostic models easily dominated by the majority class, making the global accuracy indicator mask the risk of missed detection of critical faults. In addition, the domain differences in the knowledge transfer process will lead to the superposition of class distribution offset and sample number imbalance between the source and target domains, resulting in cross-domain distortion of minority class representation.

[0004] Tong Zheming et al. have published a patent in China titled “A harmonic reducer fault diagnosis method based on multi-source heterogeneous data fusion” (patent publication number CN119557697A). This method collects raw data from multiple sensors such as vibration, torque, and temperature under different fault modes, and fuses the features of the multi-source heterogeneous data by converting the signals into two-dimensional time-frequency images, and finally outputs the fault diagnosis results. However, this method is only suitable for application to balanced data sets under constant working conditions. When faced with actual industrial data collection scenarios where the model application conditions change significantly and the distribution between sample classes is extremely unbalanced, the above method will inevitably suffer from the problem of decreased diagnostic performance, resulting in shortcomings in the application level of actual industrial environments. Therefore, how to construct an inherently robust balanced learning mechanism and effective cross-domain knowledge transfer means under a multi-source heterogeneous framework has become a key breakthrough point for improving the generalization capability of fault diagnosis for multimodal industrial data. Summary of the Invention

[0005] In response to the problems existing in the prior art, the present invention provides a cross-domain mechanical fault diagnosis method and device based on unbalanced multimodal heterogeneous data. Its purpose is to solve the problem of low utilization of multi-source heterogeneous data through a multimodal feature fusion network; to address the problem of category imbalance in industrial monitoring data by adopting a dynamic weighted imbalance learning mechanism; and to eliminate the category distribution offset in cross-domain migration by constructing a hybrid adversarial domain adaptation framework, thereby realizing cross-domain mechanical fault diagnosis of unbalanced multimodal heterogeneous data.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0007] According to a first aspect of the present invention, a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data is provided, comprising:

[0008] Acquire a multimodal heterogeneous signal of the mechanical equipment to be diagnosed, wherein the multimodal heterogeneous signal includes a one-dimensional vibration signal and a two-dimensional infrared thermal imaging image signal;

[0009] Inputting the multimodal heterogeneous signal into a pre-trained cross-domain mechanical fault diagnosis model and outputting a diagnosis result;

[0010] Among them, the cross-domain mechanical fault diagnosis model is obtained by training a dual-branch heterogeneous feature fusion network using training data. The dual-branch heterogeneous feature fusion network includes a dual-branch heterogeneous feature extractor, a feature splicing layer, and a parallel feature classifier and a domain discriminator based on an adversarial network. The dual-branch heterogeneous feature extractor includes a parallel vibration signal feature extractor and an image signal feature extractor; the training data includes one-dimensional vibration signals and two-dimensional infrared thermal imaging image signals from a certain source domain working condition and corresponding fault labels, as well as one-dimensional vibration signals and two-dimensional infrared thermal imaging image signals from another target domain working condition; the total loss function of the training process includes focal loss that strengthens inter-class imbalance and intra-class difficult and easy sample imbalance, joint deep domain adaptation loss and adversarial network domain confusion loss.

[0011] In a possible implementation of the first aspect, the vibration signal feature extractor includes several multi-layer convolution-pooling layers of different scales, the image signal feature extractor is a lightweight ResNet network with pruned channel numbers and network layers; the feature splicing layer is used to achieve heterogeneous feature fusion.

[0012] In a possible implementation of the first aspect, the vibration signal feature extractor mines deep features through multiple convolution-pooling layers with different scale convolution kernel sizes, specifically:

[0013]

[0014] Among them, ConvPooling i Represents the i-th convolution-pooling operation; is the layer-by-layer vibration feature obtained after convolution pooling with different convolution kernel sizes, where is the initial input x vbr ; B×K×N represent the batch size of input samples, the number of feature maps and the input feature length respectively.

[0015] The image signal feature extractor realizes adaptive extraction of image features through two convolution blocks containing residual structures, specifically:

[0016]

[0017] Among them, ResidualConv j Represents the jth residual convolution; is the layer-by-layer image feature obtained after residual convolution operations at different layers, where That is the initial input x irt ;

[0018] The feature concatenation layer combines the features of different modalities through the feature concatenation operation Concat to achieve feature fusion and flatten them into two dimensions, thereby obtaining the final output f of the dual-branch heterogeneous feature extractor. fused , specifically:

[0019]

[0020] Among them, B×L f Represents the final shape of the fused features.

[0021] In a possible implementation of the first aspect, the feature classifier and the domain discriminator based on the adversarial network are both multi-layer perceptron structures, wherein the feature classifier is used to obtain the final diagnostic output of the cross-domain mechanical fault diagnosis model based on the fusion features of the feature splicing layer, and the domain discriminator based on the adversarial network is used to confuse the source domain and target domain sources of the extracted features to extract the domain-invariant features of the signal.

[0022] In a possible implementation of the first aspect, the total loss function of the training process is specifically as follows:

[0023]

[0024] in, is the total loss function of the training process; Focus loss to strengthen the imbalance between classes and the imbalance between difficult and easy samples within a class; is the joint domain adaptation loss; To combat the network domain confusion loss; λ and μ are weight parameters that control the proportion of each loss;

[0025] The focus loss that strengthens the imbalance between classes and the imbalance between difficult and easy samples within a class is specifically:

[0026]

[0027] Where C is the total number of categories; y i is the true label encoded by one-hot; p is the result of the model's output vector after passing the Softmax function; pi represents the probability of predicting the i-th category; (1-p i ) γ is a modulation factor used to reflect the weight of classification errors of different categories; γ represents the focus parameter that controls the degree of weight distribution of difficult and easy samples. The larger the γ value, the lower the loss contribution of easy-to-classify samples. When γ = 0, the focus loss degenerates into cross entropy loss; F1 i and Gmean iare the F1-score and G-mean values of category i, which respectively reflect the balance between the precision and recall of the category and the overall recognition balance of the category.

[0028] In a possible implementation manner of the first aspect, the joint domain adaptation loss is specifically:

[0029]

[0030] Where K(·) is the Gaussian kernel function; n s With n t Indicates the number of samples from the source domain and the target domain; n L Indicates the total number of layers involved in the joint domain adaptation difference calculation; and Represent the input from the source domain and the target domain on the lth layer, z l ∈{f l ,y l} is the intermediate feature f l And the final output y l combination.

[0031] In a possible implementation manner of the first aspect, the adversarial network domain confusion loss is specifically:

[0032]

[0033] Among them, p d is the output vector of the domain discriminator.

[0034] According to a second aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data is implemented.

[0035] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data is implemented.

[0036] According to a fourth aspect of the present invention, a computer program product is provided. When the computer program product is executed by a processor, the method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data is implemented.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] The present invention provides a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data, which can effectively solve the problems of low utilization of multi-source heterogeneous data, imbalanced category distribution and decreased cross-working condition migration performance in current industrial scenarios. Specifically, first, by constructing a dual-branch heterogeneous feature fusion network, deep fusion of multimodal features is achieved, and the complementary characterization information of vibration and thermal imaging data is fully mined. Secondly, based on the proposed dynamic focus loss function, the imbalanced differences between classes and the difficult and easy sample differences within classes are jointly optimized to enhance the model's recognition ability for minority class faults and effectively suppress the risk of missed faults caused by the dominance of the majority class. In addition, deep domain confusion is achieved through the design of a hybrid adversarial domain adaptation framework, and the deep domain adaptive loss function is combined to force the feature extractor to learn working condition invariant features, effectively reducing the distribution differences between the source domain and the target domain, and ensuring the cross-domain consistency of minority class fault representation. Compared with traditional single-modal methods, the dual-branch heterogeneous fusion network proposed in this invention shows significant advantages in cross-working condition fault diagnosis, especially in industrial monitoring scenarios with extremely unbalanced sample distribution. It can effectively improve the collaborative utilization efficiency of multi-source heterogeneous data of vibration and thermal imaging, and realize high-precision cross-domain fault diagnosis, which has important engineering application value.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the specific embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the specific embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to the present invention;

[0042] Figure 2 This is the training graph of the dual-branch heterogeneous feature fusion network of the present invention;

[0043] Figure 3 Diagnosis results of different migration tasks of heterogeneous datasets of Northwestern Polytechnical University in the embodiment of the present invention;

[0044] Figure 4 This is the confusion matrix analysis of the heterogeneous data set of Northwestern Polytechnical University in an embodiment of the present invention;

[0045] Figure 5 These are the imbalance ratio diagnosis results of different samples in the heterogeneous dataset of Northwestern Polytechnical University in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] like Figure 1 As shown, an embodiment of the present invention provides a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data, which specifically includes the following steps:

[0048] Step 1: Acquire a multimodal heterogeneous signal of a mechanical device to be diagnosed, wherein the multimodal heterogeneous signal includes a one-dimensional vibration signal and a two-dimensional infrared thermal imaging image signal.

[0049] Specifically, a vibration signal sensor is used to collect one-dimensional vibration signals of the mechanical equipment to be diagnosed, and an infrared thermal imager is used to collect two-dimensional infrared thermal imaging image signals of the mechanical equipment to be diagnosed. The vibration signal sensor and the infrared thermal imager collect their respective signals simultaneously.

[0050] It should be understood that the collected one-dimensional vibration signal and two-dimensional infrared thermal imaging image signal are preprocessed, and the length of the one-dimensional vibration signal and the two-dimensional infrared thermal imaging image signal is made to meet the input requirements of the cross-domain mechanical fault diagnosis model through random sliding window sampling. Downsampling operation should be considered to solve the problem of incomplete period distribution of vibration signal caused by excessive sampling rate.

[0051] Step 2: Input the multimodal heterogeneous signal into a pre-trained cross-domain mechanical fault diagnosis model and output a diagnosis result. The cross-domain mechanical fault diagnosis model is obtained by training a dual-branch heterogeneous feature fusion network using training data.

[0052] Specifically, combined Figure 2 As shown, the dual-branch heterogeneous feature fusion network includes a dual-branch heterogeneous feature extractor, a feature splicing layer, and a parallel feature classifier and domain discriminator based on an adversarial network. The dual-branch heterogeneous feature extractor, through the multi-branch design of different modalities, can extract features from the input multimodal signals separately. The dual-branch heterogeneous feature extractor includes a vibration signal feature extractor and an image signal feature extractor, which are arranged in parallel. The vibration signal feature extractor includes several multi-layer convolution-pooling layers of different scales, and the image signal feature extractor is a lightweight ResNet network with pruned channels and network layers. The feature splicing layer is used to achieve heterogeneous feature fusion.

[0053] For the vibration signal feature extractor, deep feature mining can be achieved by stacking multiple convolution-pooling layers with different scale convolution kernel sizes. Specifically:

[0054]

[0055] Among them, ConvPooling i Represents the i-th convolution-pooling operation; is the layer-by-layer vibration feature obtained after convolution pooling with different convolution kernel sizes, where is the initial input x vbr ; B×K×N represent the batch size of input samples, the number of feature maps and the input feature length respectively.

[0056] For the image signal feature extractor, adaptive extraction of image features is achieved through two convolution blocks containing residual structures, specifically:

[0057]

[0058] Among them, ResidualConv j Represents the jth residual convolution, which is derived from the standard operation process of the residual block in the ResNet-18 model structure and is obtained by adding a residual connection structure to two convolutional layers. is the layer-by-layer image feature obtained after residual convolution operations at different layers, where That is the initial input x irt .

[0059] The final obtained different modal features are fused and flattened into two dimensions through the feature concatenation operation Concat, thereby obtaining the final output f of the dual-branch heterogeneous feature extractor. fused , specifically:

[0060]

[0061] Among them, B×L f Represents the final shape of the fused feature. In this embodiment, L f Defined as 128.

[0062] The feature classifier and the domain discriminator based on the adversarial network are both multi-layer perceptron structures, wherein the feature classifier is used to obtain the final diagnostic output of the cross-domain mechanical fault diagnosis model based on the fusion features of the feature splicing layer, and the domain discriminator based on the adversarial network is used to confuse the source domain and target domain sources of the extracted features to extract the domain-invariant features of the signal. Specifically, each linear connection layer of the feature classifier uses the ReLU function as the activation function for nonlinear mapping, and finally obtains the output of the model by representing and learning from the extracted fusion features. Each linear connection layer of the domain discriminator based on the adversarial network uses the ReLU function as the activation function for nonlinear mapping, and sets a gradient reversal layer in the gradient propagation process to confuse the source domain and target domain sources of the extracted features, thereby promoting the deep mining of domain-invariant features by each feature extractor.

[0063] The training data includes one-dimensional vibration signals and two-dimensional infrared thermal imaging signals from a source domain operating condition, along with corresponding fault labels, and one-dimensional vibration signals and two-dimensional infrared thermal imaging signals from a target domain operating condition. Specifically, a multimodal, heterogeneous dataset is acquired from vibration signal sensors and infrared thermal imagers, and the signals are collected and organized under different operating conditions. Labeled signal samples, due to their complete labeling and ease of access, can be considered source domain data. Unlabeled signal samples, due to their lack of labeling, often reflect the characteristics of signal acquisition under actual operating conditions and can be considered target domain data. To prepare training and test datasets that conform to the distribution characteristics of actual industrial data, data preprocessing is performed on the source and target domain signal samples. This preprocessing step involves obtaining a sample set that meets the input length of the model using random sliding window sampling, and considering downsampling to address the incomplete period distribution of the vibration signal caused by excessive sampling rates. In this embodiment, the model sampling length is set to 2048, and the downsampling rate is 4. Furthermore, the datasets with different class imbalances are partitioned and prepared for use in the model training process. The model training process includes: inputting source domain data and target domain data into a dual-branch heterogeneous feature fusion network, using labeled sample sets from the source domain to train the feature extractors and feature classifiers of the dual-branch heterogeneous feature fusion network, and using unlabeled sample sets from the target domain to train the domain discriminator based on the adversarial network to achieve deep domain confusion and realize deep decoupling of fusion features and actual working conditions.

[0064] The total loss function of the training process includes the focal loss that strengthens the imbalance between classes and the imbalance between difficult and easy samples within a class, the joint deep domain adaptation loss and the adversarial network domain confusion loss, as follows:

[0065]

[0066] in, is the total loss function of the training process; Focus loss to strengthen the imbalance between classes and the imbalance between difficult and easy samples within a class; is the joint domain adaptation loss; is to combat the network domain confusion loss; λ and μ are weight parameters that control the proportion of each loss.

[0067] Specifically, the focal loss with dynamic weight allocation In order to strengthen the learning of misclassified samples by dynamically specializing weights for different categories of samples based on the focus loss function, it is used to alleviate the problems of imbalance between classes and imbalance between difficult and easy samples within a class. Specifically, for the focus loss function:

[0068]

[0069] Among them, C is the total number of categories, y i is the true label encoded by one-hot, p is the output vector of the model after the Softmax function, p i Indicates the probability of predicting the i-th category, (1-p i ) γ is a modulation factor used to reflect the weight of classification errors of different categories. γ (γ ≥ 0) represents the focus parameter that controls the degree of weight distribution of difficult and easy samples. The larger the γ value, the lower the loss contribution of easy-to-classify samples. When γ = 0, the focus loss degenerates into cross-entropy loss. This value is empirically set to 2.

[0070] For the proposed dynamic focus loss function, the average of F1-score and G-mean is assigned as the dynamic category weight, so that The loss weight can be dynamically adjusted according to the actual performance of each category to prevent the model from ignoring difficult categories. Finally, the focal loss calculation formula that strengthens the imbalance between classes and the imbalance of difficult and easy samples within a class is as follows:

[0071]

[0072] Among them, F1 i and Gmean i are the F1-score and G-mean values of category i, which respectively reflect the balance between the precision and recall of the category (i.e., the positive classification performance) and the overall recognition balance of the category (i.e., the increased emphasis on the negative class). The calculation formula is as follows:

[0073]

[0074] Joint domain adaptation loss It is the total loss of domain adaptation distribution based on the joint maximum mean difference of features and classification results, which can be expressed as the following formula:

[0075]

[0076] Where K(·) is the Gaussian kernel function; n s With n t Indicates the number of samples from the source domain and the target domain; n L Indicates the total number of layers involved in the joint domain adaptation difference calculation, which is set to 2 in this implementation; and Represent the input from the source domain and the target domain on the lth layer, specifically, z l ∈{f l ,y l} is the intermediate feature f and the final output result y l combination.

[0077] Adversarial Network Domain Confusion Loss To implement the cross entropy loss function for domain classification, it is expressed as follows:

[0078]

[0079] Among them, p d is the output vector of the domain discriminator.

[0080] In summary, the training process of the cross-domain mechanical fault diagnosis model of the present invention is as follows:

[0081]

[0082]

[0083] In the following specific implementation cases, the experimental results of cross-load migration for different inter-class imbalance ratios show that the cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data provided by the present invention can achieve the best diagnostic effect in different migration tasks, and provides an effective solution for cross-domain migration diagnosis based on multimodal data with imbalanced class distribution.

[0084] Specifically, the verification is performed using a multimodal heterogeneous dataset obtained from the variable load gearbox experimental platform of Northwestern Polytechnical University.

[0085] Step 1: Data acquisition. The infrared thermal imaging images and vibration signals collected by a non-contact infrared thermal imager and a vibration accelerometer mounted on the gearbox surface are used in the verification experiment of the method. The experiment collected multimodal data of six different health states of gearboxes: baseline state (BS), 600ml oil shortage (OS1), 1100ml oil shortage (OS2), 50% gear tooth breakage (TB1), 100% gear tooth breakage (TB2), and reduced lubricant viscosity (VD). The infrared thermal image resolution is 3×1440×1080, and the vibration signal sampling frequency is 96kHz to ensure the effective capture of high-frequency fault characteristics.

[0086] Step 2: Data preprocessing. For vibration signals, the data set is divided using a sliding window sampling method. Considering that too high a sampling frequency may result in the signal data of a vibration cycle not being fully covered within a limited number of sampling points, the vibration signal is downsampled by a period of 4 to obtain a sample with a signal segment length of 2048. For infrared thermal imaging image signals, since the original image resolution is too large to support batch training, all image signals are downsampled by (4, 4) to obtain image signals with a size of 3×360×270. To simulate the scenario of scarcity of fault samples in actual industry, two types of imbalanced data sets are constructed: (1) exponential inter-class imbalance (simulating long-tail distribution), that is, the sample size decreases exponentially with the severity of the fault, which is quantified by the exponential coefficient σ; (2) step-type inter-class imbalance (simulating polarization distribution), that is, the fault class samples are extremely scarce compared to the normal samples, which is quantified by the imbalance ratio β (the maximum class sample size / the minimum class sample size). All test sets maintain a balanced distribution to ensure the objectivity of performance evaluation. The specific data partitioning strategy is shown in Table 1.

[0087] Table 1 Dataset division details

[0088]

[0089]

[0090] Step 3: Build a dual-branch heterogeneous feature fusion network. The specific parameters are shown in the following table:

[0091] Table 2 Specific parameters of dual-branch heterogeneous feature fusion network

[0092]

[0093] Step 4: Twelve migration tasks were set up based on four different load scenarios (0%, 30%, 70%, and 100%). The established model was trained based on the partitioned dataset. During training, the adaptive gradient descent algorithm (Adam) was used to optimize model parameters. The training epochs were set to 100, and the performance on the test set after training was used as the final metric for the model's diagnostic accuracy.

[0094] To demonstrate the superiority of the proposed method, we introduce five widely used models in the field of transfer learning for comparison: a method based on traditional cross entropy loss (CE loss), a method based on focal loss, a method based on LDAM loss, a method based on Dice loss, and a method based on balanced softmax. The comparative analysis of the six models is as follows:

[0095] Table 3 Diagnosis results of different migration tasks

[0096]

[0097] (1) Figure 3 As shown in Table 3, the present invention can achieve the highest average accuracy on different migration tasks, reaching an average of 93.27% (corresponding to Figure 3 ), and in most cases the diagnostic accuracy is better than other models in the same field.

[0098] (2) Taking the migration task L0→L2 as an example, we further analyze the specific diagnostic performance of each method. Figure 4 As shown. Figure 4 It can be seen that the confusion matrix clearly and intuitively presents the classification accuracy and error of different methods in this task. Figure 4 The data in the figure shows that the proposed method correctly classifies all fault categories, achieving optimal performance. In contrast, the method using the LDAM loss function, while achieving high accuracy, still exhibits some misclassification (for example, samples with the true label TB2 are partially misclassified as VD and BL). Notably, the balanced softmax method and the focal loss method exhibit systematic confusion between faults OS1 and OS2, and between TB1 and TB2, indicating that these comparison methods struggle to effectively assess fault severity.

[0099] On the other hand, to further address the challenges of scarcity of fault data and imbalance of different degrees in actual industrial scenarios, the diagnostic performance of the proposed method under simulated imbalance conditions is evaluated based on the constructed datasets with different imbalance ratios. Six migration tasks with more significant load changes are selected to characterize the cross-domain diagnostic effectiveness of the model. The experimental results are summarized in Table 4 and Figure 5 .

[0100] Table 4 Diagnosis results of different inter-class imbalance ratio models

[0101]

[0102] It can be seen from the results that as the coefficient increases, the model's feature extraction ability for minority class samples degrades significantly, and the accuracy drops significantly. It is worth noting that under the same degree of imbalance, the performance degradation caused by step-type imbalance is more serious than that caused by exponential imbalance, which shows that it is more challenging to handle sudden changes in category distribution. Secondly, the results of the cross-load migration task show that under mild imbalance conditions (such as β = 2 or σ ≤ 50), the model can still maintain a high accuracy of more than 90% in most load migration scenarios, which proves the adaptability of multimodal feature fusion to domain shifts. In addition, for exponential and step-type imbalanced datasets, although the overall diagnostic performance will decrease when β or σ takes extremely large values, the method of the present invention can still maintain a diagnostic accuracy of more than 80% in some tasks, which to a certain extent demonstrates the stability of the method in locating key fault features. In summary, the results of the above examples show that the method of the present invention can achieve the average maximum diagnostic accuracy in different migration tasks; and the verification experiments based on different imbalance ratios also show that the method of the present invention always maintains robust diagnostic performance, and the accuracy rate can still continue to exceed 90% under most imbalanced data conditions, thereby verifying the good cross-condition diagnostic performance of the method of the present invention in the context of imbalanced and multimodal data.

[0103] In one embodiment, the present invention provides a computer device, comprising a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data.

[0104] In one embodiment, the present invention provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data in the above embodiment.

[0105] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0109] The present invention also provides a computer program product, which is used to execute any of the above-described methods for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data. Since the computer program product provided by the present invention and the above-described method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data are based on the same inventive concept, the computer program product provided by the present invention has all the advantages of the above-described method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data. Therefore, the beneficial effects of the computer program product provided by the present invention will not be detailed here.

[0110] In the description of the present invention, it should be understood that the terms "upper", "lower", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0112] In the present invention, unless otherwise specified or limited, the terms "connected" and "connection" should be understood in a broad sense. For example, they can mean fixed connection, detachable connection, or integration; mechanical connection, electrical connection, or communication; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0113] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0114] In the present invention, the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0115] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data, characterized in that: include: Acquire a multimodal heterogeneous signal of the mechanical equipment to be diagnosed, wherein the multimodal heterogeneous signal includes a one-dimensional vibration signal and a two-dimensional infrared thermal imaging image signal; Inputting the multimodal heterogeneous signal into a pre-trained cross-domain mechanical fault diagnosis model and outputting a diagnosis result; Among them, the cross-domain mechanical fault diagnosis model is obtained by training a dual-branch heterogeneous feature fusion network using training data. The dual-branch heterogeneous feature fusion network includes a dual-branch heterogeneous feature extractor, a feature splicing layer, and a parallel feature classifier and a domain discriminator based on an adversarial network. The dual-branch heterogeneous feature extractor includes a parallel vibration signal feature extractor and an image signal feature extractor; the training data includes one-dimensional vibration signals and two-dimensional infrared thermal imaging image signals from a certain source domain working condition and corresponding fault labels, as well as one-dimensional vibration signals and two-dimensional infrared thermal imaging image signals from another target domain working condition; the total loss function of the training process includes focal loss that strengthens inter-class imbalance and intra-class difficult and easy sample imbalance, joint deep domain adaptation loss and adversarial network domain confusion loss.

2. The cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to claim 1 is characterized in that: The vibration signal feature extractor includes several multi-layer convolution-pooling layers of different scales, and the image signal feature extractor is a lightweight ResNet network with pruned channels and network layers; the feature splicing layer is used to achieve heterogeneous feature fusion.

3. The cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to claim 2 is characterized in that: The vibration signal feature extractor realizes deep feature mining through multiple convolution-pooling layers with different scale convolution kernel sizes, specifically: Among them, ConvPooling i Represents the i-th convolution-pooling operation; is the layer-by-layer vibration feature obtained after convolution pooling with different convolution kernel sizes, where is the initial input x vbr ; B×K×N represent the batch size of input samples, the number of feature maps and the input feature length respectively; The image signal feature extractor realizes adaptive extraction of image features through two convolution blocks containing residual structures, specifically: Among them, ResidualConv j Represents the jth residual convolution; is the layer-by-layer image feature obtained after residual convolution operations at different layers, where That is the initial input x irt ; The feature concatenation layer combines the features of different modalities through the feature concatenation operation Concat to achieve feature fusion and flatten them into two dimensions, thereby obtaining the final output f of the dual-branch heterogeneous feature extractor. fused , specifically: Among them, B×L f Represents the final shape of the fused features.

4. The cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to claim 2 is characterized in that: The feature classifier and the domain discriminator based on the adversarial network are both multi-layer perceptron structures, wherein the feature classifier is used to obtain the final diagnostic output of the cross-domain mechanical fault diagnosis model according to the fusion features of the feature splicing layer, and the domain discriminator based on the adversarial network is used to confuse the source domain and target domain sources of the extracted features to extract the domain-invariant features of the signal.

5. The cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to claim 1 is characterized in that: The total loss function of the training process is as follows: in, is the total loss function of the training process; Focus loss to strengthen the imbalance between classes and the imbalance between difficult and easy samples within a class; is the joint domain adaptation loss; To combat the network domain confusion loss; λ and μ are weight parameters that control the proportion of each loss; The focus loss that strengthens the imbalance between classes and the imbalance between difficult and easy samples within a class is specifically: Where C is the total number of categories; y i is the true label encoded by one-hot; p is the result of the model's output vector after passing the Softmax function; p i Indicates the probability of predicting the i-th category; (1-p i ) γ is a modulation factor used to reflect the weight of classification errors of different categories; γ represents the focus parameter that controls the degree of weight distribution of difficult and easy samples. The larger the γ value, the lower the loss contribution of easy-to-classify samples. When γ = 0, the focus loss degenerates into cross entropy loss; F1 i and Gmean i are the F1-score and G-mean values of category i, which respectively reflect the balance between the precision and recall of the category and the overall recognition balance of the category.

6. The cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to claim 5 is characterized in that: The joint domain adaptation loss is specifically: Where K(·) is the Gaussian kernel function; n s With n t Indicates the number of samples from the source domain and the target domain; n L Indicates the total number of layers involved in the joint domain adaptation difference calculation; and Represent the input from the source domain and the target domain on the lth layer, z l ∈{f l ,y l } is the intermediate feature f l And the final output y l combination.

7. The cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to claim 6 is characterized in that: The adversarial network domain confusion loss is specifically: Among them, p d is the output vector of the domain discriminator.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that When the computer program product is executed by a processor, it implements the cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data as described in any one of claims 1 to 7.

Images