Cross-domain mechanical fault diagnosis method and device based on non-equilibrium multi-modal heterogeneous data

Abstract

The application discloses a kind of based on unbalanced multimodal heterogeneous data's cross-domain mechanical fault diagnosis method and device, obtain the multimodal heterogeneous signal of mechanical equipment to be diagnosed, multimodal heterogeneous signal includes one-dimensional vibration signal and two-dimensional infrared thermal imaging image signal;Multiple modal heterogeneous signals are input into the cross-domain mechanical fault diagnosis model that is pre-trained, and output diagnosis result;Cross-domain mechanical fault diagnosis model is obtained using training data to train double-branch heterogeneous feature fusion network, double-branch heterogeneous feature fusion network includes double-branch heterogeneous feature extractor, feature splicing layer and the feature classifier and the domain discriminator based on parallelly arranged adversarial network, double-branch heterogeneous feature extractor includes parallelly arranged vibration signal feature extractor and image signal feature extractor;The total loss function of training process includes the focal loss of reinforcing inter-class imbalance and intra-class difficult sample imbalance, joint deep domain self-adaptive loss and adversarial network domain confusion loss.The purpose of the application is to solve the problem of low utilization rate of multiple source heterogeneous data by multimodal feature fusion network.

Description

Technical Field

[0001] This invention belongs to the field of mechanical equipment fault diagnosis technology, specifically relating to a cross-domain mechanical fault diagnosis method and device based on unbalanced multimodal heterogeneous data. Background Technology

[0002] Equipment Health Management (PHM), as a core technology of intelligent operation and maintenance in modern industry, has irreplaceable strategic value in the full lifecycle management of critical equipment such as aero-engines and gas turbines. Data-driven intelligent diagnostic methods, with their powerful feature extraction capabilities, have become the most promising technological direction in the field of equipment health management. These methods not only achieve automatic extraction of fault features through end-to-end learning, avoiding the subjectivity and limitations of traditional manual feature extraction processes, but also possess the ability to characterize complex nonlinear fault modes. However, the single-sensor monitoring schemes commonly used in current industrial settings have significant limitations in perception dimensions: on the one hand, key fault features are easily lost when sensors are limited by installation location or environmental interference; on the other hand, single-dimensional perception data cannot comprehensively depict the evolution of equipment status under complex operating conditions. This perception blind spot problem severely restricts the high-precision requirements of modern industry for early fault warning. The widespread adoption of multi-sensor networks in the era of industrial big data provides a new opportunity to overcome this predicament. Multi-source sensor networks enable the simultaneous acquisition of multi-modal data such as vibration, acoustic emission, and thermal imaging, significantly improving state perception capabilities through the complementary characteristics between modes. Compared to traditional single-channel sensing methods, multi-source sensing exhibits significant advantages in signal diversity, robustness, and comprehensive fault characterization.

[0003] However, current fault diagnosis methods based on multi-source sensing still face several key challenges: First, multimodal data such as acoustic, vibration, and temperature data collected by different sensors often exhibit significant differences in sampling frequency, signal-to-noise ratio, and feature distribution. Traditional methods struggle to fully exploit their collaborative representational value, resulting in low utilization of multi-source heterogeneous data. Second, due to the sporadic nature of mechanical equipment failures, time-series signals collected by industrial monitoring systems often exhibit extreme class imbalances. This data characteristic makes traditional diagnostic models prone to majority class dominance, causing global accuracy metrics to mask the risk of missed detections of critical faults. Furthermore, domain differences during knowledge transfer can lead to a superposition of class distribution shifts and sample imbalances between the source and target domains, resulting in cross-domain distortion of minority class representations.

[0004] Tong Zheming et al. published a Chinese patent, "A Fault Diagnosis Method for Harmonic Reducers Based on Multi-Source Heterogeneous Data Fusion" (Patent Publication No. CN119557697A). This method collects raw data from multiple sensors, including vibration, torque, and temperature, under different fault modes. It converts the signals into two-dimensional time-frequency images and fuses the features of the multi-source heterogeneous data to output the fault diagnosis result. However, this method is only suitable for balanced datasets under constant operating conditions. When faced with real-world industrial data acquisition scenarios where the model's operating conditions change significantly and the distribution of sample classes is extremely unbalanced, the above method inevitably suffers from a decline in diagnostic performance, thus exhibiting shortcomings in practical industrial applications. Therefore, how to construct an inherently robust balanced learning mechanism and effective cross-domain knowledge transfer methods within a multi-source heterogeneous framework has become a key breakthrough in improving the generalization ability of fault diagnosis for multimodal industrial data. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a cross-domain mechanical fault diagnosis method and device based on unbalanced multimodal heterogeneous data. Its purpose is to solve the problem of low utilization rate of multi-source heterogeneous data by using a multimodal feature fusion network; to address the problem of class imbalance in industrial monitoring data by adopting a dynamic weighted unbalanced learning mechanism; and to eliminate class distribution shift in cross-domain migration by constructing a hybrid adversarial domain adaptation framework, thereby realizing cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data.

[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0007] According to a first aspect of the present invention, a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data is provided, comprising:

[0008] Acquire multimodal heterogeneous signals of the mechanical equipment to be diagnosed, wherein the multimodal heterogeneous signals include one-dimensional vibration signals and two-dimensional infrared thermal imaging image signals;

[0009] The multimodal heterogeneous signals are input into a pre-trained cross-domain mechanical fault diagnosis model, and the diagnosis results are output.

[0010] The cross-domain mechanical fault diagnosis model is trained using training data on a dual-branch heterogeneous feature fusion network. This network includes a dual-branch heterogeneous feature extractor, a feature splicing layer, and a parallel feature classifier and an adversarial network-based domain discriminator. The dual-branch heterogeneous feature extractor includes a parallel vibration signal feature extractor and an image signal feature extractor. The training data includes one-dimensional vibration signals and two-dimensional infrared thermal imaging images from a source domain, along with corresponding fault labels, and one-dimensional vibration signals and two-dimensional infrared thermal imaging images from another target domain. The total loss function during training includes focus loss to reinforce inter-class imbalance and intra-class imbalance of easy and difficult samples, joint deep domain adaptive loss, and adversarial network domain confusion loss.

[0011] In one possible implementation of the first aspect, the vibration signal feature extractor comprises several multi-layer convolutional-pooling layers of different scales, and the image signal feature extractor is a lightweight ResNet network with a reduced number of channels and pruned network layers; the feature splicing layer is used to achieve heterogeneous feature fusion.

[0012] In one possible implementation of the first aspect, the vibration signal feature extractor achieves deep feature mining through multiple convolutional-pooling layers with convolutional kernels of different sizes, specifically:

[0013]

[0014] Among them, ConvPooling i This represents the i-th convolution-pooling operation; These are the layer-by-layer vibrational features obtained after convolutional pooling operations with kernel sizes of different scales, where... For the initial input x vbr B×K×N represent the batch size of the input samples, the number of feature maps, and the length of the input features, respectively.

[0015] The image signal feature extractor achieves adaptive extraction of image features through two convolutional blocks containing residual structures, specifically:

[0016]

[0017] Among them, ResidualConv j Represents the j-th residual convolution; These are the layer-by-layer image features obtained after convolutional operations on different layers of residuals, where... That is, the initial input x irt ;

[0018] The feature concatenation layer fuses features from different modalities through a feature concatenation operation (Concat) and flattens them into two dimensions, thereby obtaining the final output f of the dual-branch heterogeneous feature extractor. fused Specifically:

[0019]

[0020] Among them, B×L f The final shape representing the fusion characteristics.

[0021] In one possible implementation of the first aspect, both the feature classifier and the domain discriminator based on the adversarial network are multilayer perceptron structures. The feature classifier is used to obtain the final diagnostic output of the cross-domain mechanical fault diagnosis model based on the fusion features of the feature splicing layer. The domain discriminator based on the adversarial network is used to confuse the source domain and the target domain of the extracted features in order to extract the domain-invariant features of the signal.

[0022] In one possible implementation of the first aspect, the total loss function of the training process is specifically as follows:

[0023]

[0024] in, The total loss function for the training process; To enhance the focus loss for imbalance between classes and imbalance between easy and difficult samples within a class; For joint domain adaptive loss; To combat network domain confusion loss; λ and μ are weighting parameters that control the proportion of each loss term;

[0025] The focus loss that enhances the imbalance between classes and the imbalance between easy and difficult samples within a class is specifically as follows:

[0026]

[0027] Where C is the total number of categories; y i is the true label encoded by one-hot encoding; p is the result of the model's output vector after passing through the Softmax function; pi represents the probability of being predicted as the i-th class; (1-p i ) γ γ is a modulation factor used to represent the weight of classification errors for different categories; γ represents a focusing parameter that controls the degree of weight allocation between easy and difficult samples. The larger the value of γ, the lower the loss contribution of easy-to-classify samples. When γ = 0, the focus loss degenerates into cross-entropy loss; F1 i With Gmean iF1-score and G-mean values ​​for category i reflect the balance between precision and recall for that category, as well as the overall recognition balance of that category.

[0028] In one possible implementation of the first aspect, the joint domain adaptive loss is specifically:

[0029]

[0030] Where K(·) is the Gaussian kernel function; n s With n t n represents the number of samples from the source and target domains; L This indicates the total number of layers involved in the joint domain adaptation difference calculation; and Z represents the input from the source domain and the target domain at layer l, respectively. l ∈{f l ,y l} is an intermediate feature f l With the final output result y l The combination of.

[0031] In one possible implementation of the first aspect, the adversarial network domain confusion loss specifically refers to:

[0032]

[0033] Where, p d This is the output vector of the domain discriminator.

[0034] According to a second aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the aforementioned method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data.

[0035] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data.

[0036] According to a fourth aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the aforementioned method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] This invention provides a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data, which effectively solves the problems of low utilization rate of multi-source heterogeneous data, imbalanced class distribution, and degraded cross-condition transfer performance in current industrial scenarios. Specifically, firstly, a dual-branch heterogeneous feature fusion network is constructed to achieve deep fusion of multimodal features, fully exploring the complementary representational information of vibration and thermal imaging data. Secondly, based on the proposed dynamic focus loss function, the inter-class imbalance difference and intra-class easy / difficult sample difference are jointly optimized to enhance the model's ability to identify minority class faults and effectively suppress the risk of missed fault detection caused by majority class dominance. In addition, a hybrid adversarial domain adaptation framework is designed to achieve deep domain obfuscation, and combined with a deep domain adaptive loss function, the feature extractor is forced to learn condition-invariant features, effectively reducing the distribution difference between the source and target domains and ensuring cross-domain consistency of minority class fault representation. Compared with traditional single-modal methods, the dual-branch heterogeneous fusion network proposed in this invention shows significant advantages in cross-condition fault diagnosis. Especially in industrial monitoring scenarios with extremely unbalanced sample distribution, it can effectively improve the collaborative utilization efficiency of multi-source heterogeneous data from vibration and thermal imaging, achieve high-precision cross-domain fault diagnosis, and has important engineering application value.

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the specific embodiments of the present invention, the drawings used in the description of the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data according to the present invention;

[0042] Figure 2 This is a training diagram of the dual-branch heterogeneous feature fusion network of the present invention;

[0043] Figure 3 These are the diagnostic results for different transfer tasks on the heterogeneous dataset of Northwestern Polytechnical University in this embodiment of the invention.

[0044] Figure 4 This is an example of confusion matrix analysis of the heterogeneous dataset from Northwestern Polytechnical University in this invention embodiment;

[0045] Figure 5 This is the diagnostic result of the imbalance ratio of different samples in the heterogeneous dataset of Northwestern Polytechnical University in this embodiment of the invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, this invention provides a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data, specifically including the following steps:

[0048] Step 1: Acquire the multimodal heterogeneous signals of the mechanical equipment to be diagnosed. The multimodal heterogeneous signals include one-dimensional vibration signals and two-dimensional infrared thermal imaging image signals.

[0049] Specifically, a vibration signal sensor is used to collect one-dimensional vibration signals of the mechanical equipment to be diagnosed, and an infrared thermal imager is used to collect two-dimensional infrared thermal image signals of the mechanical equipment to be diagnosed. The vibration signal sensor and the infrared thermal imager collect their respective signals simultaneously.

[0050] It should be understood that the collected one-dimensional vibration signal and two-dimensional infrared thermal imaging image signal should be preprocessed, and the length of the one-dimensional vibration signal and two-dimensional infrared thermal imaging image signal should meet the input requirements of the cross-domain mechanical fault diagnosis model by random sliding window sampling. Downsampling operation should also be considered to solve the problem of incomplete periodic distribution of vibration signal caused by excessive sampling rate.

[0051] Step 2: Input the multimodal heterogeneous signals into the pre-trained cross-domain mechanical fault diagnosis model and output the diagnosis results. The cross-domain mechanical fault diagnosis model is obtained by training a dual-branch heterogeneous feature fusion network using training data.

[0052] Specifically, combined Figure 2 As shown, the dual-branch heterogeneous feature fusion network includes a dual-branch heterogeneous feature extractor, a feature concatenation layer, and a parallel feature classifier and a domain discriminator based on an adversarial network. The dual-branch heterogeneous feature extractor, through multi-branch design for different modalities, can extract features from the input multimodal signals separately. The dual-branch heterogeneous feature extractor includes a parallel vibration signal feature extractor and an image signal feature extractor. Specifically, the vibration signal feature extractor contains several multi-layer convolutional-pooling layers of different scales, and the image signal feature extractor is a lightweight ResNet network with optimized channel count and network layer pruning. The feature concatenation layer is used to achieve heterogeneous feature fusion.

[0053] For vibration signal feature extractors, deep feature mining can be achieved by stacking multiple convolutional-pooling layers with different kernel sizes. Specifically:

[0054]

[0055] Among them, ConvPooling i This represents the i-th convolution-pooling operation; These are the layer-by-layer vibrational features obtained after convolutional pooling operations with kernel sizes of different scales, where... For the initial input x vbr B×K×N represent the batch size of the input samples, the number of feature maps, and the length of the input features, respectively.

[0056] For the image signal feature extractor, adaptive extraction of image features is achieved through two convolutional blocks containing residual structures, specifically:

[0057]

[0058] Among them, ResidualConv j This represents the j-th residual convolution, which originates from the standard operation of the residual block in the ResNet-18 model structure. It is obtained by adding a residual connection structure to two convolutional layers. These are the layer-by-layer image features obtained after convolutional operations on different layers of residuals, where... That is, the initial input x irt .

[0059] The resulting different modal features are fused and flattened into two dimensions using the feature concatenation operation Concat, thus obtaining the final output f of the dual-branch heterogeneous feature extractor. fused Specifically:

[0060]

[0061] Among them, B×L f The final shape representing the fusion feature, in this implementation design, L f It is defined as 128.

[0062] Both the feature classifier and the domain discriminator based on adversarial networks (ADNs) are multilayer perceptron structures. The feature classifier obtains the final diagnostic output of the cross-domain mechanical fault diagnosis model based on the fused features of the feature concatenation layer. The ADN-based domain discriminator obfuscates the source and target domain sources of the extracted features to extract domain-invariant features of the signal. Specifically, each linear connection layer of the feature classifier uses the ReLU function as the activation function for nonlinear mapping, and the model output is obtained through representation learning from the extracted fused features. Similarly, each linear connection layer of the ADN-based domain discriminator uses the ReLU function as the activation function for nonlinear mapping, and a gradient reversal layer is set during gradient propagation to obfuscate the source and target domain sources of the extracted features, promoting deeper mining of domain-invariant features by each feature extractor.

[0063] The training data includes one-dimensional vibration signals and two-dimensional infrared thermal imaging images from a source domain operating condition, along with corresponding fault labels, and one-dimensional vibration signals and two-dimensional infrared thermal imaging images from a target domain operating condition. Specifically, multimodal heterogeneous datasets are acquired from vibration signal sensors and infrared thermal imagers, and signals are collected and processed under different operating conditions. Labeled signal samples, due to their complete labeling and ease of acquisition, can be considered source domain data; while unlabeled signal samples, due to their lack of labeling, often conform to the characteristics of signal acquisition under actual operating conditions and can therefore be considered target domain data. To prepare training and testing datasets that conform to the distribution characteristics of actual industrial data, data preprocessing is performed on the source and target domain signal samples. The preprocessing steps include: obtaining a sample set that meets the input model length through random sliding window sampling, and considering downsampling to address the problem of incomplete vibration signal period distribution caused by excessive sampling rate. In this embodiment, the model sampling length is set to 2048, and the downsampling rate is 4. Furthermore, datasets with imbalanced proportions between different classes are partitioned and prepared for use in the model training process. The training process of the model includes: inputting source domain data and target domain data into a dual-branch heterogeneous feature fusion network; training each feature extractor and feature classifier of the dual-branch heterogeneous feature fusion network using a labeled sample set from the source domain; and training a domain discriminator based on an adversarial network using an unlabeled sample set from the target domain to achieve deep domain confusion and achieve deep decoupling between the fused features and the actual working conditions.

[0064] The overall loss function during training includes the focus loss that reinforces the imbalance between classes and the imbalance between difficult and easy samples within classes, the joint deep domain adaptation loss, and the adversarial network domain confusion loss, as detailed below:

[0065]

[0066] in, The total loss function for the training process; To enhance the focus loss for imbalance between classes and imbalance between easy and difficult samples within a class; For joint domain adaptive loss; To counteract network domain confusion loss; λ and μ are weight parameters that control the proportion of each loss.

[0067] Specifically, the focus loss of dynamic weight allocation To enhance the learning of misclassified samples by dynamically specializing weights for different class samples based on the focus loss function, thereby simultaneously mitigating the problems of inter-class imbalance and intra-class imbalance between easy and difficult samples, specifically, regarding the focus loss function:

[0068]

[0069] Where C is the total number of categories, y i p is the true label encoded by one-hot encoding, and p is the result of the model's output vector after passing through the Softmax function. i This represents the probability of being predicted as class i, (1-p) i ) γ γ is a modulation factor used to reflect the weight of classification errors for different categories. γ (γ≥0) represents the focusing parameter that controls the degree of weight allocation between easy and difficult samples. The larger the value of γ, the lower the loss contribution of easy-classified samples. When γ=0, the focus loss degenerates into cross-entropy loss. This value is empirically set to 2.

[0070] For the proposed dynamic focus loss function, the average of the F1-score and G-mean is used as the dynamic class weights, so that... The loss weights can be dynamically adjusted based on the actual performance of each category, preventing the model from ignoring difficult categories. Finally, the formula for calculating the focus loss, which reinforces the imbalance between classes and the imbalance between easy and difficult samples within a class, is as follows:

[0071]

[0072] Among them, F1 i With Gmean i Let F1 and G-mean be the values ​​of category i. They reflect the balance between precision and recall (i.e., positive class classification performance) and the overall recognition balance of the category (i.e., increased emphasis on negative classes), respectively. Their calculation formulas are as follows:

[0073]

[0074] Joint domain adaptive loss The total loss for the joint maximum mean difference domain adaptation distribution based on features and classification results can be expressed as the following formula:

[0075]

[0076] Where K(·) is the Gaussian kernel function; n s With n t n represents the number of samples from the source and target domains; L This represents the total number of layers participating in the joint domain adaptation difference calculation, which is set to 2 in this embodiment; and Let z represent the inputs from the source and target domains at layer l, respectively. Specifically, z l ∈{f l ,y l} represents the intermediate features f and the final output y. l The combination of.

[0077] Adversarial network domain confusion loss The cross-entropy loss function for domain classification is expressed as follows:

[0078]

[0079] Where, p d This is the output vector of the domain discriminator.

[0080] In summary, the training process of the cross-domain mechanical fault diagnosis model of this invention is as follows:

[0081]

[0082]

[0083] In the following specific implementation cases, the cross-load migration experiment results for different class imbalance ratios show that the cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data provided by the present invention can achieve the best diagnosis effect in different migration tasks, and provides an effective solution for cross-domain migration diagnosis based on multimodal data with class distribution imbalance.

[0084] Specifically, the validation was performed using a multimodal heterogeneous dataset obtained from the variable load gearbox experimental platform at Northwestern Polytechnical University.

[0085] Step 1: Data Acquisition. Infrared thermal images and vibration signals acquired using a non-contact infrared thermal imager and a vibration acceleration sensor mounted on the gearbox surface were used for verification experiments of the method. The experiment collected multimodal data on the gearbox under six different health conditions: baseline (BS), 600ml oil shortage (OS1), 1100ml oil shortage (OS2), 50% tooth breakage (TB1), 100% tooth breakage (TB2), and reduced lubricating oil viscosity (VD). The infrared thermal image resolution was 3×1440×1080, and the vibration signal sampling frequency was 96kHz to ensure effective capture of high-frequency fault characteristics.

[0086] Step 2: Data preprocessing. For vibration signals, the dataset is divided using sliding window sampling. Considering that excessively high sampling frequency may not be able to completely cover the signal data of a vibration cycle within a limited number of sampling points, the vibration signals are downsampled with a period of 4 to obtain samples with a signal segment length of 2048. For infrared thermal imaging image signals, since the original image resolution is too large to support batch training, all image signals are downsampled by (4,4) to obtain image signals with a size of 3×360×270. To simulate the scenario of scarce fault samples in actual industry, two types of imbalanced datasets are constructed: (1) exponential inter-class imbalance (simulating long-tail distribution), that is, the sample size decreases exponentially with the severity of the fault, which is quantified by the exponential coefficient σ; (2) step-type inter-class imbalance (simulating polarized distribution), that is, fault class samples are extremely scarce compared with normal samples, which is quantified by the imbalance ratio β (maximum class sample size / minimum class sample size). All test sets maintain a balanced distribution to ensure the objectivity of performance evaluation. The specific data partitioning strategy is shown in Table 1.

[0087] Table 1. Details of Dataset Partitioning

[0088]

[0089]

[0090] Step 3: Construct a dual-branch heterogeneous feature fusion network, the specific parameters of which are shown in the table below:

[0091] Table 2. Specific parameters of the dual-branch heterogeneous feature fusion network

[0092]

[0093] Step 4: Twelve transfer tasks were set up based on four different load scenarios (0%, 30%, 70%, and 100% load). The established model was trained on the partitioned dataset. During training, the adaptive gradient descent algorithm (Adam) was used to optimize the model parameters. The training epochs were set to 100, and the performance on the test set after training was used as the final metric for the model's diagnostic accuracy.

[0094] To demonstrate the superiority of the proposed method, five widely used models in transfer learning are introduced for comparison: methods based on traditional cross-entropy loss (CE loss), methods based on focal loss, methods based on LDAM loss, methods based on Dice loss, and methods based on Balanced softmax. The comparative analysis of these six models is as follows:

[0095] Table 3 Diagnostic results for different migration tasks

[0096]

[0097] (1) As Figure 3 As shown in Table 3, the present invention achieves the highest average accuracy across different transfer tasks, averaging 93.27% (corresponding to...). Figure 3 The proposed method (in this paper) has superior diagnostic accuracy compared to other models in the same field in most cases.

[0098] (2) Taking the transfer task L0→L2 as an example, the specific diagnostic performance of each method is further analyzed. This is achieved using confusion matrix analysis, such as... Figure 4 As shown. By Figure 4 As can be seen, the confusion matrix clearly and intuitively presents the classification accuracy and error rate of different methods in this task. Figure 4 The information indicates that the method of this invention achieves correct classification of all fault categories, reaching optimal performance. In contrast, while the method using the LDAM loss function achieves higher accuracy, it still exhibits some misclassification (e.g., samples with the true label TB2 are partially misclassified as VD and BL). Notably, the Balanced softmax method and the focus loss method show systematic confusion between faults OS1 and OS2, and between TB1 and TB2, indicating that these comparative methods are difficult to effectively assess the severity of faults.

[0099] On the other hand, to further address the challenges of scarce fault data and varying degrees of class imbalance in real-world industrial scenarios, the diagnostic performance of the proposed method under simulated unbalanced operating conditions was evaluated based on datasets with different imbalance ratios. Six transfer tasks with more significant load changes were selected to characterize the model's cross-domain diagnostic efficacy. Experimental results are summarized in Table 4 and... Figure 5 .

[0100] Table 4 Diagnostic results of the imbalance ratio model among different categories

[0101]

[0102] The results show that as the coefficients increase, the model's ability to extract features from minority class samples significantly degrades, and the accuracy drops dramatically. Notably, under the same degree of imbalance, step imbalance causes a more severe performance degradation than exponential imbalance, indicating that handling abrupt changes in class distribution is more challenging. Secondly, the results on the cross-load migration task show that under mild imbalance conditions (e.g., β = 2 or σ ≤ 50), the model can still maintain a high accuracy of over 90% in most load migration scenarios, demonstrating the adaptability of multimodal feature fusion to domain shifts. Furthermore, for exponential and step imbalance datasets, although the overall diagnostic performance decreases when β or σ is extremely large, the method of this invention can still maintain a diagnostic accuracy of over 80% in some tasks, which to some extent demonstrates the stability of the method in locating key fault features. In summary, the results of the above embodiments show that the method of the present invention can achieve the average maximum diagnostic accuracy in different migration tasks; and the verification experiments based on different imbalance ratios also demonstrate that the method of the present invention maintains robust diagnostic performance, with the accuracy still exceeding 90% under most imbalanced data conditions. This verifies the good cross-condition diagnostic performance of the method of the present invention in the context of imbalanced, multimodal data.

[0103] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used in the operation of a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data.

[0104] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data in the above embodiment.

[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] This invention also provides a computer program product, which is used to execute any of the aforementioned methods for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data. Since the computer program product provided by this invention belongs to the same inventive concept as the aforementioned method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data, it possesses all the advantages of the aforementioned method. Therefore, the beneficial effects of the computer program product provided by this invention will not be elaborated upon here.

[0110] In the description of this invention, it should be understood that the terms "upper", "lower", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0112] In this invention, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0113] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0114] In this invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0115] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data, characterized in that, include: Acquire multimodal heterogeneous signals of the mechanical equipment to be diagnosed, wherein the multimodal heterogeneous signals include one-dimensional vibration signals and two-dimensional infrared thermal imaging image signals; The multimodal heterogeneous signals are input into a pre-trained cross-domain mechanical fault diagnosis model, and the diagnosis results are output. The cross-domain mechanical fault diagnosis model is trained using training data on a dual-branch heterogeneous feature fusion network. This network includes a dual-branch heterogeneous feature extractor, a feature splicing layer, and a parallel feature classifier and an adversarial network-based domain discriminator. The dual-branch heterogeneous feature extractor includes a parallel vibration signal feature extractor and an image signal feature extractor. The training data includes one-dimensional vibration signals and two-dimensional infrared thermal imaging images from a source domain, along with corresponding fault labels, and one-dimensional vibration signals and two-dimensional infrared thermal imaging images from another target domain. The total loss function during training includes focus loss to reinforce inter-class imbalance and intra-class imbalance of easy and difficult samples, joint deep domain adaptive loss, and adversarial network domain confusion loss. The total loss function for the training process is as follows: in, The total loss function for the training process; To enhance the focus loss for imbalance between classes and imbalance between easy and difficult samples within a class; For joint domain adaptive loss; To combat network domain confusion losses; and Weighting parameters for controlling the proportion of each type of loss; The focus loss that enhances the imbalance between classes and the imbalance between easy and difficult samples within a class is specifically as follows: in, It is the total number of categories; It is a true label encoded by one-hot encoding; It is the result of the model's output vector passing through the Softmax function; Indicates the prediction is the first i The probability of a class; This is a modulation factor used to represent the weight of classification errors for different categories; A focusing parameter that indicates the degree to which weights are allocated to easy and difficult samples. The larger the value, the lower the loss contribution of easily classified samples. The time-focus loss degenerates into cross-entropy loss; and For category i The F1-score and G-mean values ​​reflect the balance between precision and recall for this class, as well as the overall recognition balance of this class.

2. The method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data according to claim 1, characterized in that, The vibration signal feature extractor contains several multi-layer convolutional-pooling layers of different scales, and the image signal feature extractor is a lightweight ResNet network with reduced channel count and network layer pruning; the feature splicing layer is used to achieve heterogeneous feature fusion.

3. The method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data according to claim 2, characterized in that, The vibration signal feature extractor achieves deep feature mining through multiple convolution-pooling layers with different kernel sizes, specifically: in, Representing the i Sub-convolution-pooling operation; These are the layer-by-layer vibrational features obtained after convolution-pooling operations with different kernel sizes, where... For initial input ; These represent the batch size of the input samples, the number of feature maps, and the length of the input features, respectively. The image signal feature extractor achieves adaptive extraction of image features through two convolutional blocks containing residual structures, specifically: in, Representing the j Secondary residual convolution; These are the layer-by-layer image features obtained after convolutional operations on different layers of residuals, where... That is, the initial input ; The feature concatenation layer combines features from different modalities through feature concatenation operations. The feature is fused and flattened into two dimensions to obtain the final output of the dual-branch heterogeneous feature extractor. Specifically: in, The final shape representing the fusion characteristics.

4. The method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data according to claim 2, characterized in that, Both the feature classifier and the domain discriminator based on the adversarial network are multilayer perceptron structures. The feature classifier is used to obtain the final diagnostic output of the cross-domain mechanical fault diagnosis model based on the fusion features of the feature splicing layer. The domain discriminator based on the adversarial network is used to confuse the source domain and the target domain of the extracted features in order to extract the domain-invariant features of the signal.

5. The method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data according to claim 1, characterized in that, The joint domain adaptive loss is specifically as follows: in, The Gaussian kernel function; and This indicates the number of samples from the source domain and the target domain; This indicates the total number of layers involved in the joint domain adaptation difference calculation; and They represent the first time in the second month. l The inputs from the source and target domains at the layer, It is an intermediate feature With the final output The combination of.

6. The method for cross-domain mechanical fault diagnosis based on unbalanced multimodal heterogeneous data according to claim 5, characterized in that, The adversarial network domain confusion loss is specifically as follows: in, This is the output vector of the domain discriminator.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, When the computer program product is executed by the processor, it implements a cross-domain mechanical fault diagnosis method based on unbalanced multimodal heterogeneous data as described in any one of claims 1 to 6.

Images