Adaptive fault diagnosis method for multi-modal data volume imbalance
By combining multi-scale convolutional neural networks and deep reinforcement learning frameworks with a reward function that considers prior weights and the utilization of real-time modalities, the problem of data imbalance in multimodal fault diagnosis is solved, achieving more accurate and stable fault diagnosis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF ENERGY HEFEI COMPREHENSIVE NAT SCI CENT (ANHUI ENERGY LAB)
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-05
AI Technical Summary
Existing multimodal fault diagnosis methods have failed to effectively overcome the problem of data imbalance between modes, resulting in limited model robustness, generalization ability, and sensitivity to key fault symptoms.
We employ a multi-scale convolutional neural network and a deep reinforcement learning framework. By processing multimodal data through time alignment and normalization, we generate single-channel images using different time-series image encoding methods. By combining prior weights and a reward function that reflects the real-time modality utilization, we dynamically adjust the model's utilization of information from each modality.
It achieves fair and full utilization of multimodal information under imbalanced data conditions, improves the accuracy and robustness of diagnosis, reduces the risk of missed and false diagnoses, and enhances the stability and generalization performance of the system under complex working conditions.
Smart Images
Figure CN121980347A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent fault diagnosis technology, and in particular to an adaptive fault diagnosis method for multimodal data imbalance. Background Technology
[0002] With the rapid development of intelligent manufacturing and industrial IoT technologies, online condition monitoring and intelligent fault diagnosis of critical industrial equipment have become core means to ensure production safety and improve operational efficiency. In real-world industrial scenarios, information from a single sensor is often insufficient to comprehensively and robustly reflect the complex operating status and early signs of failure of equipment. Therefore, integrating multimodal sensor data from various physical quantities such as vibration, temperature, and current to improve the accuracy and reliability of diagnosis through information complementarity has become an important research direction in this field.
[0003] However, multimodal data fusion faces a common and severe challenge: a significant imbalance in the amount of data between modes. This mainly stems from the inherent order-of-magnitude differences in the sampling frequencies of sensors based on different physical principles. For example, vibration acceleration sensors typically acquire data at frequencies of several Hz or even higher, generating tens of thousands of data points per second; while temperature and pressure sensors usually have lower sampling frequencies, perhaps only a few Hz or lower, resulting in a vast difference in the number of effective samples from different modes within the same time window. This imbalance is not random noise, but a systematic bias.
[0004] Existing deep learning-based multimodal fault diagnosis methods mostly employ end-to-end feature fusion architectures, such as multichannel convolutional neural networks (CNNs) or attention-based fusion networks. These methods typically concatenate or weighted aggregate multimodal data at the input or feature layers. However, their optimization objectives (such as cross-entropy loss) are often geared towards overall classification accuracy, without explicitly considering the significant differences in the number of samples between modalities. During training, massive amounts of high-frequency modal data (such as vibration) dominate the gradient update direction, causing the model to tend to learn and rely on feature patterns in these modalities, while ignoring low-frequency modalities with sparse samples but potentially containing crucial fault information (such as gradual temperature changes indicating overheating trends). This essentially leads to "overfitting" of the model to high-data-volume modalities and "underlearning" of low-data-volume modalities. During testing, diagnostic performance significantly degrades once high-frequency modal signals are disturbed or specific faults are more clearly represented in low-frequency modalities.
[0005] To alleviate the problem of data imbalance, previous studies have attempted to introduce cost-sensitive learning or oversampling of a few modalities. However, these methods are mostly static or heuristic strategies, failing to dynamically adjust according to the real-time learning state during model training. In recent years, deep reinforcement learning (DRL) has been introduced into the field of fault diagnosis. Its agents learn the optimal diagnostic strategy through trial and error with the environment, demonstrating good adaptive potential. However, the reward function design in existing DRL diagnostic frameworks is mostly limited to the final classification correctness (such as sparse +1 / -1 rewards). This single accuracy reward signal cannot perceive the differences in the degree of utilization of information from different modalities during the agent's decision-making process. Therefore, traditional DRL methods also struggle to automatically balance the contributions of modalities with different amounts of data, and cannot achieve true adaptive fusion learning under data imbalance conditions.
[0006] In summary, existing technologies for multimodal fault diagnosis suffer from limitations in robustness, generalization ability, and sensitivity to critical but sparse fault symptoms due to their inability to effectively overcome model bias caused by data imbalance between modes. Therefore, there is an urgent need for an intelligent fault diagnosis method that can dynamically sense and adaptively compensate for modal data imbalance, guiding the model to fairly and fully utilize all modal information, thereby achieving more reliable and accurate equipment health status assessment and fault early warning in complex industrial environments. Summary of the Invention
[0007] The purpose of this invention is to address the problem in the prior art that the robustness, generalization ability, and sensitivity to key but sparse fault symptoms of multimodal fault diagnosis are limited due to the failure to effectively overcome the model bias caused by the imbalance of data between modes. This invention proposes an adaptive fault diagnosis method for multimodal data imbalance.
[0008] The technical solution of this invention: an adaptive fault diagnosis method for multimodal data imbalance, comprising the following steps:
[0009] S1. Synchronously acquire multimodal data from industrial equipment, and perform time alignment and standardization processing on the multimodal data to obtain a standardized signal;
[0010] S2. Different time series image encoding methods are used to convert the standardized signals of different modes to generate corresponding single-channel images. All single-channel images are then stitched together along the channel dimension to form a multi-channel fused image.
[0011] S3. Use a convolutional neural network to extract features from the multi-channel fused image to obtain a feature vector representing the joint state of the devices;
[0012] S4. The feature vector is input into the agent in the deep reinforcement learning framework, and the agent outputs a fault diagnosis action. The environment module calculates the basic reward based on the fault diagnosis action and the real fault label, and calculates the adaptive balance reward by combining the prior weights of each modality data and the real-time modality utilization. The total reward is then fed back to the agent.
[0013] S5. Using the total reward, update the policy network parameters of the agent through the policy gradient algorithm to form a feedback learning loop, and after training, use the trained policy network to diagnose faults in the newly input multimodal data.
[0014] Optionally, in step S1, the multimodal data includes at least vibration signals, temperature signals, and current signals;
[0015] The time alignment is based on the time series of the mode signal with the highest sampling rate, and other low-frequency mode signals are interpolated and resampled.
[0016] The standardization is a modal-independent Min-Max normalization that maps signal values to the [-1, 1] interval.
[0017] Optionally, the different time-series image encoding methods are specifically as follows:
[0018] The vibration signal is encoded using Gram angle and field (GASF) to generate the first single-channel image;
[0019] The temperature signal is encoded using Gram Angular Difference Field (GADF) to generate a second single-channel image;
[0020] The current signal is encoded using Markov Transfer Field (MTF) to generate a third single-channel image;
[0021] The first single-channel image, the second single-channel image, and the third single-channel image are stitched together to form a three-channel RGB fused image.
[0022] Optionally, in step S3, the convolutional neural network includes a multi-scale convolution module and a channel attention module;
[0023] The multi-scale convolution module is used to extract spatial features at different scales in parallel.
[0024] The channel attention module is used to adaptively weight the importance of each channel feature.
[0025] Optionally, in step S4, the total reward The calculation formula is:
[0026]
[0027] in, The false positive rate at time t, which can be either 0 or 1; This is the balance coefficient; For the first The prior weights of each modality are calculated based on the reciprocal of the number of training samples for each modality; For the first The real-time modality utilization of each modality is calculated based on the proportion of the gradient magnitude of the feature of that modality in the classification loss.
[0028] Optionally, the prior weights The calculation formula is:
[0029]
[0030] in, For the first The number of training samples for each modality. is the number of valid training samples for the k-th modality.
[0031] Optionally, the degree of real-time modal utilization Its function is to quantify the actual dependence of the model on the information of the m-th modality in the current decision, and its calculation formula is:
[0032]
[0033] in, For the first Feature vectors of each modality Let cross-entropy be the loss function. Indicates loss right gradient, Describing the L2 norm, This represents the total number of modes.
[0034] Optionally, in step S5, the policy gradient algorithm is the proximal policy optimization (PPO) algorithm; the feedback learning loop updates the policy network parameters using the trajectory generated by the interaction between the agent and the environment and the advantage function calculated by generalized advantage estimation (GAE).
[0035] Optionally, in step S5, the sign that the training is complete is that the diagnostic accuracy of the validation set has not improved for a consecutive preset number of rounds, or the maximum number of training rounds has been reached.
[0036] In the diagnosis phase, the real-time multimodal data is processed through steps S1 to S3 to obtain feature vectors, which are then input into the trained policy network, and the fault category with the highest probability is output as the diagnosis result.
[0037] Compared with the prior art, this application includes at least one of the following beneficial technical effects:
[0038] By designing a reward function that integrates prior weights and real-time modality utilization, this mechanism can dynamically perceive the model's dependence on each modality's data during training. When the system ignores low-data-volume modalities, the reward function automatically provides adjustment signals, driving the agent's strategy to optimize towards a more balanced utilization of multimodal information, thereby directly compensating for data volume deviations caused by differences in sampling frequency at the algorithm level.
[0039] The method of this invention forces the model to enhance its learning and utilization of features in modes with low data volume (such as temperature and current). This enables the system to make effective diagnoses based on key but sparse signs such as temperature trends or current state transitions, even when vibration signals are not obvious or are disturbed, reducing the risk of missed or false diagnoses due to single-mode failure or insufficient information.
[0040] Traditional modality-weighted fusion relies on expert experience to pre-set weights or statically adjust weights through grid search. This invention, through the trial-and-error mechanism and adaptive rewards of reinforcement learning, achieves online, dynamic, and automated adjustment of fusion weights, reducing subjective intervention and improving the method's adaptability and deployability.
[0041] By forcing the model to learn all modal information fairly, the system avoids overfitting to the specific features of a particular high-data-volume mode. This enables it to make decisions based on a more comprehensive feature representation when faced with operating condition changes not fully covered by the training data, sensor noise, or degraded quality of some modal signals, thus exhibiting stronger stability and generalization ability.
[0042] In summary, this invention introduces an adaptive reward mechanism based on prior weights and the utilization of real-time modalities, enabling the diagnostic system to dynamically balance the utilization of modalities with different data volumes, effectively overcoming information usage bias caused by differences in sampling frequency. This method automatically enhances the perception of fault symptoms in low-data-volume but critical modalities, reduces overfitting to high-data-volume modalities, and lowers the model's dependence on manually set weights. Ultimately, the system achieves fairer and more robust multimodal information fusion without human intervention, improving overall diagnostic accuracy and generalization performance under complex operating conditions. Attached Figure Description
[0043] Figure 1 This is the overall flowchart of a multimodal imbalance fault diagnosis system.
[0044] Figure 2 This is a flowchart of the DRL multimodal adaptive fusion decision layer.
[0045] Figure 3 This is a schematic diagram illustrating the fusion input generated from multimodal sensor signals using different time-series image encoding methods; where,
[0046] (a) is the image obtained by GASF encoding of the vibration signal, as the red (R) channel;
[0047] (b) is the image obtained by encoding the temperature signal using GADF, which is used as the green (G) channel;
[0048] (c) is the image obtained by encoding the current signal via MTF, which is used as the blue (B) channel;
[0049] (d) is an RGB fused image synthesized from the three channels of (a), (b), and (c) after independent normalization, and is used as the input of the convolutional neural network. Detailed Implementation
[0050] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.
[0051] Example
[0052] like Figure 1 , Figure 2 and Figure 3 As shown, the present invention proposes an adaptive fault diagnosis method for imbalanced multimodal data. The core components of the method include a multimodal data standardization layer, a temporal visualization layer, a feature extraction layer, a multimodal adaptive fusion decision layer, and a feedback learning loop.
[0053] The multimodal data collected in the multimodal data normalization layer comes from a synchronous monitoring system of the same industrial equipment. Vibration signals are acquired using piezoelectric accelerometers, temperature signals using Pt100 resistance temperature detectors (RTDs), and current signals using Hall effect current sensors for non-invasive acquisition. Data alignment is based on the time series data with the most data points in the multimodal signal, and other low-sampling-rate signals are resampled using linear interpolation methods to select a period of equipment operation. Then, a unified timeline is generated: the time point with the highest sampling rate (usually the vibration signal) is used as the base time series.
[0054]
[0055] in, This represents the timestamp of the i-th sampling moment. and These are the start and end times of a defined time window. The sampling time interval, Where N is the sampling rate of the vibration signal, and N is the total number of samples in a single acquisition. For low-frequency signals, linear interpolation or spline interpolation is used for interpolation, calculating the sampling rate at each frequency. The estimated time value is aligned with the vibration signal time. Then, it is processed using the Min-Max normalization method according to modal-independent standardization, linearly mapping it to... The interval is determined to satisfy the mathematical requirements of GASF / GADF. Specifically, for the... The original signal sequence of each mode Its standardized form is:
[0056]
[0057] in, Indicates the first The first mode Each sample value, Indicates the first The first mode A standardized signal value.
[0058]
[0059]
[0060] in, Indicates the first The maximum value of each modal signal. Indicates the first The minimum value of each modal signal.
[0061] The temporal visualization layer uses GASF, GADF, and MTF methods to encode vibration, temperature, and current signals, respectively.
[0062] Vibration signal Angles calculated after normalization Generate GASF image ;in Let i be the i-th normalized value of the vibration signal. Indicates the vibration signal number 1 Phase angle at each time point Indicates the location of the GASF image. Pixel values;
[0063] Temperature signal Similarly, after normalization, a GADF image is generated. ,in Indicates the temperature signal number 1 Phase angle at each time point Indicates the location of the GADF image. Pixel values;
[0064] Current signal After normalization, the data is divided into several state intervals. The transition frequencies are statistically analyzed and normalized into a probability matrix. An MTF image is obtained through interpolation. These three data points are used as the red (R), green (G), and blue (B) channels, respectively. After independent normalization, they are synthesized into a 64×64×3 RGB input image. (See...) Figure 3 .
[0065] The convolutional neural network with a feature extraction layer includes a multi-scale feature extraction module and a channel attention module.
[0066] Specifically, the input 64×64×3 fused image is first processed through three parallel branches: Branch 1: 32 3×3 convolutional kernels; Branch 2: 32 5×5 convolutional kernels; Branch 3: 32 3×3 convolutional kernels with a dilation rate of 2;
[0067] The three outputs are concatenated to obtain a 96-channel feature map. This feature map is then input into the SE attention module: first, global average pooling is performed to obtain a 96-dimensional vector; then, channel weights are generated through a process of "FC→ReLU→FC→Sigmoid," and multiplied channel-by-channel with the original feature. Finally, a state vector is output through global average pooling and a 256-dimensional fully connected layer. .
[0068] In the multimodal adaptive fusion decision layer, the environment is based on the actions output by the agent. And the misjudgment rate calculated from the actual fault labels ,Right now:
[0069]
[0070] in, For intelligent agents in The action output at all times, For a true fault label, when the agent outputs an action that matches the true fault label, When the action differs from the label, The prior weights are determined in advance based on the number of training samples for each modality, i.e.:
[0071]
[0072] in For the first The number of training samples for each modality. This represents the number of effective training samples for the k-th modality; simultaneously, the modality utilization is calculated based on the gradient magnitude of the classification loss with respect to the features of each modality. This is to quantify the actual dependence of the model on each modal information at the current moment.
[0073]
[0074] in, For the first Feature vectors of each modality Let cross-entropy be the loss function. Indicates loss right gradient, Describing the L2 norm, This represents the total number of modes.
[0075] in, : No. Modality utilization of each mode (values ∈ [0, 1]); : No. The feature vector obtained after feature extraction of each modality; The classification loss function is cross-entropy loss; :loss For modes Features The gradient; : L2 norm; M : total number of modes.
[0076] Ultimately, the environment determines the reward function based on the aforementioned reward function. A reward signal is generated and fed back to the agent. This reward function retains the misclassification rate as a basic accuracy indicator, and introduces an adaptive balance term composed of prior weights and the degree of utilization of real-time modalities. This automatically incentivizes the agent to increase its attention to low-frequency modal information when it is ignored, thereby achieving fair learning under imbalanced data conditions.
[0077] The policy gradient algorithm in the feedback learning loop employs the proximal policy optimization (PPO) algorithm. During training, the agent interacts with the environment to generate trajectories. ,in For the first Multimodal fusion state of the step. The actions output by the intelligent agent. As a comprehensive reward for environmental feedback, The maximum number of interaction steps for a single diagnostic task is determined, and the advantage function is calculated using generalized advantage estimation (GAE). The policy network is updated by maximizing the following objective function:
[0078]
[0079] in, This represents the objective function for PPO strategy optimization. This represents the expectation at the t-th time step. Let be the probability density of the current policy. The probability density of the old strategy. This is the advantage estimate. clip(x,a,b) is the clipping function, defined as:
[0080]
[0081] Training continues until the fault diagnosis accuracy on the validation set fails to improve for five consecutive rounds, or the maximum number of training rounds (5000) is reached. After training, in the diagnosis phase, the real-time multimodal data is processed through the aforementioned steps to obtain the state. The input strategy network is used to select the category corresponding to the maximum value in the output probability distribution as the final fault diagnosis result.
[0082] This invention constructs a closed-loop dynamic learning system to address the problem of imbalanced multimodal data. Its core lies in designing an adaptive reward function that integrates diagnostic accuracy and modality utilization efficiency, and embedding it within a deep reinforcement learning framework to drive the agent in policy optimization. Specifically, the system first converts asynchronously acquired multimodal time-series signals such as vibration, temperature, and current into Gram angle field images or Markov transfer field images based on their physical characteristics, thereby preserving and highlighting their respective dynamic modes in the visual domain. Subsequently, a convolutional neural network extracts joint feature representations from the synthesized multi-channel images as environmental information describing the current state of the device.
[0083] In a deep reinforcement learning loop, the agent outputs a diagnostic action based on its current state. When calculating the reward, the environment doesn't solely rely on the correctness of the action's classification; instead, it introduces an adaptive balancing term consisting of two parts: one part is a prior weight calculated inversely proportional to the number of training samples for each modality, assigning higher potential value to modalities with lower data volumes at the start of training; the other part is the modality utilization level calculated in real-time based on the gradient magnitude of the classification loss on the feature vectors of each modality, quantifying the proportion of modality information actually relied upon by the current decision strategy. The final reward signal is a weighted sum of the base accuracy reward and this balancing term. This design allows the reward function to dynamically reflect whether the agent has fairly utilized all modal information.
[0084] From a learning mechanism perspective, the agent's goal is to maximize long-term cumulative rewards. Therefore, when the model over-relies on a high-sample-volume mode due to differences in data volume, the combination of high prior weights and low real-time utilization of neglected low-sample-volume modes will produce a negative balancing term, thereby reducing the overall reward and incentivizing the agent to adjust its policy to more fully explore and utilize this information. The entire system iterates continuously through the policy gradient algorithm, ultimately driving the policy network to spontaneously balance the contributions of each modality's information while making accurate diagnoses. This invention significantly improves the model's ability to perceive and utilize sparsely sampled but potentially critical fault symptoms without requiring manual setting of modality weights, thus enhancing the robustness and generalization of the diagnostic system under varying operating conditions while maintaining or improving overall diagnostic accuracy.
[0085] The above specific embodiments are merely several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.
Claims
1. An adaptive fault diagnosis method for multimodal data imbalance, characterized in that, Includes the following steps: S1. Synchronously acquire multimodal data from industrial equipment, and perform time alignment and standardization processing on the multimodal data to obtain a standardized signal; S2. Different time series image encoding methods are used to convert the standardized signals of different modes to generate corresponding single-channel images. All single-channel images are then stitched together along the channel dimension to form a multi-channel fused image. S3. Use a convolutional neural network to extract features from the multi-channel fused image to obtain a feature vector representing the joint state of the devices; S4. Input the feature vector into the agent in the deep reinforcement learning framework, and the agent outputs a fault diagnosis action; The environment module calculates the basic reward based on the fault diagnosis action and the real fault label, and calculates the adaptive balance reward by combining the prior weights of each modality data and the real-time modality utilization, and generates a total reward that is fed back to the agent. S5. Using the total reward, update the policy network parameters of the agent through the policy gradient algorithm to form a feedback learning loop, and after training, use the trained policy network to diagnose faults in the newly input multimodal data.
2. The adaptive fault diagnosis method for multimodal data imbalance according to claim 1, characterized in that, In step S1, the multimodal data includes at least vibration signals, temperature signals, and current signals; The time alignment is based on the time series of the mode signal with the highest sampling rate, and other low-frequency mode signals are interpolated and resampled. The standardization is a modal-independent Min-Max normalization that maps signal values to the [-1, 1] interval.
3. The adaptive fault diagnosis method for multimodal data imbalance according to claim 2, characterized in that, The different time-series image encoding methods are as follows: The vibration signal is encoded using Gram angle and field coding to generate the first single-channel image. Gram difference field coding is used on the temperature signal to generate a second single-channel image; Markov transfer field coding is applied to the current signal to generate a third single-channel image; The first single-channel image, the second single-channel image, and the third single-channel image are stitched together to form a three-channel RGB fused image.
4. The adaptive fault diagnosis method for multimodal data imbalance according to claim 1, characterized in that, In step S3, the convolutional neural network includes a multi-scale convolution module and a channel attention module; The multi-scale convolution module is used to extract spatial features at different scales in parallel. The channel attention module is used to adaptively weight the importance of each channel feature.
5. The adaptive fault diagnosis method for multimodal data imbalance according to claim 1, characterized in that, In step S4, the total reward The calculation formula is: ； in, The false positive rate at time t, which can be either 0 or 1; This is the balance coefficient; For the first The prior weights of each modality are calculated based on the reciprocal of the number of training samples for each modality; For the first The real-time modality utilization of each modality is calculated based on the proportion of the gradient magnitude of the feature of that modality in the classification loss.
6. The adaptive fault diagnosis method for multimodal data imbalance according to claim 5, characterized in that, The prior weights The calculation formula is: ； in, For the first The number of training samples for each modality. For the first The number of effective training samples for each modality.
7. The adaptive fault diagnosis method for multimodal data imbalance according to claim 5, characterized in that, The degree of real-time modal utilization Used to quantify the actual dependence of the model on the information of the m-th modality in the current decision, and is a scalar. The larger the value, the more the model currently relies on that mode for decision-making. Its calculation formula is: ； in, For the first Feature vectors of each modality Let cross-entropy be the loss function. Indicates loss right gradient, Describing the L2 norm, This represents the total number of modes.
8. The adaptive fault diagnosis method for multimodal data imbalance according to claim 1, characterized in that, In step S5, the policy gradient algorithm is a proximal policy optimization algorithm; the feedback learning loop updates the policy network parameters using the trajectory generated by the interaction between the agent and the environment and the advantage function calculated by the generalized advantage estimation.
9. The adaptive fault diagnosis method for multimodal data imbalance according to claim 1, characterized in that, In step S5, the sign that the training is complete is that the diagnostic accuracy of the validation set has not improved for a preset number of consecutive training rounds, or the maximum number of training rounds has been reached. In the diagnosis phase, the real-time multimodal data is processed through steps S1 to S3 to obtain feature vectors, which are then input into the trained policy network, and the fault category with the highest probability is output as the diagnosis result.