Equipment fault diagnosis method and device, equipment and storage medium

Abstract

The invention relates to an equipment fault diagnosis method and device, equipment and a storage medium, and the method comprises the steps: obtaining multi-modal data of a unified timestamp of the equipment; the multi-modal data with the unified timestamp is preprocessed; extracting fault features from the preprocessed multi-modal data, and generating a joint feature vector of a predetermined number of dimensions and a unified timestamp; multi-modal feature fusion is carried out on the joint feature vectors of the unified timestamps of the preset number of dimensions, dynamic adjustment of modal weights is carried out through an attention mechanism algorithm, and fused joint feature vectors of the unified timestamps are generated; and performing fault reasoning by improving a ResNet model according to the fused joint feature vector of the unified timestamp to generate a fault type and the confidence coefficient of the fault type. According to the method, the limitation of single-mode data acquisition is broken through, three-dimensional coverage of fault features is realized through multi-mode synchronous acquisition, and the accuracy of fault diagnosis and the adaptability to complex scenes are improved.

Description

Technical Field

[0001] This application relates to the field of equipment fault diagnosis, and in particular to a method, apparatus, device and storage medium for diagnosing equipment faults. Background Technology

[0002] Currently, existing technologies use single-modal data for refrigerator fault diagnosis, or integrate multimodal data through traditional methods such as simple splicing and weighted averaging, and then combine traditional machine learning or simple CNN models to achieve refrigerator fault classification.

[0003] The following problems exist in the existing technology: The reliance on single-modal data leads to incomplete coverage of fault features, making it easy to miss early latent faults. Summary of the Invention

[0004] This application provides a network distribution method, device, terminal, electronic device, and storage medium to solve all or part of the problems mentioned above in the prior art.

[0005] In a first aspect, this application provides a method for diagnosing equipment malfunctions, including: Acquire multimodal data with a unified timestamp from the device; The multimodal data with the unified timestamp is preprocessed; Fault features are extracted from preprocessed multimodal data to generate a joint feature vector with a predetermined number of dimensions of unified timestamps; Multimodal feature fusion is performed on the joint feature vector of the unified timestamps of the predetermined number of dimensions, and the modal weights are dynamically adjusted through an attention mechanism algorithm to generate the fused joint feature vector of the unified timestamps. Based on the joint feature vector fused from the unified timestamp, fault reasoning is performed by improving the ResNet model to generate fault types and their confidence levels.

[0006] The method further includes: Based on the fault type and the confidence level of the fault type, a graded processing method is adopted.

[0007] Before the step of acquiring the unified timestamp of the device's multimodal data, the method further includes: Collect aggregated fault data to train an improved ResNet model, thereby training the attention weight calculation algorithm and ResNet parameters of the improved ResNet model.

[0008] When the device is a refrigerator, the multimodal data includes one or more of the following data in any combination: The refrigerator's compressor operating vibration signal; the refrigerator's energy consumption data; the refrigerator's global temperature field data.

[0009] The preprocessing steps for the multimodal data include: The preprocessing of the operational vibration signal specifically includes: performing a predetermined layer decomposition on the operational vibration signal using a wavelet basis to remove a predetermined power frequency interference; extracting the time-domain and frequency-domain features of the operational vibration signal; calculating the feature mean of a predetermined number of time windows through a sliding window of predetermined duration; and / or The preprocessing of the energy consumption data specifically includes: eliminating grid fluctuation interference through a predetermined sliding window filter; extracting the statistical features of the energy consumption data and calculating the trend features of the energy consumption data by combining it with a predetermined group of historical data for a predetermined duration; The preprocessing of the global temperature field data is as follows: using the K-means algorithm, a predetermined number of pixels are divided into different temperature zones, the statistical features of the temperature in each temperature zone are extracted, and combined with a predetermined group of historical data for a predetermined time period, the trend features of the global temperature field data are calculated. The preprocessed multimodal data is used to form a multimodal joint feature vector for a single sample, and the timestamp of the acquisition is marked.

[0010] The step of performing multimodal feature fusion on the joint feature vector of the predetermined number of unified timestamps to generate a fused joint feature vector with unified timestamps includes: Feature projection and positional encoding are performed, specifically by mapping the joint feature vector into a query Q, key K, and value V matrix through three learnable linear layers, and adding sinusoidal positional encoding to inject modal sequence information. Self-attention weight calculation specifically involves calculating the association weights of each modality by scaling the dot product attention. Dynamic adjustment of modality weights: Specifically, weights are automatically assigned based on feature similarity; The weighted features are nonlinearly transformed through a two-layer feedforward neural network, and the fused joint feature vector is output.

[0011] The steps of performing fault reasoning and generating fault types and their confidence levels based on the fused joint feature vector using the unified timestamp and by improving the ResNet model include: Feature adaptation and extraction are performed as follows: the first layer uses a one-dimensional convolutional kernel to adapt to the fused joint feature vector; convolution operation is performed on the input features to output the feature map; a Dropout layer is inserted after each residual block in the intermediate stage. Global average pooling is performed by converting the feature map into a vector of a predetermined number of dimensions, inputting it into a fully connected layer, activating it through the Softmax function, and outputting the confidence probability distribution of a predetermined number of faults. Based on the maximum confidence value in the confidence probability distribution and the corresponding fault type characteristics, the fault type and the confidence level of the fault type are determined.

[0012] On the other hand, the present invention provides a diagnostic apparatus for equipment malfunctions, comprising: The acquisition unit is used to acquire multimodal data with a unified timestamp from the device. The preprocessing unit is used to preprocess the multimodal data with the unified timestamp. The extraction unit is used to extract fault features from preprocessed multimodal data and generate a joint feature vector with a predetermined number of dimensions of unified timestamps; The fusion unit is used to perform multimodal feature fusion on the joint feature vector of the unified timestamp of the predetermined number of dimensions, and dynamically adjust the modal weights through an attention mechanism algorithm to generate the fused joint feature vector of the unified timestamp. The generation unit is used to perform fault reasoning based on the fused joint feature vector of the unified timestamp, by improving the ResNet model, and to generate fault types and confidence scores for the fault types.

[0013] The present invention also provides an electronic device, comprising: a processor and a memory, wherein the processor is configured to execute a network distribution method stored in the memory to implement the method.

[0014] The present invention also provides a storage medium storing one or more programs, which can be executed by one or more processors to implement the method.

[0015] Compared with the prior art, the technical solution provided in this application has the following advantages, and the present invention has the following beneficial effects: It acquires multimodal data of the device with a unified timestamp; preprocesses the multimodal data with the unified timestamp; extracts fault features from the preprocessed multimodal data to generate a joint feature vector of unified timestamps with a predetermined number of dimensions; performs multimodal feature fusion on the joint feature vector of unified timestamps with the predetermined number of dimensions, dynamically adjusts the modal weights through an attention mechanism algorithm, and generates a fused joint feature vector of unified timestamps; based on the fused joint feature vector of unified timestamps, it performs fault inference by improving the ResNet model, generating fault types and their confidence levels. In this invention, the limitation of "single-modal data acquisition" in the prior art is overcome, and multimodal synchronous acquisition achieves three-dimensional coverage of fault features, improving the accuracy of fault diagnosis and adaptability to complex scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 A schematic flowchart illustrating the equipment fault diagnosis method provided in this application embodiment; Figure 2 A flowchart illustrating the equipment fault diagnosis method provided for the application scenario of this application; Figure 3 A schematic diagram of the structure of a device for diagnosing equipment faults provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0022] like Figure 1As shown, this invention provides a method for diagnosing equipment faults, comprising: Step 11: Obtain multimodal data of the device with a unified timestamp; when the device is a refrigerator, the multimodal data includes one or more of the following data in any combination: the operating vibration signal of the refrigerator's compressor; the energy consumption data of the refrigerator; and the global temperature field data of the refrigerator.

[0023] Pre-configuration: 1) Fix the triaxial accelerometer (range ±2g, sampling frequency 1kHz) to the top of the compressor housing using a shockproof bracket to collect the compressor's vibration signal during operation; 2) A high-precision energy metering chip (error ±2%) is connected in series to the refrigerator power supply circuit to monitor energy consumption data in real time; a 64×64 pixel infrared thermal imaging sensor is installed on the top of the refrigerator liner, with the lens covering the refrigerator compartment, freezer compartment and condenser area to capture the temperature distribution of the entire area.

[0024] Step 12: Preprocess the multimodal data with the unified timestamp; Step 13: Extract fault features from the preprocessed multimodal data and generate a joint feature vector with a predetermined number of dimensions of unified timestamps; Step 14: Perform multimodal feature fusion on the joint feature vector of the unified timestamps of the predetermined number of dimensions, and dynamically adjust the modal weights through an attention mechanism algorithm to generate the fused joint feature vector of the unified timestamps. Step 15: Based on the fused joint feature vector of the unified timestamp, perform fault reasoning by improving the ResNet model to generate fault type and fault type confidence.

[0025] This invention breaks through the limitations of the existing technology of "single-modal data acquisition" and achieves three-dimensional coverage of fault characteristics through multi-modal synchronous acquisition, thereby improving the accuracy of fault diagnosis and adaptability to complex scenarios.

[0026] The method further includes: Step 16: Based on the fault type and the confidence level of the fault type, perform hierarchical processing.

[0027] This step can be: performing differentiated alarm operations based on the fault level to ensure accurate information delivery. Specific steps include: (1) Early warning of potential faults (such as early refrigerant leakage, confidence level 68%): Local response: The yellow LED light on the control panel flashes once every 2 seconds without a buzzer (to avoid disturbing the user); (2) Data storage: The fault type ("minor refrigerant leak"), confidence level (0.68), acquisition time (accurate to the second) and the original data (vibration, energy consumption, temperature) 5 minutes before and after the fault are encrypted and stored in the storage space to retain the traceability basis; (3) Subsequent actions: shorten the data collection cycle to 1 minute, continuously monitor the fault development trend, and automatically upgrade to a clear fault alarm if the confidence level rises to 80% or more.

[0028] (4) Clearly identify fault alarms (e.g., compressor jamming, confidence level 0.94): Local response: The red LED light on the control panel remains constantly lit, and the buzzer sounds an alarm continuously at 1-second intervals (volume 65dB) until the user manually presses the "Alarm Cancel" button; (5) Remote push: The alarm information is pushed to the cloud platform via the Wi-Fi module and the MQTT protocol, and then forwarded to the user's APP. The push content includes: [Refrigerator Fault Warning] Fault type: Compressor mechanical fault; Confidence level: 84%; Occurrence time: 2025-12-15 14:32:18; Key data: Root mean square vibration 0.98g, power increased by 42% compared with normal. (6) After-sales linkage: The cloud platform will push the fault information to the brand's after-sales system, marking the user's address and refrigerator model, so that after-sales personnel can take the initiative to contact and repair the refrigerator.

[0029] (7) Handling of combined faults (such as condenser dust accumulation + refrigerant leakage, with confidence levels of 0.85 and 0.72 respectively): Alarm information: The APP push notification is marked "Main fault: Reduced condenser heat dissipation efficiency (85%), Related fault: Minor refrigerant leakage (72%)", prompting after-sales personnel to check both types of problems at the same time; (8) Data support: Store characteristic data corresponding to two types of faults, such as condenser temperature 48℃ (normal ≤40℃) and freezer temperature -8℃ (set -18℃), to provide accurate basis for maintenance.

[0030] Optionally, prior to step 11, the method further includes: Step 10: Collect aggregated fault data to train an improved ResNet model, thereby training the attention weight calculation algorithm and ResNet parameters of the improved ResNet model.

[0031] The model training process is as follows: Training environment: PC based on PyTorch framework, using Adam optimizer, and cross-entropy loss function.

[0032] Training parameters: batch size 32, training 100 epochs, training stops when the validation set accuracy shows no improvement for 10 consecutive epochs; Model deployment: The model is quantized into INT8 format and ported to the microcontroller to improve inference speed and meet real-time diagnostic requirements.

[0033] Step 12 includes: Step 121 involves preprocessing the operational vibration signal, specifically: decomposing the operational vibration signal into a predetermined layer using a wavelet basis to remove predetermined power frequency interference; extracting the time-domain and frequency-domain features of the operational vibration signal; calculating the feature mean of a predetermined number of time windows through a sliding window of predetermined duration; and / or Step 122, the preprocessing of the energy consumption data, specifically includes: eliminating grid fluctuation interference through a predetermined sliding window filter; extracting the statistical features of the energy consumption data, and calculating the trend features of the energy consumption data by combining it with a predetermined group of historical data for a predetermined duration; Step 123, the preprocessing of the global temperature field data, specifically: using the K-means algorithm, dividing a predetermined number of pixels into different temperature zones, extracting the statistical features of the temperature in each temperature zone, and combining them with a predetermined group of historical data for a predetermined duration to calculate the trend features of the global temperature field data. Step 124: The preprocessed multimodal data is used to form a multimodal joint feature vector of a single sample, and the acquisition timestamp is marked.

[0034] Step 13 includes: Step 131 involves feature projection and position encoding, specifically: mapping the joint feature vector into a query Q, key K, and value V matrix through three learnable linear layers, adding sinusoidal position encoding, and injecting modal sequence information; Step 132, self-attention weight calculation, specifically: calculate the association weights of each modality data by scaling the dot product attention; Step 133, Dynamic adjustment of modal weights: Specifically, weights are automatically assigned based on feature similarity; Step 134: The weight features are nonlinearly transformed through a two-layer feedforward neural network to output the fused joint feature vector.

[0035] Furthermore, this invention introduces an attention module to dynamically calculate the weights of each modality's data under different fault scenarios, and then achieves accurate fusion of multimodal features through weighted fusion, replacing the traditional simple splicing or fixed weighting method. Simultaneously, the ResNet model output layer is improved by combining fault type and confidence level to help distinguish between single and compound faults. Therefore, this invention breaks away from the crude mode of existing technologies such as "fixed weight fusion" and "indiscriminate feature splicing," and achieves adaptive adjustment of modal weights based on fault scenarios through an attention mechanism. This allows key modal information (such as vibration signals in compressor faults and temperature data in leakage faults) to dominate, avoiding the dilution of key features by redundant information. At the same time, by improving the fault confidence level output of the ResNet model and combining it with the detailed differences in multimodal weighted features, this invention solves the pain point of traditional solutions being unable to distinguish compound or similar faults, reducing fault location error rate and improving the accuracy of compound fault identification.

[0036] Optionally, step 5 includes: Step 51, feature adaptation and extraction are performed, specifically: the first layer uses a one-dimensional convolution kernel to adapt to the fused joint feature vector; convolution operation is performed on the input features to output the feature map; a Dropout layer is inserted after each residual block in the intermediate stage; Step 52, perform global average pooling, specifically: transform the feature map into a vector of a predetermined number of dimensions, input it into the fully connected layer, activate it through the Softmax function, and output the confidence probability distribution of a predetermined number of faults; Step 53: Determine the fault type and the confidence level of the fault type based on the maximum confidence level in the confidence probability distribution and the corresponding fault type characteristics.

[0037] like Figure 2 As shown, the application scenarios of the present invention are described below. 1. Environmental preparation Deployment of core sensors.

[0038] 1) Fix the triaxial accelerometer (range ±2g, sampling frequency 1kHz) to the top of the compressor housing using a shockproof bracket to collect the compressor's vibration signal during operation; 2) A high-precision energy metering chip (error ±2%) is connected in series to the refrigerator power supply circuit to monitor energy consumption data in real time; a 64×64 pixel infrared thermal imaging sensor is installed on the top of the refrigerator liner, with the lens covering the refrigerator compartment, freezer compartment and condenser area to capture the temperature distribution of the entire area.

[0039] 3) Integration of main control and communication modules: The STM32H743 microcontroller with a main frequency of 480MHz is selected as the main control unit. It connects to each sensor through the I2C interface and communicates with the cloud platform through the Wi-Fi module (supporting 802.11b / g / n protocol). It is equipped with storage space as a local storage medium for caching data and fault records.

[0040] 4) Alarm component configuration: A 65dB buzzer and red / yellow dual-color LED lights are embedded in the refrigerator control panel to correspond to the local alarm requirements of explicit faults and potential faults, respectively.

[0041] 5) Local preprocessing program development: Based on MicroPython, write sensor data acquisition scripts to achieve synchronous triggering and timestamp alignment of vibration, energy consumption, and temperature field data, with synchronization error controlled within ±10ms.

[0042] 6) Model deployment environment configuration: Set up a PyTorch training environment on the PC to complete model training. After training, use the TorchQuantization tool to quantize the model into INT8 format, export the model as an onnx file that the microcontroller can recognize, and port it to the flash storage area of ​​the microcontroller.

[0043] 7) Cloud platform and APP integration: A simple cloud platform is built based on the MQTT protocol to complete the communication link debugging between the microcontroller and the cloud platform and the user APP, ensuring that alarm information can be pushed in real time.

[0044] 2. Multimodal data acquisition and preprocessing This step is the "data source" for fault diagnosis. By simultaneously collecting three types of core data and performing targeted preprocessing, effective fault features are extracted to provide high-quality input for model inference, thus solving the problem of incomplete feature coverage of single-modal data in existing technologies.

[0045] 2.1 Synchronous Acquisition of Multimodal Data - Vibration signal acquisition: The compressor vibration data is continuously acquired at a frequency of 1000Hz, and a sample of 10 seconds (containing 10,000 data points) is generated every 3 minutes, covering the compressor start-up, shutdown and stable operation stages; - Energy consumption data acquisition: The power metering chip records the average power and cumulative power consumption data every 3 minutes, which is precisely aligned with the vibration signal acquisition timestamp; - Infrared temperature field acquisition: The infrared sensor acquires a temperature image every 3 minutes and outputs the temperature value of 64x64 pixels (range -20℃~60℃), and simultaneously records the refrigerator's set temperature at the time of acquisition.

[0046] Key logic: Synchronize and correlate the three types of data through a unified timestamp to ensure that the fault chain characteristics of "abnormal vibration - increased energy consumption - temperature deviation" can be analyzed later. For example, when the compressor is stuck, the three types of data will show abnormalities at the same time, rather than a single data fluctuation.

[0047] 2.2 Data Preprocessing and Feature Extraction To address the noise characteristics of data from different modalities, a differentiated preprocessing method was employed, ultimately extracting a 224-dimensional joint feature vector, as detailed below: - Vibration signal preprocessing (outputting 80-dimensional features): The vibration data is decomposed into 5 levels using the db4 wavelet basis to remove 50Hz power frequency interference; then time-domain features (e.g., peak value, root mean square, kurtosis, etc.) and frequency-domain features (e.g., center frequency, spectral entropy, etc.) are extracted, and the feature mean of 7 time windows is calculated through a sliding window (window size 1s), for a total of 10×8=80 dimensions; - Energy consumption data preprocessing (output 64-dimensional features): Eliminate grid fluctuation interference through third-order sliding window filtering; extract statistical features (e.g., including mean, variance, maximum and minimum values, etc., 8 items), and combine them with 19 sets of historical data from the past hour to calculate trend features (slope, fluctuation amplitude, etc., 8 items), for a total of 8×8=64 dimensions; - Temperature field data preprocessing (output 80-dimensional features): The K-means algorithm is used to divide the 64x64 pixels into three temperature zones: refrigerator, freezer and condenser. Statistical features of each temperature zone are extracted (e.g., 10 features such as average temperature, temperature extremes, uniformity (standard deviation) etc.). Trend features are calculated by combining 11 sets of historical data from the past hour, for a total of 10×8=80 dimensions. - Feature fusion: The 224-dimensional features (80+64+80=224) are concatenated in the order of "vibration-energy consumption-temperature" to form a multimodal joint feature vector of a single sample. After being marked with the collection timestamp, the vector is temporarily stored in the storage space.

[0048] This step transforms "raw data into effective features" through denoising and feature extraction, removing environmental interference (such as power grid fluctuations and external vibrations) while retaining core fault-related information; the 224-dimensional feature vector covers multiple dimensions of the state while controlling the feature size to adapt to the input requirements of subsequent models.

[0049] 3. Multimodal feature fusion and fault reasoning The modality weights are dynamically adjusted through the attention mechanism module, and then the fault classification is completed through the improved ResNet module. This solves the problem of key features being diluted due to traditional fixed-weighted fusion, and finally outputs the confidence scores of 8 types of faults.

[0050] 3.1 Dynamically Weighted Fusion of Attention Mechanism A four-layer Transformer module is used to perform weighted optimization on the 224-dimensional feature vector through a self-attention mechanism, outputting a 128-dimensional deep fusion feature. The specific process is as follows: (1) Feature projection and position encoding: Through three learnable linear layers, the 224-dimensional features are mapped into query (Q), key (K), and value (V) matrices (each with a dimension of 128×3, where “3” represents the three modes of vibration, energy consumption, and temperature), and sinusoidal position encoding is added to inject modal sequence information; (2) Self-attention weight calculation: Calculate the association weight of each modality by scaling the dot product attention.

[0051] (3) Dynamic adjustment of modal weights: Weights are automatically assigned based on feature similarity, for example: When the vibration characteristic shows "root mean square > 0.8g" and the energy consumption characteristic shows "power increase of 30%", it is determined to be a compressor-related fault. The vibration mode weight increases to 65%-70%, the energy consumption mode weight is 25%-30%, and the temperature mode weight is <5%. When the temperature characteristics show "average temperature of the freezer compartment > -10℃" and "temperature uniformity deviation > 8℃", it is judged as a refrigeration system failure. The weight of the temperature mode increases to 70%-75%, the weight of the energy consumption mode is 20%-25%, and the weight of the vibration mode is <5%. (4) Feature output: After passing through a 2-layer feedforward neural network, the weighted features are nonlinearly transformed to finally output 128-dimensional deep fusion features. These features highlight the key modal information related to the fault and weaken redundant interference.

[0052] 3.2 Improved Fault Classification Inference of ResNet Model The 128-dimensional fused features are input into the improved ResNet-50 model to complete the identification and confidence output of eight types of faults (compressor mechanical faults, refrigeration system refrigerant leakage faults, heat dissipation / air supply fan faults, temperature control module malfunctions, abnormal door sealing performance faults, sensor data drift / failure faults, poor electrical circuit contact faults, and condenser heat dissipation efficiency reduction faults). The specific process is as follows: (1) Feature adaptation and extraction: The first layer uses a 7×1 one-dimensional convolution kernel (adapted to 128-dimensional one-dimensional features) to perform convolution operation on the input features and output a 64-dimensional feature map; a Dropout layer (dropout probability 0.1) is inserted after each residual block in the intermediate stage to prevent overfitting; (2) Fault classification and confidence output: The feature map is transformed into a 1024-dimensional vector by global average pooling, and then activated by the Softmax function after being input into the fully connected layer. The output is the confidence probability distribution of 8 types of faults (the sum is 1). (3) Fault Judgment Example: Based on the maximum confidence level and corresponding characteristics, the fault type is accurately located. The specific judgment logic is as follows: For example, in the judgment of compressor mechanical fault (bearing wear), the triaxial accelerometer collects an abnormal signal with a vibration root mean square of 0.95g and a center frequency of 132Hz. The power metering chip simultaneously monitors that the operating power is 40.2% higher than normal, while the infrared temperature field data shows no obvious shift. The three types of features are preprocessed to form a 224-dimensional vector and then input into the system. The attention module determines that it is a compressor fault scenario and increases the weights of vibration and energy consumption modes to 68% and 27%, respectively. After the generated 128-dimensional fusion features are inferred by the improved ResNet model, the output compressor mechanical fault confidence score is 0.94, triggering the local red LED to stay on and the buzzer alarm, and pushing fault information containing key data to the APP.

[0053] In the diagnosis of refrigerant leakage faults in the refrigeration system, the process begins with the infrared sensor capturing an abnormal temperature of -7°C and a temperature deviation of 9.3°C in the freezer compartment. Energy consumption data shows that the power consumption variance is 197.7% higher than normal, while vibration data is normal. The attention module adjusts the temperature mode weight to 72% to highlight the core features. The model ultimately outputs a leakage fault confidence score of 0.91. Alarm information simultaneously marks the details of the abnormal temperature, providing a basis for accurate after-sales troubleshooting.

[0054] 4. Fault classification and alarm push notification This step is the "terminal response" of fault diagnosis. It implements graded alarms based on the confidence level of the model output, which not only avoids missed and false alarms, but also enables users and after-sales personnel to quickly obtain fault information and complete the "diagnosis-response" closed loop.

[0055] 4.1 Fault Classification Based on Confidence Two threshold levels are set (60%-70% for early warning and >80% for alarm). Based on the characteristic stability of eight types of faults, fault status determination is completed. Specific logic: - No faults (confidence level < 60%): The confidence level of all fault types is below the threshold, indicating that the refrigerator is operating normally; - Potential faults (60% ≤ confidence level ≤ 80%): If the confidence level of a certain type of fault falls within this range, it is determined to be an early latent fault (such as minor refrigerant leakage, initial wear of compressor bearings), triggering the early warning mechanism; - Clearly define the fault (confidence level > 80%): If the confidence level of a certain type of fault exceeds the threshold, it is determined that the fault has been manifested (such as compressor jamming, fan stoppage), and the emergency alarm mechanism is triggered; if the confidence level of both types of faults is > 70%, it is determined to be a compound fault (such as condenser dust accumulation + refrigerant leakage), and the fault with the highest confidence level is the primary fault, and the secondary fault information is output in association.

[0056] 4.2 Tiered Alarm and Information Push Perform differentiated alarm operations based on the fault level to ensure accurate information delivery. Specific steps include: (1) Early warning of potential faults (such as early refrigerant leakage, confidence level 68%): Local response: The yellow LED light on the control panel flashes once every 2 seconds without a buzzer (to avoid disturbing the user); (2) Data storage: The fault type ("minor refrigerant leak"), confidence level (0.68), acquisition time (accurate to the second) and the original data (vibration, energy consumption, temperature) 5 minutes before and after the fault are encrypted and stored in the storage space to retain the traceability basis; (3) Subsequent actions: shorten the data collection cycle to 1 minute, continuously monitor the fault development trend, and automatically upgrade to a clear fault alarm if the confidence level rises to 80% or more.

[0057] (4) Clearly identify fault alarms (e.g., compressor jamming, confidence level 0.94): Local response: The red LED light on the control panel remains constantly lit, and the buzzer sounds an alarm continuously at 1-second intervals (volume 65dB) until the user manually presses the "Alarm Cancel" button; (5) Remote push: The alarm information is pushed to the cloud platform via the Wi-Fi module and the MQTT protocol, and then forwarded to the user's APP. The push content includes: [Refrigerator Fault Warning] Fault type: Compressor mechanical fault; Confidence level: 84%; Occurrence time: 2025-12-15 14:32:18; Key data: Root mean square vibration 0.98g, power increased by 42% compared with normal. (6) After-sales linkage: The cloud platform will push the fault information to the brand's after-sales system, marking the user's address and refrigerator model, so that after-sales personnel can take the initiative to contact and repair the refrigerator.

[0058] (7) Handling of combined faults (such as condenser dust accumulation + refrigerant leakage, with confidence levels of 0.85 and 0.72 respectively): Alarm information: The APP push notification is marked "Main fault: Reduced condenser heat dissipation efficiency (85%), Related fault: Minor refrigerant leakage (72%)", prompting after-sales personnel to check both types of problems at the same time; (8) Data support: Store characteristic data corresponding to two types of faults, such as condenser temperature 48℃ (normal ≤40℃) and freezer temperature -8℃ (set -18℃), to provide accurate basis for maintenance.

[0059] 5. System operation assurance and iterative optimization 5.1 Routine Maintenance and Data Management The microcontroller automatically cleans up normal operating data older than 3 months from the storage space every month, while retaining all fault data; users can export fault reports via the APP for reference during after-sales maintenance.

[0060] 5.2 Model Iterative Update Each quarter, fault data (with user authorization) is collected from the cloud platform. The model is then retrained on the PC, the attention weight calculation logic and ResNet parameters are optimized, and the model file in the microcontroller is updated via OTA (Over-The-Air) to continuously improve the accuracy of fault identification.

[0061] The following describes another application scenario of the present invention: a refrigerator fault diagnosis system based on multimodal attention fusion.

[0062] This invention utilizes a deep learning model for fault detection in smart home appliances.

[0063] In existing technologies, reliance on single-modal data leads to incomplete coverage of fault features, making it easy to miss early latent faults. Furthermore, traditional machine learning models have weak generalization capabilities and low diagnostic accuracy in complex scenarios. Moreover, multimodal data fusion methods are crude and do not consider the correlation between fault scenarios and modal information, making key fault features easily interfered with and resulting in insufficient accuracy in fault location.

[0064] In this invention, temperature data not only reflects the refrigerator's cooling effect but also captures early fault characteristics from dimensions such as mechanical structure (e.g., compressor bearing wear) and electrical circuits (e.g., fan coil aging). Furthermore, it overcomes the problem of strong dependence on data distribution in existing technologies using the K-means algorithm. When the refrigerator's operating environment changes (e.g., high summer temperatures causing temperature baseline shifts), cluster boundaries easily become blurred, leading to early latent faults such as slight compressor jamming or slow refrigerant leakage going undetected, resulting in unstable and unreliable output results. Therefore, existing technologies struggle to meet the refrigerator's needs for early fault warning and stable diagnosis under complex environments.

[0065] Furthermore, this invention introduces a dynamic weight allocation mechanism to integrate multi-dimensional fault features, improving the comprehensiveness of fault diagnosis. It also considers the varying importance of different modal data under different fault scenarios; for example, vibration data is more valuable in indicative of compressor failure, while temperature data is more critical in refrigeration system leaks. This avoids the dilution of key modal features by redundant information due to simple data splicing, and allows for dynamic adaptation to changes in fault types. Moreover, it avoids the limitations of existing simple CNN models in extracting multimodal heterogeneous features, which are susceptible to interference from power grid fluctuations and environmental noise, leading to large fault location errors and an inability to accurately distinguish between single and compound faults. Therefore, it meets the demand for precise fault diagnosis in smart refrigerators.

[0066] This invention establishes a multimodal data acquisition module to simultaneously acquire refrigerator vibration signals (via a built-in triaxial accelerometer, sampling frequency 500-1000Hz), energy consumption data (via a high-precision smart meter, acquisition cycle 1-5 minutes), and infrared temperature field data (via an infrared thermal imaging sensor, sampling interval 2-10 minutes), achieving full coverage of mechanical, electrical, and refrigeration state data across multiple dimensions. Simultaneously, an improved ResNet model replaces the traditional K-means algorithm, enhancing the model's resistance to interference and generalization ability in complex environments. Therefore, this invention overcomes the limitations of existing "single-modal data acquisition" technologies, achieving comprehensive coverage of fault characteristics through multimodal synchronous acquisition, capturing early latent characteristics of mechanical structures (such as compressor bearing wear) and electrical circuits (such as fan coil aging) (e.g., abnormal root mean square vibration, changes in energy consumption variance). Furthermore, it eliminates the strong dependence of traditional clustering algorithms on data distribution, improving adaptability to complex scenarios. This enhances fault diagnosis accuracy and adaptability to complex scenarios.

[0067] This invention includes the following steps: I. Data Collection and Construction 1. Dataset type collected Simultaneously collect three types of datasets covering the multidimensional operating status of the refrigerator: Vibration signal: acquired via a triaxial accelerometer (±2g, 500-1000Hz), including operating condition data such as compressor normal operation and bearing wear, and extracting time-frequency domain features such as peak value and root mean square.

[0068] Energy consumption data: Data is collected every 3 minutes using a high-precision power metering chip (±0.2%), covering power changes under normal conditions and faults such as refrigerant leaks, and calculating statistical characteristics such as mean and variance.

[0069] Infrared temperature field: Using an 8×8 pixel infrared sensor, data is collected every 5 minutes to capture the normal and abnormal temperature distribution of each temperature zone (such as temperature control failure), and to extract regional extreme values, uniformity and other features.

[0070] 2. Dataset Construction Method A labeled sample set of 10,000 groups was constructed, covering 8 typical faults, with no less than 1,000 samples for each fault type, and normal operation samples accounting for 30% to ensure balance.

[0071] In the data preprocessing stage, the vibration signal is denoised using the db4 wavelet basis, the energy consumption data is filtered by a sliding window, and the temperature field data is segmented using K-means. Finally, a 224-dimensional joint feature vector is extracted and constructed.

[0072] All samples were divided into training, validation, and test sets in a 7:2:1 ratio, which were used for model parameter fitting, hyperparameter tuning, and final generalization performance evaluation, respectively.

[0073] II. Model Construction and Training 1. Overall Model Architecture It consists of an "attention mechanism module + an improved ResNet module": Attention Mechanism Module: This module achieves scene-adaptive allocation of modal weights through a cross-modal attention mechanism. The input is a 224-dimensional multimodal feature vector containing vibration signals, energy consumption data, and temperature data.

[0074] The specific components of the module include: Modal feature separation unit: Separates the input feature vector into vibration feature subspace, energy consumption feature subspace and temperature feature subspace according to the modal source.

[0075] Feature projection layer: Maps the features of each subspace to a feature representation of a unified dimension through independent linear transformations.

[0076] Cross-modal attention computation unit: Employs a scaled dot product attention mechanism to calculate the interdependencies between features of different modalities and generate a dynamic attention weight matrix.

[0077] Weighted fusion layer: Based on the attention weight matrix, the features of each modality are weighted and summed to output a 128-dimensional deep fusion feature vector.

[0078] The core of this module lies in automatically adjusting the weight coefficients of each mode for different fault types. In compressor fault diagnosis scenarios, the vibration mode is given a relatively high weight range of 0.6-0.8; in fluid leakage fault scenarios, the temperature mode obtains a dominant weight of 0.7-0.9; and for system efficiency anomaly faults, the weight of the energy consumption mode is increased to an importance level of 0.5-0.7.

[0079] Improved ResNet module: Targeted optimizations are made based on the ResNet-50 architecture. The first-layer convolutional kernel is adapted to a 7x1 one-dimensional convolution to match the 128-dimensional deep fusion features of the input. After the core residual block in the middle stage of the network, a Dropout layer (with a dropout probability set to 0.1) is inserted to enhance the model's generalization ability and prevent overfitting. Finally, the fully connected output layer after the global average pooling layer is activated using the Softmax function, outputting the confidence probability distribution corresponding to the 8 fault states.

[0080] III. Fault Alarm Push 1. Alarm Trigger Logic Fault confidence-based triggering: Warning threshold (60%-70%): When the fault confidence level is in this range, it is judged as a potential fault, and only the LED yellow light will flash, while the warning data is stored to the 16GB SD card; Alarm threshold (80%): When the confidence level is >80%, it is determined to be a clear fault and triggers a dual alarm: the local buzzer (volume ≥65dB) is activated + the red LED is constantly lit, and the alarm is pushed to the cloud platform remotely via Wi-Fi module through MQTT, and then forwarded to the user's APP.

[0081] 2. Alarm Information Content The push notification includes core fault elements: fault type (e.g., "compressor stuck"), confidence level (e.g., 0.92), occurrence time (accurate to the second), and key data snippets (vibration / energy consumption / temperature data 5 minutes before and after the fault), making it easy for users to quickly understand the fault situation and for after-sales personnel to accurately troubleshoot.

[0082] Alternatively, the present invention can also use Mamba or CNN+Transformer architecture to replace ResNet50 for feature extraction, thereby improving the recognition rate of fault diagnosis.

[0083] Alternatively, this invention can replace the cross-entropy loss with Label Smoothing Cross Entropy, which alleviates the overfitting of the model to "similar faults" by adding a smoothing coefficient of 0.1 to the true label.

[0084] Furthermore, the core of this invention lies in utilizing deep networks to learn fault representations from fused multimodal features. In addition to the aforementioned ResNet-50 architecture, other advanced models can be used as replacements.

[0085] 1. State-space model architecture: To improve the modeling efficiency of long-sequence multimodal features, state-space model-based architectures such as Mamba can be used to replace ResNet50. In this refrigerator diagnostic scenario, the Mamba architecture can more efficiently handle the mixed long-range dependencies between vibration signals (high-sampling-rate time-series data) and energy consumption sequences (low-frequency time-series data), thereby more accurately capturing the subtle vibration pattern evolution in the early stages of faults such as "compressor piston jamming," or the long-term trend of slowly increasing energy consumption in faults such as "condenser heat dissipation efficiency decline," thus improving the early identification rate of progressive faults.

[0086] 2. Feature extraction scheme based on CNN+Transformer hybrid architecture.

[0087] A hybrid architecture of "CNN local feature extraction + Transformer global correlation modeling" is constructed to replace and improve ResNet-50, taking into account both local features and global correlations of refrigerator multimodal data.

[0088] (1) CNN branch: Three layers of one-dimensional convolution (with kernel sizes of 3, 5 and 7) are used to extract local features in the time-frequency domain of vibration signals, statistical local features of energy consumption data, and regional extreme value features of temperature field data, and output a 128-dimensional local feature vector. (2) Transformer branch: A 2-layer lightweight Transformer encoder (with only 4 attention heads) is used to perform cross-modal global correlation modeling on the local features of the CNN output, and to capture the fault chain features of "abnormal vibration - increased energy consumption - temperature shift". (3) Feature fusion layer: The local features of CNN and the global features of Transformer are fused by adding elements one by one to output 128-dimensional deep fusion features, which are then input into the Softmax classification layer to output 8 types of fault confidence.

[0089] In addition, considering the dual attributes of refrigerator fault characteristics of "local specificity + global correlation", CNN can accurately capture single-modal local abnormal features (such as peak abrupt changes in vibration signals), and Transformer can mine cross-modal global correlations (such as vibration abnormalities accompanied by increased energy consumption). This can solve the pain point of the single ResNet model's insufficient ability to capture complex fault features.

[0090] In addition, the original cross-entropy loss function is replaced with Label Smoothing Cross Entropy, with a smoothing coefficient set to 0.1. During model training, targeted optimizations are performed for similar refrigerator faults (such as temperature control module failure and abnormal door sealing performance, sensor drift and poor electrical circuit contact). By using label smoothing, the model's dependence on "either / or" hard labels is reduced, allowing the model to output non-zero confidence for similar faults. Combined with the modal weights output by the attention mechanism (such as temperature control failure focusing on temperature features, and door sealing focusing on energy consumption features), similar faults are further distinguished.

[0091] In addition, to address the industry pain point of "confusing similar fault features" in refrigerator fault diagnosis (such as temperature control failure and door sealing abnormality both leading to increased temperature and energy consumption), label smoothing is used to alleviate the overfitting of the model to similar faults and avoid misjudging "door sealing abnormality" as "temperature control failure".

[0092] like Figure 3As shown, the present invention provides a diagnostic device for equipment faults, comprising: Acquisition unit 31 is used to acquire multimodal data with a unified timestamp from the device; Preprocessing unit 32 is used to preprocess the multimodal data with the unified timestamp; Extraction unit 33 is used to extract fault features from preprocessed multimodal data and generate a joint feature vector of unified timestamps of a predetermined number of dimensions; The fusion unit 34 is used to perform multimodal feature fusion on the joint feature vector of the unified timestamp of the predetermined number of dimensions, and dynamically adjust the modal weights through the attention mechanism module to generate the fused joint feature vector of the unified timestamp. The generation unit 35 is used to perform fault reasoning based on the fused joint feature vector of the unified timestamp, by improving the ResNet model, and to generate fault type and fault type confidence.

[0093] Optionally, the device further includes: Processing unit 36 ​​is used to perform hierarchical processing based on the fault type and the confidence level of the fault type.

[0094] The present invention also provides an electronic device, comprising: a processor and a memory, wherein the processor is configured to execute a network distribution method stored in the memory to implement the method.

[0095] The present invention also provides a storage medium storing one or more programs, which can be executed by one or more processors to implement the method.

[0096] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204. The processor 1201, communication interface 1202, and memory 1203 communicate with each other via the communication bus 1204. Memory 1203 is used to store computer programs; Processor 1201 is used to execute the program stored in memory 1203. The aforementioned communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0097] The communication interface is used for communication between the aforementioned devices.

[0098] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0099] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0100] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to perform any of the methods described in the above embodiments.

[0101] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in the above embodiments.

[0102] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0103] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0105] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for diagnosing equipment faults, characterized in that, include: Acquire multimodal data with a unified timestamp from the device; The multimodal data with the unified timestamp is preprocessed; Fault features are extracted from preprocessed multimodal data to generate a joint feature vector with a predetermined number of dimensions of unified timestamps; Multimodal feature fusion is performed on the joint feature vector of the unified timestamps of the predetermined number of dimensions, and the modal weights are dynamically adjusted through an attention mechanism algorithm to generate the fused joint feature vector of the unified timestamps. Based on the joint feature vector fused from the unified timestamp, fault reasoning is performed by improving the ResNet model to generate fault types and their confidence levels.

2. The method according to claim 1, characterized in that, The method further includes: Based on the fault type and the confidence level of the fault type, a graded processing method is adopted.

3. The method according to claim 1, characterized in that, Before the step of acquiring the unified timestamp of the device's multimodal data, the method further includes: Collect aggregated fault data to train an improved ResNet model, thereby training the attention weight calculation algorithm and ResNet parameters of the improved ResNet model.

4. The method according to claim 1, characterized in that, When the device is a refrigerator, the multimodal data includes one or more of the following data in any combination: The refrigerator's compressor operating vibration signal; the refrigerator's energy consumption data; the refrigerator's global temperature field data.

5. The method according to claim 4, characterized in that, The preprocessing steps for the multimodal data include: The preprocessing of the operational vibration signal specifically includes: performing a predetermined layer decomposition on the operational vibration signal using a wavelet basis to remove a predetermined power frequency interference; extracting the time-domain and frequency-domain features of the operational vibration signal; calculating the feature mean of a predetermined number of time windows through a sliding window of predetermined duration; and / or The preprocessing of the energy consumption data specifically includes: eliminating grid fluctuation interference through a predetermined sliding window filter; extracting the statistical features of the energy consumption data and calculating the trend features of the energy consumption data by combining it with a predetermined group of historical data for a predetermined duration; The preprocessing of the global temperature field data is as follows: using the K-means algorithm, a predetermined number of pixels are divided into different temperature zones, the statistical features of the temperature in each temperature zone are extracted, and combined with a predetermined group of historical data for a predetermined time period, the trend features of the global temperature field data are calculated. The preprocessed multimodal data is used to form a multimodal joint feature vector for a single sample, and the timestamp of the acquisition is marked.

6. The method according to claim 1, wherein the step of performing multimodal feature fusion on the joint feature vector of the predetermined number of unified timestamps to generate a fused joint feature vector of unified timestamps includes: Feature projection and positional encoding are performed, specifically by mapping the joint feature vector into a query Q, key K, and value V matrix through three learnable linear layers, and adding sinusoidal positional encoding to inject modal sequence information. Self-attention weight calculation specifically involves calculating the association weights of each modality by scaling the dot product attention. Dynamic adjustment of modality weights: Specifically, weights are automatically assigned based on feature similarity; The weighted features are nonlinearly transformed through a two-layer feedforward neural network, and the fused joint feature vector is output.

7. The method according to claim 1, wherein the step of performing fault reasoning and generating fault type and fault type confidence by improving the ResNet model based on the fused joint feature vector of the unified timestamp includes: Feature adaptation and extraction are performed as follows: the first layer uses a one-dimensional convolutional kernel to adapt to the fused joint feature vector; convolution operation is performed on the input features to output the feature map; a Dropout layer is inserted after each residual block in the intermediate stage. Global average pooling is performed by converting the feature map into a vector of a predetermined number of dimensions, inputting it into a fully connected layer, activating it through the Softmax function, and outputting the confidence probability distribution of a predetermined number of faults. Based on the maximum confidence value in the confidence probability distribution and the corresponding fault type characteristics, the fault type and the confidence level of the fault type are determined.

8. A diagnostic device for equipment malfunctions, characterized in that, include: The acquisition unit is used to acquire multimodal data with a unified timestamp from the device. The preprocessing unit is used to preprocess the multimodal data with the unified timestamp. The extraction unit is used to extract fault features from preprocessed multimodal data and generate a joint feature vector with a predetermined number of dimensions of unified timestamps; The fusion unit is used to perform multimodal feature fusion on the joint feature vector of the unified timestamp of the predetermined number of dimensions, and dynamically adjust the modal weights through an attention mechanism algorithm to generate the fused joint feature vector of the unified timestamp. The generation unit is used to perform fault reasoning based on the fused joint feature vector of the unified timestamp, by improving the ResNet model, and to generate fault types and confidence scores for the fault types.

9. An electronic device, comprising: A processor and a memory, the processor being configured to execute a network distribution method stored in the memory to implement the method of any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the method of any one of claims 1 to 7.

Images