Electric energy meter detection assembly line fault diagnosis and prediction method based on multi-mode time sequence analysis

Abstract

The invention discloses an electric energy meter detection assembly line fault diagnosis and prediction method based on multi-modal time sequence analysis. The method comprises the steps of collecting multi-modal data including electric energy meter visual data, time sequence sensor data and text log data in real time; performing cross-modal fusion of time sequence alignment on the multi-modal data to generate joint feature representation; performing joint optimization of fault diagnosis and prediction; performing fault diagnosis based on the joint feature representation, and outputting a current fault type and probability; predicting a future fault probability based on the equipment state continuous evolution model; in response to batch conduction characteristics which are output by the prediction model and reach a preset abnormal value, triggering recalculation of the associated modal data; correcting an initial condition of the equipment state continuous evolution model according to the fault type obtained through re-calculation; and dynamically adjusting a diagnosis decision threshold according to the prediction confidence output by the corrected equipment state continuous evolution model.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for industrial equipment, and in particular to a method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis. Specifically, it is a method that integrates multimodal data from vision, time series sensors, and text logs, and utilizes large model technology to achieve real-time fault diagnosis and prediction of an energy meter testing pipeline. Background Technology

[0002] Current electricity meter testing lines generally employ analysis methods based on manual feature engineering, which face significant limitations in real-world industrial scenarios. Manually designed feature rules rely heavily on preset patterns and thresholds, making it difficult to effectively capture complex and ever-changing fault patterns, especially sudden faults and hidden correlations. Sudden faults are often instantaneous and unpredictable, and their characteristics can easily exceed the coverage of preset rules, leading to the omission of key anomaly signals. Furthermore, deep-seated correlations hidden in different modal data (such as the intrinsic link between appearance defects and current fluctuations) are difficult to fully utilize due to the lack of automated mining mechanisms.

[0003] At the data processing level, existing technologies typically perform separate analysis of vision, sensor, and text logs. The vision module focuses only on the appearance status, the sensor module processes changes in physical quantities such as current and voltage independently, and the text log independently parses fault records. This fragmented approach leads to the loss of collaborative information between different modal data; for example, it is impossible to correlate visual anomalies of terminal blocks with simultaneous current surges. Furthermore, the differences in acquisition frequencies of different modal data (e.g., 30 frames per second images versus 1000 sensor samples per second) make timing alignment difficult, further reducing the accuracy of condition assessment.

[0004] Existing detection models mostly maintain static characteristics after deployment, making it difficult to adapt to the changing needs of actual production environments. When new fault modes (such as material aging caused by new processes) or new sensor types emerge, the system needs to re-label a large amount of data and perform full model training, consuming significant time and computational resources. For rare faults with low occurrence frequency, static models lack diagnostic capabilities due to the absence of continuous learning mechanisms. These problems collectively restrict the efficiency and reliability of electricity meter detection, necessitating a technical solution that can adapt to dynamic environments and deeply integrate multimodal information. Furthermore, existing technologies do not pay sufficient attention to the characteristics that trigger some rare faults, resulting in inaccurate detection and incomplete detection feature data for handling rare faults. Summary of the Invention

[0005] To address the shortcomings and deficiencies of existing technologies, this invention provides a method for fault diagnosis and prediction in a power meter testing pipeline based on multimodal time-series analysis. This method constructs a multimodal data collaborative analysis framework by simultaneously acquiring visual images of key components such as the power meter terminal blocks and LCD screen, high-frequency sampled current and voltage sensor sequences, and fault text logs. The core innovation lies in:

[0006] Dynamic temporal alignment mechanism: Employing interpolation fusion and learnable time offset compensation technology, it solves the problem of millisecond-level timestamp misalignment between visual and sensor data, ensuring accurate synchronization of cross-modal features;

[0007] Prediction-diagnosis closed-loop system: Based on the continuous evolution model, it predicts the fault trend of the same batch of energy meters. When abnormal transmission characteristics are detected, the diagnosis module is triggered to focus on the recalculation of the associated modes and feeds back the initial state of the evolution model through the diagnosis results to form a self-optimizing feedback.

[0008] Adaptive Industrial Scene Design:

[0009] By dynamically adjusting diagnostic thresholds based on batch risk coefficients, collaborative management of equipment within the same batch can be achieved.

[0010] When adding a new sensor mode, a parameter isolation mechanism is used to fine-tune the adaptation layer only, and online expansion is completed within 24 hours.

[0011] By combining historical case retrieval with abnormal state simulation, the response speed for rare faults can be improved.

[0012] This method breaks through the limitations of traditional static models, significantly improving the ability to capture sudden faults and implicit correlation patterns, and providing real-time and accurate fault control support for electricity meter production lines.

[0013] The present invention specifically adopts the following technical solution:

[0014] A method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis includes:

[0015] Real-time acquisition of multimodal data, including visual data from electricity meters, time-series sensor data, and text log data; time-aligned cross-modal fusion of the multimodal data to generate joint feature representations;

[0016] Perform joint optimization of fault diagnosis and prediction:

[0017] Fault diagnosis is performed based on the joint feature representation, and the current fault type and probability are output.

[0018] Predicting future failure probabilities based on a continuous evolution model of equipment state;

[0019] And execute closed-loop feedback:

[0020] In response to the correlation fault characteristics (such as the temperature anomaly conduction pattern of equipment in the same batch appearing at the same time in the same production batch, the judgment criterion is that more than 30% of the equipment in the same batch have the same early anomaly) presented in the output of the prediction model reaching the preset anomaly value, the recalculation of the correlation modal data is triggered.

[0021] The initial conditions of the continuous evolution model of the equipment state are corrected based on the fault type obtained from the recalculation.

[0022] The diagnostic decision threshold is dynamically adjusted based on the prediction confidence level output by the modified equipment state continuous evolution model.

[0023] Furthermore, the continuous evolution model of the device state is a neural differential equation. The initial state is determined by the joint encoding of visual features, sensor features and text features, and the state evolution process is modeled through a continuous-time dynamic system.

[0024] Furthermore, the recalculation includes:

[0025] Retrieve similar faults from the historical case database and match the abnormal features output by the prediction model;

[0026] Focusing on the modal data subset associated with the abnormal features; specifically: based on the modal source of the abnormal features (such as current abnormality associated with sensor data, appearance abnormality associated with visual data), extract the time window subset of the corresponding modality (such as data from 10 seconds before to 5 seconds after the abnormality occurs).

[0027] Optimization is achieved by minimizing the mean squared error between the recalculation result and the true label. Specifically, the recalculation parameters can be iteratively optimized using gradient descent until the mean squared error is less than a preset threshold.

[0028] Furthermore, the time-aligned cross-modal fusion includes:

[0029] Cubic spline interpolation is used to generate continuous time axis aligned features from visual data and sensor data;

[0030] Features are fused using a time-aware gated attention mechanism, in which a learnable offset based on the acquisition time difference is added to the query vector;

[0031] Text features are used as conditional biases in the calculation of attention weights.

[0032] The fused features include time-aligned visual features, sensor features, and text features; the time-aware gated attention mechanism dynamically adjusts the fusion weights of features at different time points by introducing a time difference factor (such as the exponential decay coefficient of the time interval between two-modal data acquisitions) into the attention weight calculation.

[0033] Furthermore, the adjusted diagnostic decision threshold is linearly adjusted according to the deviation between the predicted confidence level and the benchmark value. The adjustment range is controlled by the batch risk coefficient, and the same risk coefficient is used for electricity meters in the same batch. The adjusted threshold is used as an input parameter in the next round of prediction. Preferably, the benchmark value can be set to 0.8 times the average confidence level of historical similar faults.

[0034] Furthermore, when adding new sensor modes:

[0035] Distribute lightweight adapters via a modal routing controller;

[0036] Freeze the backbone network parameters and train only the low-rank adaptation layer and routing module of the newly added modality.

[0037] Furthermore, multimodal anomaly state simulation data is generated during the prediction process.

[0038] Furthermore, the visual data acquisition includes key components such as the power meter terminal block, LCD screen, and lead seal.

[0039] Visual feature extraction uses the Vision Transformer model, and the small sample generalization ability is optimized through pre-training of a mask autoencoder; the fault diagnosis calls the electricity meter's dedicated defect database, which contains defect samples such as terminal oxidation and shell cracks and their corresponding feature parameters.

[0040] Furthermore, in the process of fault diagnosis based on the joint feature representation, the detection of rare fault feature blind zones is performed by calculating the manifold blind zone index ζ(x) of the detection sample x, including:

[0041] Evaluation of local distribution anomalies based on probability density gradient;

[0042] Dimensional correction is performed based on the sparsity of neighboring samples;

[0043] Sensitivity is quantified by the ratio of eigenvalues ​​of the sensor matrix.

[0044] Furthermore, when a missing dimension of sensor data is detected, it is determined to be a structural blind spot;

[0045] When an abnormal pattern is detected that lacks training samples, it is determined to be a data blind spot;

[0046] When the blind zone index exceeds the dynamic threshold, physical characteristic compensation is triggered, including:

[0047] The variation pattern of contact resistance of integrated terminal blocks;

[0048] Introduce material thermal expansion characteristic parameters.

[0049] The present invention also provides a fault diagnosis and prediction system for an electricity meter testing pipeline, comprising: a multimodal acquisition unit for synchronously acquiring terminal block visual images, current / voltage sensor sequences and fault text logs; a timing fusion unit for performing cross-modal fusion as described above; a prediction and diagnosis unit for implementing joint optimization as described above; and a closed-loop control unit for performing closed-loop feedback steps as described above.

[0050] And an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0051] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0052] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0053] 1. Breaking through the bottleneck of complex pattern capture: Through the multimodal temporal dynamic alignment mechanism, it effectively overcomes the limitations of traditional methods in identifying sudden failures and implicit correlation patterns, significantly improves the ability to perceive instantaneous anomalies and cross-modal correlation features, and solves the problem that manual feature engineering is difficult to adapt to complex data forms.

[0054] 2. Achieving Synergistic Optimization of Diagnosis and Prediction: A prediction-diagnosis closed-loop system is constructed, utilizing a continuous evolution model to predict fault trends and provide real-time feedback to correct diagnostic strategies. When abnormal features are detected, the system dynamically focuses on recalculating related modes and adjusts thresholds through a batch risk mechanism, significantly improving the accuracy of early warnings for faults and overcoming the shortcomings of static models that separate diagnosis and prediction.

[0055] 3. Enhance adaptability to industrial scenarios:

[0056] Employing a parameter-isolated incremental architecture, it supports rapid expansion of new sensor modes, requiring only minor adjustments to the adaptation layer to complete online upgrades, significantly shortening the deployment cycle of new features;

[0057] By integrating historical case retrieval and abnormal state simulation, the ability to respond to rare faults is enhanced, and the diagnostic blind spot of traditional models when facing low-frequency faults is addressed.

[0058] By using text-conditional bias and time-aware gating, the interpretability of multimodal collaboration is improved (e.g., automatically associating appearance regions when current changes suddenly).

[0059] 4. Optimize resource utilization efficiency: Based on the dedicated defect library of electricity meters and the detection of key parts (terminal blocks, LCD screens, etc.), it can achieve accurate feature extraction in small sample scenarios, reduce data labeling and computing power consumption, and provide a cost-effective intelligent operation and maintenance solution for the production line.

[0060] 5. Increased attention to the characteristics of some rare faults, solving the problems of insufficient accurate detection and incomplete detection feature data in existing technologies for handling rare faults. Attached Figure Description

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0062] Figure 1 This is a diagram illustrating the overall architecture of an embodiment of the present invention.

[0063] Figure 2 The diagram shows an embodiment of the present invention.

[0064] Figure 3 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0065] In the following, specific embodiments of this application will be described in detail with reference to the accompanying drawings. Based on these detailed descriptions, those skilled in the art will be able to clearly understand and implement this application. Without departing from the principles of this application, features from various embodiments can be combined to obtain new implementations, or certain features from some embodiments can be substituted to obtain other preferred implementations.

[0066] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings:

[0067] Existing electricity meter testing pipelines have weak capabilities in capturing complex patterns. Traditional multimodal time-series analysis relies on manual feature engineering, which is insufficient for capturing sudden faults and implicit correlation patterns. Pipeline testing uses a single-modal model, processing only image or sensor data, and does not fully utilize multimodal collaborative information (such as the correlation between appearance defects and current fluctuations). Static models are difficult to adapt to new fault types or sensor modes.

[0068] like Figures 1-3As shown, the embodiments of this invention aim to address the challenges of electricity meter testing pipelines by proposing a method and system that integrates multimodal data from vision, time-series sensors, and text logs, utilizing advanced modeling techniques to achieve real-time fault diagnosis and prediction. Through steps such as multimodal unified representation learning, dynamic cross-modal fusion, fault diagnosis and prediction, and online adaptive learning, the invention effectively overcomes the shortcomings of traditional methods in feature extraction, multimodal information utilization, and model adaptability, thereby improving the accuracy and efficiency of fault diagnosis and prediction.

[0069] 1. The core technical solution architecture is as follows:

[0070] (I) Multimodal Unified Representation Learning

[0071] The input data sources are as follows: visual modality is obtained from appearance inspection images of electricity meters captured by a high-resolution camera; time-series modality is composed of time-series sequences acquired by voltage, current, and temperature sensors at a sampling rate of 1 kHz.

[0072] Fault codes and maintenance records in the equipment log, such as "voltage over-limit", form a text modal.

[0073] The feature extraction process is as follows: Vision Transformer (ViT) is used to extract image features, and a masked autoencoder (MAE) is pre-trained to optimize few-shot generalization ability. A hybrid architecture of Temporal Convolutional Network (TCN) and Mamba is used to effectively capture long-short-term dependencies. Semantic embeddings are generated based on a fine-tuned LLaMA-3 model to accurately extract key concepts for fault description.

[0074] (ii) Dynamic cross-modal fusion

[0075] The design incorporates a gated cross-attention module that uses visual features as queries and temporal features as key-value pairs. It dynamically calculates the weight matrix and introduces textual features as conditional biases to enhance the interpretability of the fusion process, such as prioritizing the appearance of specific regions during sudden current changes.

[0076] (III) Fault Diagnosis and Prediction

[0077] Diagnostic branch: The fault classification results are output using a multilayer perceptron (MLP), such as "loose terminal block" or "damaged motherboard".

[0078] Prediction branch: Construct a Neural ODE model, input the current multimodal state, and output the failure probability curve for the next N hours. Jointly train the diffusion model to generate possible future multimodal abnormal states, such as predicting "cracks and temperature rise may occur in the meter terminals after a certain period of time".

[0079] (iv) Online adaptive learning

[0080] Deploy the LoRa (LoRa) module for efficient parameter fine-tuning: when a new fault type is detected, only the low-rank matrix is ​​updated, preserving the backbone network parameters for rapid adaptation. Store rare fault cases and improve the speed of recurring fault diagnosis through retrieval-aumented processing.

[0081] 2. Detailed Plan Description

[0082] (a) Multimodal dynamic alignment

[0083] This embodiment proposes a time-domain synchronized cross-modal attention mechanism. It preprocesses data from different modalities using methods such as dual-timescale interpolation and event timestamp parsing to generate aligned features on a unified time axis. A Time-AwareGating module is designed to introduce time-off weights into the cross-attention process. When there is a delay in the acquisition of image and sensor data, the feature weights of non-aligned time periods are automatically reduced, effectively solving the problem of timestamp misalignment between visual and sensor data in traditional methods.

[0084] Temporally synchronized cross-modal attention mechanism

[0085] 1. Temporally synchronized cross-modal attention mechanism

[0086] (1) Double time-scaled interpolation

[0087] For visual interpolation and sensor interpolation Alignment features on a uniform time axis t are generated using cubic spline interpolation:

[0088] (Visual interpolation)

[0089] (Sensor interpolation)

[0090] Where a i ,b j The coefficients of the cubic spline interpolation are obtained by fitting the original data using the least squares method; t is the time variable on a unified time axis. Let be the timestamp of the i-th frame of the visual data. The timestamp of the j-th sampling point of the sensor data;

[0091] The boundary continuity condition is:

[0092]

[0093] The above boundary continuity condition is used to solve for the coefficient 'a' of cubic spline interpolation.i and b j By solving simultaneous equations, the continuity and smoothness of the interpolation function at the time axis segmentation points are ensured, and finally, alignment features on a unified time axis are generated.

[0094] (2) Time-aware gating attention

[0095]

[0096] Where α is the learnable decay coefficient, d is the feature dimension, and given query Q, key K, value V, and time difference Δt:

[0097] The aforementioned visual and sensor interpolation can be derived from images of key components of the electricity meter, such as terminal blocks, LCD screens, and lead seals, captured by high-resolution industrial cameras. Further feature extraction is performed, and a model is pre-trained on an electricity meter defect database (containing image samples of terminal oxidation, casing cracks, etc.). Sensor-related differences are extracted from current / voltage sensors (1kHz sampling, detecting short-time pulses, harmonic distortion, and other grid disturbances). The training strategy targets small-sample defects (such as aging of rare-model sealing rings). The data is then subjected to dual-time-scale interpolation alignment. The alignment process incorporates time-aware gating attention training and control. Given the query condition Q, key data parameter reference key value K, and sensor and visual differences, the final calculated data value V and time difference Δt are:

[0098] (II) Diagnostic Joint Optimization

[0099] By learning shared representations, a Neural ODE model is constructed to model the state evolution of the device. The diffusion model generates hypotheses about future states, enabling the prediction results to feed back into the diagnostic model. For example, early features discovered by the prediction module can be used to correct the diagnostic threshold, and the diagnostic results are used to correct the initial conditions of the Neural ODE, re-predict the remaining lifespan, and adjust the diagnostic threshold in real time based on the confidence level of the prediction results.

[0100] Diagnostic Joint Optimization Prediction

[0101] Neural differential equation state evolution:

[0102]

[0103] in As a combined characterization, it can be understood as a comprehensive equipment health index, reflecting the trend of fault development over time. As a visual feature, For sensor characteristics, For example, in a scenario where a batch of loose terminals in a certain type of electricity meter is detected, the prediction module finds that the temperature rise rate deviates from the baseline trajectory, triggering the diagnostic module to focus on checking the screw torque characteristics. As a visual feature, For sensor characteristics, (Text features) are combined with the above features to perform neural differential equation state evolution, thereby predicting possible faults and defects, and monitoring them. The monitored fault information is further integrated into the text features to form records and loops. After confirming the fault, the prediction confidence threshold of the same batch of meters is automatically lowered.

[0104] (III) Incremental Architecture

[0105] This embodiment supports online expansion of new modalities (such as adding acoustic sensors) without retraining the entire model. Through a modal routing controller, new modalities are connected via lightweight adapters, generating corresponding key-value matrices, and the controller dynamically allocates computational resources. A parameter isolation mechanism is employed to freeze the backbone network parameters, training only the LoRA layer and routing decision module of the new modalities.

[0106] (iv) Closed-loop feedback of prediction-diagnosis joint optimization

[0107] Shared representation space construction: Neural ODE is used to model the device state evolution. The diffusion model adds noise to the current state through a forward process to simulate fault evolution, and the reverse process optimizes the decision boundary of the diagnostic module based on the prediction error.

[0108] Feedback closed-loop design, prediction-guided diagnosis: When the prediction module detects an early anomaly, it automatically triggers a local recalculation in the diagnosis module, focusing on relevant modes. For example, if the temperature rise rate of batch A meters is detected to be faster than the baseline, a focus command is sent to correct the prediction. The diagnosis result is used to correct the initial conditions of the Neural ODE and re-predict the remaining lifespan. Dynamic threshold adjustment: Based on the confidence level of the prediction results, the diagnostic threshold is adjusted in real time, and a confidence threshold is set for the temperature rise rate of the next batch B meters. Preferably, the dynamic threshold can be set to the 95th percentile of the historical blind zone index of similar devices.

[0109] Self-correcting closed-loop systems include:

[0110] (1) Local recalculation and correction prediction

[0111] When the prediction module detects an early anomaly, it automatically triggers a local recalculation of the diagnostic module, focusing on the relevant modes. Let... For input data, For the predicted results, If the values ​​are the true labels, then the goal of local recomputation is to minimize the following loss function:

[0112]

[0113] in:

[0114] : Input data vector.

[0115] : The prediction result vector after local recalculation.

[0116] : Real label vector.

[0117] : The joint probability distribution of input data and true labels.

[0118] L2 norm.

[0119] : Expectation under the joint probability distribution.

[0120] (2) Adjust the diagnostic threshold in real time based on the confidence level of the prediction results.

[0121] set up The confidence level of the prediction result. The current diagnostic threshold, Given the adjusted diagnostic threshold, the dynamic threshold adjustment can be expressed as:

[0122]

[0123] in:

[0124] : Confidence level of the prediction result.

[0125] Current diagnostic threshold.

[0126] : Adjusted diagnostic threshold.

[0127] Adjust the step size to control the magnitude of threshold adjustment.

[0128] The midpoint of the confidence level serves as the benchmark for adjustment.

[0129] When adjusting the confidence threshold for the temperature rise rate of the next batch of B meters, simply adjust the corresponding prediction confidence level. Substitute into the above formula, where This is the true label for the temperature rise rate of batch B meters. This is the input data for the temperature rise rate of meter B in batch B.

[0130] 3. Application Scenarios Examples

[0131] (I) Cooperative design of multimodal dynamic alignment and incremental expansion

[0132] The cross-modal attention mechanism for temporal synchronization involves input preprocessing: performing dual-time-scale interpolation on visual data (30fps video stream) and sensor data (1kHz sampling), and generating alignment features on a unified time axis through a cubic spline function; text logs are mapped to the synchronization time axis through event timestamp parsing (e.g., regular expression matching "2024-03-05 14:00:00: ERROR 1102").

[0133] Dynamic Gated Attention: The Time-Aware Gating module is designed to introduce time offset weights in cross-attention, thereby automatically reducing the weights of features in non-aligned time periods.

[0134] Incremental modal expansion and modal routing controller: New modalities are connected via lightweight adapters, generating corresponding key-value matrices, and the controller dynamically allocates computing resources. Parameter isolation mechanism: Backbone network parameters are frozen, and only the LoRA layer and routing decision module of the newly added modal are trained.

[0135] (ii) Closed-loop feedback of prediction-diagnosis joint optimization

[0136] Shared representation space construction: Neural ODE is used to model the device state evolution. The diffusion model adds noise to the current state through a forward process to simulate fault evolution, and the reverse process optimizes the decision boundary of the diagnostic module based on the prediction error.

[0137] (III) Implementation flow of each module's collaborative operation

[0138] Real-time monitoring phase: Collect image, current, and temperature data, generate time-synchronized joint features through the dynamic alignment module, and output the current fault probability through the diagnostic module, such as "motherboard short circuit: 72%".

[0139] Prediction Feedback Phase: Neural ODE predicts the state for the next 2 hours, and the diffusion model generates possible anomaly images. The prediction module detects abnormal voltage fluctuation trends and sends a focusing command to the diagnostic module, enhancing the detection weight of the power supply module.

[0140] Incremental learning phase: When a new vibration sensor is added, the routing controller automatically allocates adapter resources and completes online fine-tuning within 24 hours. New modal data is accessed through the dynamic alignment module and participates in the next round of prediction-diagnosis closed loop.

[0141] Multimodal dynamic alignment: This embodiment proposes a temporally synchronized cross-modal attention mechanism to solve the problem of misaligned timestamps between visual and sensor data in traditional methods. Prediction-diagnosis joint optimization: Through shared representation learning, prediction results are fed back into the diagnostic model (e.g., early features discovered by the prediction module are used to correct diagnostic thresholds). Incremental architecture: Supports online expansion to new modalities (e.g., adding acoustic sensors) without retraining the entire model.

[0142] Furthermore, for some rare faults, this embodiment incorporates consideration of the missing detection of feature blind spots that lead to these faults. For example, it fully considers the existence of feature blind spots in nearby or similar functional sensors, which can cause data sample bias. This is addressed by optimizing relevant algorithms and ultimately forming a rare defect detection database. This solves the problem in existing technologies where insufficient attention is paid to the features that trigger some rare faults, resulting in inaccurate detection and incomplete detection feature data for rare fault handling.

[0143] The following algorithm is used for blind spot detection:

[0144] Let the test sample be Its manifold blind zone index It consists of the following three parts:

[0145]

[0146] Parameter description:

[0147] Local probability density estimation;

[0148] : of - Neighborhood sample set ( );

[0149] Sensor matrix;

[0150] Dimensional correction factor;

[0151] In a preferred embodiment, this application addresses the problem of missed detection of rare faults due to sensor limitations and insufficient samples by constructing an intelligent blind zone assessment system. When the detection system identifies significant information gaps or insufficient measurement dimensions in the sensor data, it automatically determines it as a structural blind zone caused by hardware limitations; when certain abnormal patterns lack sufficient training samples, it determines it as a data-related blind zone. The system employs a dynamic threshold mechanism to monitor detection quality in real time, automatically triggering a compensation procedure when the evaluation value exceeds the warning line. By incorporating the physical characteristics of the terminal blocks (such as contact resistance variation and material thermal expansion characteristics) into the detection algorithm, the system can both adhere to the basic working principles of electrical equipment and accurately capture subtle defect features that are difficult to detect using traditional methods, such as initial oxidation and micro-cracks, ensuring detection accuracy while meeting the real-time requirements of the production line.

[0152] The technical solutions provided in the embodiments of the present invention have at least the following advantages and effects:

[0153] Overcoming the subjectivity and limitations of feature extraction: By introducing advanced technologies such as deep learning, the model can automatically learn features from multimodal data of the electricity meter inspection line, avoiding information omissions due to differences in feature understanding and selection by different personnel. For subtle defect features in visual images, automated methods can cover more possible shapes and variations, more comprehensively and accurately reflecting the essential characteristics of the data.

[0154] Enhanced ability to capture complex patterns: Develop a feature extraction mechanism that adapts to complex data patterns, overcoming the limitations of relying on preset rules and patterns. Faced with complex and diverse data patterns in real-world industrial scenarios, especially sudden failures and implicit correlation patterns, the new mechanism can respond flexibly. For sudden failures characterized by unpredictability and transient nature, even if their features do not conform to conventional design principles, they can be captured promptly and accurately. Simultaneously, it delves into implicit correlation patterns hidden in the complex interactions of multimodal data, fully leveraging the advantages of multimodal data to improve the detection capability for such failures and patterns.

[0155] Leveraging multimodal collaboration to address the problem of information fragmentation: Constructing a comprehensive model capable of processing data from multiple modalities overcomes the limitations of single-modal processing. For example, in electricity meter testing, the model can analyze external image information, acquire sensor data such as current and voltage, and reference information such as operation records and fault histories in text logs. By comprehensively evaluating the operating status of the electricity meter, the model can capture the correlation between external defects and different modal data such as current fluctuations, promptly identifying early warning signals of potential faults and avoiding missing fault diagnosis opportunities due to fragmented information.

[0156] Leveraging synergistic effects: Design effective multimodal data fusion and analysis methods to fully utilize the inherent synergistic relationships between different modalities of data. Visual data, time-series sensor data, and text logs, among other modalities, complement each other, and fusion analysis can significantly improve the accuracy of fault diagnosis and prediction. For example, in intelligent operation and maintenance of industrial equipment, comprehensively utilizing information from appearance, changes in operating parameters, and operation records provides a more comprehensive and reliable basis for equipment condition assessment, enhancing the model's application effectiveness in complex industrial scenarios.

[0157] Enhancing adaptability to new fault types and sensor modes: Employing a dynamic model architecture with self-learning and adaptive capabilities, enabling it to automatically adjust its internal structure and parameters based on new fault samples. In complex and ever-changing industrial equipment operating environments where new fault types may emerge, the dynamic model can quickly learn and identify these new faults, avoiding misclassification as known fault types or failure to identify them, thus maintaining a high fault diagnosis accuracy. For example, as new fault modes arise with improvements in electricity meter manufacturing processes and the application of new materials, the dynamic model can respond promptly to these changes.

[0158] Improve compatibility with new sensor modes: Develop a highly scalable model structure that can easily handle new sensor mode data. When new sensor modes are introduced into practical applications to more comprehensively monitor equipment operation, the model does not require large-scale modifications and retraining; simple adjustments are all that's needed to utilize the new sensor data for fault diagnosis. For example, when a sensor for monitoring ambient humidity is added to an electricity meter testing line, the model can quickly adapt and utilize this sensor data, reducing development costs and time while avoiding any adverse impact on the performance of the original model.

[0159] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0160] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0161] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0162] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

[0163] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other various forms of fault diagnosis and prediction methods for electricity meter testing pipelines based on multimodal time series analysis. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis, characterized in that, include: Real-time acquisition of multimodal data, including visual data from electricity meters, time-series sensor data, and text log data; Perform time-aligned cross-modal fusion on multimodal data to generate joint feature representations; Perform joint optimization of fault diagnosis and prediction: Fault diagnosis is performed based on the joint feature representation, and the current fault type and probability are output. Predicting future failure probabilities based on a continuous evolution model of equipment state; And execute closed-loop feedback: In response to the correlated fault characteristics exhibited in the same batch of electricity meters whose output of the prediction model reaches the preset anomaly value, the recalculation of the correlated mode data is triggered. The initial conditions of the continuous evolution model of the equipment state are corrected based on the fault type obtained from the recalculation. The diagnostic decision threshold is dynamically adjusted based on the prediction confidence level output by the modified equipment state continuous evolution model. The continuous evolution model of the device state is a neural differential equation. The initial state is determined by the joint encoding of visual features, sensor features and text features, and the state evolution process is modeled by a continuous-time dynamic system.

2. The method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis according to claim 1, characterized in that: The recalculation includes: Retrieve similar faults from the historical case database and match the abnormal features output by the prediction model; Focusing on the modal data subset associated with the aforementioned abnormal features; Optimization is achieved by minimizing the mean squared error between the recalculated results and the true labels.

3. The method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis according to claim 1, characterized in that: The time-aligned cross-modal fusion includes: The fused features include temporally aligned visual features, sensor features, and text features; Cubic spline interpolation is used to generate continuous time axis aligned features from visual data and sensor data; Features are fused using a time-aware gated attention mechanism, in which a learnable offset based on the acquisition time difference is added to the query vector; Text features are used as conditional biases in the calculation of attention weights.

4. The method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis according to claim 1, characterized in that: The adjusted diagnostic decision threshold is adjusted linearly according to the deviation between the predicted confidence level and the benchmark value. The adjustment range is controlled by the batch risk coefficient, and the same risk coefficient is used for electricity meters in the same batch. The adjusted threshold is used as an input parameter in the next round of prediction.

5. The method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis according to claim 1, characterized in that: When adding a new sensor mode: Distribute lightweight adapters via a modal routing controller; Freeze the backbone network parameters and train only the low-rank adaptation layer and routing module of the newly added modality.

6. The method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis according to claim 1, characterized in that: The visual data acquisition includes key parts such as the power meter terminal block, LCD screen, and lead seal. Visual feature extraction uses the Vision Transformer model, and the few-shot generalization ability is optimized by pre-training a mask autoencoder. The fault diagnosis calls upon a dedicated defect database for electricity meters, which includes samples of defects such as terminal oxidation and casing cracks.

7. The method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis according to claim 1, characterized in that: In the process of fault diagnosis based on the joint feature representation, the detection of rare fault feature blind zones is performed by calculating the manifold blind zone index ζ(x) of the detection sample x, including: Evaluation of local distribution anomalies based on probability density gradient; Dimensional correction is performed based on the sparsity of neighboring samples; Sensitivity is quantified by the ratio of eigenvalues ​​of the sensor matrix.

8. The method for fault diagnosis and prediction of an energy meter testing pipeline based on multimodal time series analysis according to claim 1, characterized in that: When a missing dimension of sensor data is detected, it is determined to be a structural blind spot; When an abnormal pattern is detected that lacks training samples, it is determined to be a data blind spot; When the blind zone index exceeds the dynamic threshold, physical characteristic compensation is triggered, including: The variation pattern of contact resistance of integrated terminal blocks; Introduce material thermal expansion characteristic parameters.

9. An electronic device, characterized in that, include: processor; A memory storing a computer program; when the program is executed by a processor, it implements the method of any one of claims 1-8.

Images