Automatic diagnostic analysis system based on computer vision and multi-modal data fusion

Abstract

The invention discloses an automatic diagnosis and analysis system based on computer vision and multi-modal data fusion, and belongs to the technical field of abnormality diagnosis of an electric power measurement assembly line. The cloud comprises a fault library and a diagnosis library; the user side comprises a monitoring verification module, a monitoring module, a configuration module and a diagnosis module; the fault library is used for storing fault problems and fault performance characteristics; the diagnosis library is used for storing diagnosis schemes and diagnosis models; the monitoring verification module is used for carrying out scheme verification on the monitoring scheme to obtain a corresponding verification result, and carrying out corresponding processing according to the verification result; the configuration module is used for performing configuration analysis, acquiring a monitoring simulation model, and configuring a corresponding diagnosis scheme and a diagnosis model for the diagnosis module based on the diagnosis library and the monitoring simulation model; the monitoring module is used for monitoring the metrological verification assembly line in real time to obtain multi-modal data; and the diagnosis module is used for analyzing the multi-modal data to obtain corresponding fault diagnosis data.

Description

Technical Field

[0001] This invention belongs to the field of abnormal diagnosis technology for power metering pipelines, specifically an automated diagnostic analysis system based on computer vision and multimodal data fusion. Background Technology

[0002] With the continuous improvement of automation in metrological verification lines, the introduction of advanced technologies, equipment, and management models has significantly increased work efficiency and effectively enhanced the verification capabilities and testing quality of metrological equipment. However, in terms of line operation and maintenance, the traditional model relying on manual inspection has gradually revealed significant limitations, including low detection efficiency, delayed fault identification, and high rates of misjudgment and missed detection, failing to meet the demands of modern automated systems for efficient, accurate, and real-time monitoring. Simultaneously, the increase in equipment types and the complexity of the operating environment render traditional methods inadequate in dealing with emergencies and complex operating conditions. Therefore, the construction of an anomaly monitoring and analysis system for metrological lines is particularly necessary. By introducing automated fault diagnosis and analysis technologies, the speed of fault detection and response can be significantly improved, the accuracy of fault monitoring can be enhanced, manual intervention can be reduced, and operation and maintenance efficiency can be increased, thereby better adapting to the high standards required for equipment monitoring in automated production systems.

[0003] Breakthroughs in multimodal data fusion technology have provided a key path to solving this problem. By integrating heterogeneous data such as electrical signals from power equipment, infrared thermal imaging, visible light images, vibration spectra, and environmental parameters, diagnostic accuracy can be greatly improved.

[0004] Based on this, the present invention provides an automated diagnostic analysis system based on computer vision and multimodal data fusion. Summary of the Invention

[0005] To address the problems of the above solutions, this invention provides an automated diagnostic analysis system based on computer vision and multimodal data fusion.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] An automated diagnostic analysis system based on computer vision and multimodal data fusion, including cloud and user terminals;

[0008] The cloud platform includes a fault database and a diagnostic database;

[0009] The fault database is used to store various faults in the metrology verification line and the corresponding fault characteristics.

[0010] The diagnostic library is used to store various diagnostic schemes and corresponding diagnostic models for fault diagnosis.

[0011] The user terminal includes a monitoring and verification module, a monitoring module, a configuration module, and a diagnostic module;

[0012] The monitoring and verification module verifies the monitoring scheme, obtains the monitoring scheme, connects to the fault database in the cloud, verifies the monitoring scheme based on the fault database, and obtains the corresponding verification results. The verification results are either qualified or unqualified, along with the reasons for unqualification. The module then performs corresponding processing based on the verification results.

[0013] Furthermore, the monitoring scheme is validated based on the fault database, including:

[0014] A visual model is established for the metrological verification process. According to the monitoring plan, the corresponding monitoring arrangements are carried out in the visual model. The completed visual model is marked as a monitoring simulation model.

[0015] Based on the fault database, generate fault simulation backgrounds; based on the fault simulation backgrounds, generate monitoring simulation data, and supplement the monitoring simulation model with the monitoring simulation data;

[0016] Based on the monitoring simulation model, conduct simulation analysis to determine whether the monitoring simulation model can meet the fault diagnosis requirements under the corresponding fault simulation background;

[0017] When there is a fault simulation background that does not meet the fault diagnosis requirements, the monitoring scheme is evaluated as unqualified, and corresponding reasons for non-compliance are generated based on the fault simulation background that does not meet the fault diagnosis requirements.

[0018] When there is no fault simulation background that does not meet the fault diagnosis requirements, the monitoring scheme is evaluated and verified as qualified.

[0019] Furthermore, the fault simulation backgrounds generated based on the fault database include:

[0020] Identify various faults in the fault database, and combine them according to whether they can occur simultaneously or independently to obtain several fault combinations; generate corresponding fault simulation backgrounds based on the fault combinations.

[0021] Furthermore, the fault simulation backgrounds generated based on the fault database include:

[0022] Identify various faults in the fault database, obtain user monitoring requirements, filter faults based on monitoring requirements, generate several problem combinations based on the remaining faults, and generate corresponding fault simulation backgrounds based on the problem combinations.

[0023] Furthermore, simulation analysis is conducted based on the monitoring simulation model, including:

[0024] The monitoring simulation model is used to collect and simulate data, and the monitoring simulation data is used to identify all fault characteristics that can be collected during monitoring. Based on the identified fault characteristics, it is determined whether all fault problems corresponding to the fault simulation background can be identified.

[0025] When it is determined that all fault problems corresponding to the fault simulation background can be identified, the fault simulation background is evaluated to meet the fault diagnosis requirements.

[0026] If it is determined that not all fault problems corresponding to the fault simulation background can be identified, the fault simulation background is deemed not to meet the fault diagnosis requirements.

[0027] The configuration module is used to perform configuration analysis, obtain the corresponding monitoring simulation model according to the application's monitoring scheme, connect to the cloud-based diagnostic library, and configure the corresponding diagnostic scheme and diagnostic model for the diagnostic module based on the diagnostic library and the monitoring simulation model.

[0028] Furthermore, based on the diagnostic library and monitoring simulation model, corresponding diagnostic schemes and models are configured for the diagnostic module, including:

[0029] Obtain the fault simulation background, and generate corresponding multimodal simulation data for the fault simulation background based on the monitoring simulation model and the fault simulation background; integrate the various multimodal simulation data into diagnostic simulation data;

[0030] Identify the various diagnostic schemes corresponding to the diagnostic library, filter the diagnostic schemes based on the diagnostic simulation data, determine the diagnostic schemes and diagnostic models to be configured, and configure the diagnostic schemes and diagnostic models in the diagnostic module.

[0031] The monitoring module is used to monitor the metrological verification line in real time, obtain corresponding multimodal data, and send the multimodal data to the diagnostic module.

[0032] The diagnostic module is used to perform diagnostic analysis, identify the received multimodal data, and analyze the multimodal data through the configured diagnostic scheme and diagnostic model to obtain corresponding fault diagnosis data.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] This invention completely transforms the traditional, experience-dependent, and inefficient passive maintenance model of manual inspection by constructing an automated diagnostic and analysis system based on computer vision and multimodal data fusion. The system can collect and analyze multi-source heterogeneous data in real time, including electrical signals, infrared thermal imaging, visible light images, vibration spectra, and environmental parameters from power metering and verification lines, achieving second-level fault response and precise fault location. By setting up fault and diagnostic databases in the cloud, the platform can easily aggregate fault problems and fault characteristics from multiple users, improving resource utilization and work efficiency. Furthermore, by cooperating with monitoring and verification modules at various user sites, the system can verify and analyze user monitoring solutions. Simultaneously, the data analysis process occurs entirely at the user end, reducing the risk of data leakage. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a block diagram illustrating the principle of the present invention. Detailed Implementation

[0037] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, an automated diagnostic analysis system based on computer vision and multimodal data fusion includes a cloud platform and a user terminal; and a communication connection between the cloud platform and the user terminal.

[0039] The cloud platform includes a fault database and a diagnostic database.

[0040] The fault database is used to store various faults in the metering verification line and the corresponding fault characteristics, such as intermittent meter position faults. Fault characteristics include: poor contact due to pin wear, resulting in intermittent communication between the electricity meter and the verification device; meter displacement (such as tray positioning deviation), causing the electricity meter to fail to accurately connect to the verification interface; and faults occurring periodically or randomly, resulting in large fluctuations in the verification pass rate.

[0041] In one embodiment, the fault problems and fault performance characteristics are summarized and statistically analyzed based on a large amount of historical fault data.

[0042] The diagnostic library is used to store various diagnostic schemes and corresponding diagnostic models for fault diagnosis. One diagnostic scheme can correspond to one diagnostic model or multiple diagnostic models.

[0043] The diagnostic plan is pre-configured by the platform based on existing monitoring data, facilitating rapid configuration for users later; specifically, it is set according to various current fault identification technologies.

[0044] For example, the diagnostic scheme is as follows:

[0045] A deep learning model based on LSTM and Transformer is constructed. LSTM excels at processing long-sequence data and can capture the changing trends of equipment operating status over time, such as abnormal temperature increases and vibration frequency changes. Transformer uses a self-attention mechanism to explore the correlations between data at different time steps, such as the relationship between different frequency components in vibration data, further improving the model's ability to identify complex fault modes. Through time-series modeling and residual analysis techniques, the system can detect abnormal fluctuation patterns in equipment operation and identify performance degradation problems caused by mechanical wear, aging, or environmental changes. For example, when the vibration spectrum is distorted or the temperature gradient is abnormal, the system can quickly locate the fault and predict the trend of equipment performance degradation. The model design balances lightweight design with high accuracy. By optimizing the network structure and parameters, it ensures stable operation at the edge, achieving full automation from data acquisition to fault warning, and providing timely and accurate data support for equipment maintenance.

[0046] Convolutional Neural Networks (CNNs) are used for feature extraction and pattern recognition. CNNs extract key features from images (such as meter edges and tray outlines) through convolutional layers, pooling layers reduce data dimensionality and computational load, and fully connected layers integrate feature information to support fault localization. To address issues such as varying lighting and dust interference in industrial environments, the image preprocessing workflow is optimized. Adaptive histogram equalization technology is used to improve uneven lighting, and filtering algorithms are combined to remove image noise, ensuring the algorithm's robustness in complex environments. A lightweight model is deployed through an edge computing architecture, achieving millisecond-level real-time detection, significantly reducing the frequency of manual re-inspection and improving production line efficiency. The system can quickly identify common surface anomalies such as meter skew, tray misalignment, and auxiliary pin crimping failures, ensuring real-time monitoring and fault warning of equipment operation status.

[0047] By combining image features (such as surface crack morphology) with sensor temporal features (such as vibration energy distribution), cross-modal data fusion technology improves the comprehensiveness and accuracy of fault classification. Image data can intuitively reflect the surface condition of equipment, while sensor data provides quantitative information on the internal operating status of the equipment. The combination of the two can more comprehensively describe the health status of the equipment. A hierarchical classification strategy is designed to first distinguish major fault categories (such as mechanical faults and electrical faults), and then refine them to specific fault modes (such as bearing wear level and meter installation deviation). To address the potential conflicts in multimodal data, a dynamic weight adjustment mechanism is introduced to automatically adjust the fusion weights based on data reliability and importance, reducing false positives and false negatives.

[0048] The various models corresponding to the diagnostic plan are regarded as diagnostic models. The establishment of diagnostic models is as shown in the example above, which is based on existing intelligent technologies such as existing machine learning and deep learning algorithms.

[0049] For example, the establishment of a diagnostic model:

[0050] Model architecture design:

[0051] Data Acquisition Layer:

[0052] Sensor deployment: Vibration sensors, temperature sensors, current sensors, vision cameras, and acoustic sensors are installed at key nodes of the production line (such as robotic arms, conveyor belts, and testing instruments) to collect equipment operating status data synchronously.

[0053] Log and text data: Integrate text information such as equipment operation logs, maintenance records, and operation instructions, and extract fault-related semantic features through natural language processing (NLP).

[0054] Data alignment: Utilize timestamps to synchronize multimodal data and solve the problem of sensor sampling frequency differences (such as spatiotemporal alignment between high-frequency vibration signal sampling and low-frequency temperature sampling).

[0055] Feature fusion layer:

[0056] Feature extraction:

[0057] Time series data (vibration, current): LSTM or Transformer are used to extract dynamic change features and capture equipment degradation trends.

[0058] Image data (visual inspection): Using convolutional neural networks (CNN) to identify visual features such as part defects and assembly anomalies.

[0059] Acoustic data: Abnormal sound features are extracted using Mel frequency cepstral coefficients (MFCC), and the source of the fault is located by combining the spectrum analysis.

[0060] Text data: The BERT model is used to encode log text into semantic vectors and associate them with historical failure cases.

[0061] Fusion strategy:

[0062] Feature-level fusion: The multimodal feature vectors are concatenated or weighted and summed, and then input into the fully connected layer for preliminary fusion.

[0063] Attention mechanism: Introduce a self-attention module (such as Multi-Head Attention in Transformer) to dynamically allocate weights for features of different modalities and highlight key fault signals.

[0064] Diagnostic decision-making level:

[0065] Classification model: XGBoost or LightGBM is used to classify faults (such as mechanical jamming, sensor failure, and communication interruption) based on the fused features.

[0066] Regression model: Based on LSTM, predict the remaining useful life (RUL) of the equipment and trigger maintenance warnings in advance.

[0067] Knowledge graph enhancement: Construct a knowledge graph of equipment failures in the production line, combine model diagnostic results with expert rules to improve interpretability (e.g., "excessive vibration + sudden temperature rise → bearing wear").

[0068] Multimodal deep learning and edge computing:

[0069] Multimodal pre-trained models:

[0070] Using contrastive learning (such as SimCLR) or self-supervised learning (such as MAE) to pre-train on unlabeled data can solve the problem of scarce labeled data in industrial scenarios.

[0071] Example: Map vibration signals and visual images to the same latent space to learn cross-modal shared features.

[0072] Lightweight model deployment:

[0073] Model pruning and quantization (such as INT8) techniques are used to compress the model size and adapt it to edge computing devices (such as NVIDIA Jetson).

[0074] Real-time optimization: Inference is accelerated through TensorRT to ensure fault diagnosis latency <100ms.

[0075] Dynamic threshold adjustment:

[0076] Dynamic alarm thresholds are set based on statistical process control (SPC) to adapt to normal fluctuations under different operating conditions (such as the impact of ambient temperature changes on sensor readings).

[0077] The user terminal includes a monitoring and verification module, a monitoring module, a configuration module, and a diagnostic module;

[0078] The monitoring and verification module verifies the monitoring scheme to be applied, ensuring that subsequent comprehensive and accurate diagnostic analysis can be performed based on the collected multimodal data to obtain the monitoring scheme. The monitoring scheme refers to the deployment of various monitoring devices on the metering production line, the data collection through these devices, such as the placement of sensors corresponding to data on temperature, pressure, and vibration, and the installation of industrial-grade high-definition cameras next to the metering production line, specifying the installation location and number, etc. The module connects to a cloud-based fault database and verifies the monitoring scheme based on the database, obtaining corresponding verification results. The verification results are either qualified or unqualified, along with the reasons for unqualified. Appropriate processing is performed based on the verification results. For example, the verification results are displayed to the user. If the verification is unqualified, the monitoring scheme is adjusted according to the reasons for unqualified, and then re-verified.

[0079] By setting up a fault database in the cloud, the platform can easily aggregate fault issues and fault characteristics from multiple users, improving resource utilization and work efficiency. This, combined with monitoring and verification modules at each user's location, enables the verification and analysis of user monitoring solutions. Furthermore, since the data analysis process occurs entirely on the user's end, the risk of data leakage is reduced. For example, each user's superior unit can be responsible for cloud-based management and services, facilitating the service and management of its subordinate users.

[0080] In one embodiment, when the metering line undergoes changes or updates, the previously applied monitoring scheme can also be regarded as the monitoring scheme to be applied, that is, it is not limited to the previous monitoring scheme.

[0081] In one embodiment, the monitoring scheme is validated based on a fault database, including:

[0082] A corresponding visualization model can be established for the metrological verification production line using appropriate 3D visualization technologies. Based on the monitoring plan, the corresponding monitoring arrangements can be made within the visualization model to obtain a corresponding monitoring simulation model. This model, after supplementing with relevant information such as monitoring equipment, is labeled as the monitoring simulation model. Through the monitoring simulation model, the monitoring range and data of each monitoring device can be determined. For example, for an industrial-grade high-definition camera at a certain location, the image data of the monitored equipment can be assumed based on the camera's monitoring range. In particular, it can be assumed that the equipment has a certain fault, and the image data when the equipment has this fault can be analyzed.

[0083] Identify various faults in the fault database, and combine them according to the faults that may occur in the actual process to form corresponding fault simulation backgrounds. The fault simulation background is determined according to the corresponding fault combination. For example, if there are faults A, B, and C, A, B, and C may appear individually in the actual process, or they may appear in combination with AB and BC, but the combination faults AC and ABC will not occur. That is, first determine the various fault combinations based on the actual fault situation; form the fault combinations corresponding to A, B, C, AB, and BC, and determine the fault background for simulation according to the fault combinations to form the corresponding fault simulation background.

[0084] Based on the fault combination corresponding to the fault simulation background, the corresponding historical monitoring data is collected to form the monitoring simulation data corresponding to the fault simulation background. The monitoring simulation data is then added to the monitoring simulation model. Monitoring is carried out according to the monitoring method corresponding to the monitoring scheme to obtain the monitoring simulation data of the equipment corresponding to the monitoring method. The monitoring simulation data is then analyzed to determine whether the fault problem can be diagnosed.

[0085] If all faults can be diagnosed, the fault diagnosis requirements are met; otherwise, they are not. The system performs simulation analysis based on the monitoring simulation model to determine whether the model can meet the fault diagnosis requirements under the corresponding fault simulation background. The fault diagnosis requirements are set by the user according to the fault diagnosis needs. For example, under the preset fault simulation background, the monitoring simulation model can accurately identify key information such as fault type, location, and severity through multimodal data, and meet the preset diagnostic accuracy, response time, and reliability indicators.

[0086] When there is a fault simulation background that does not meet the fault diagnosis requirements, the evaluation and monitoring scheme verification is unqualified, and corresponding reasons for non-compliance are generated based on the fault simulation background that does not meet the fault diagnosis requirements, such as the multimodal data collected under the fault simulation background not meeting the diagnostic needs.

[0087] When there is no fault simulation background that does not meet the fault diagnosis requirements, the evaluation and monitoring scheme is deemed qualified.

[0088] In one embodiment, due to the user's actual monitoring needs, some fault issues may not need to be monitored by the user due to monitoring costs, fault impacts, or other reasons. In this case, before generating a fault simulation background based on the fault issues, the fault issues are screened according to the user's monitoring needs, and fault issues that do not meet the user's monitoring needs are removed and do not participate in the generation of the fault simulation background.

[0089] In one embodiment, fault issues are filtered according to the user's monitoring needs. This can be done by analyzing existing demand calibration technologies, such as intelligent models built based on machine learning and deep learning algorithms, or by direct manual filtering. Fault issues that do not meet the user's monitoring needs are then eliminated.

[0090] In one embodiment, simulation analysis based on a monitoring simulation model includes:

[0091] The monitoring simulation model is used to collect and simulate data, which is then used to identify all fault characteristics that can be collected during monitoring, such as all fault characteristics of a single fault or all fault characteristics of multiple faults. The system analyzes the collectable fault characteristics based on the monitoring method and deployment, combining relevant historical monitoring data and equipment performance to determine whether all faults corresponding to the simulated fault background can be identified based on the identified fault characteristics. Finally, the system determines the required fault characteristics for identifying the corresponding faults under the simulated fault background.

[0092] When it is determined that all fault problems corresponding to the fault simulation background can be identified, the fault simulation background is evaluated as meeting the fault diagnosis requirements; this is equivalent to the monitoring scheme meeting the fault monitoring requirements of the fault simulation background.

[0093] If it is determined that not all fault problems corresponding to the fault simulation background can be identified, the fault simulation background is deemed not to meet the fault diagnosis requirements.

[0094] The monitoring module is used to monitor the metrology and calibration production line in real time, obtain corresponding multimodal data, such as integrated data including image data, temperature data, pressure data, and vibration data, and send the multimodal data to the diagnostic module.

[0095] The configuration module is used to perform configuration analysis, obtain the corresponding monitoring simulation model according to the application's monitoring scheme, that is, the monitoring simulation model corresponding to the above-mentioned evaluation monitoring scheme verification is qualified; connect to the cloud-based diagnostic library, and configure the corresponding diagnostic scheme and diagnostic model for the diagnostic module based on the diagnostic library and the monitoring simulation model; subsequently, the diagnostic module can analyze multimodal data according to the diagnostic scheme to determine the corresponding fault diagnosis results.

[0096] In one embodiment, the diagnostic module is configured with corresponding diagnostic schemes and models based on a diagnostic library and a monitoring simulation model, including:

[0097] Obtain the fault simulation background in the above embodiments, generate corresponding multimodal simulation data according to the monitoring simulation model and the fault simulation background, that is, generate corresponding multimodal data according to the various fault performance characteristics and monitoring simulation model corresponding to the fault simulation background, and mark it as multimodal simulation data. It can be a possible dataset to improve the representativeness of the verification. That is, one multimodal simulation data corresponds to one fault simulation background.

[0098] The various multimodal simulation data are integrated into diagnostic simulation data, the diagnostic solutions corresponding to the diagnostic library are identified, and the diagnostic solutions are screened based on the diagnostic simulation data to determine the diagnostic solutions and diagnostic models to be configured.

[0099] Configure the diagnostic plan and diagnostic model in the diagnostic module.

[0100] In one embodiment, diagnostic schemes are screened based on diagnostic simulation data. Diagnostic schemes that cannot be diagnosed based on the diagnostic simulation data are first eliminated, and then the remaining diagnostic schemes are prioritized and the user selects the diagnostic scheme to apply. Prioritization can be based on factors such as diagnostic accuracy, efficiency, and cost.

[0101] For example, each diagnostic solution is evaluated to determine whether it can perform fault diagnosis based on diagnostic simulation data, and diagnostic solutions that cannot perform fault diagnosis based on diagnostic simulation data are eliminated.

[0102] The remaining diagnostic solutions after elimination are prioritized, and the user selects the diagnostic solution to apply based on the ranking.

[0103] If sorting is done solely based on factors such as cost, accuracy, and efficiency, a combined sorting can also be performed accordingly.

[0104] The diagnostic module is used to perform diagnostic analysis, identify the received multimodal data, and analyze the multimodal data through the configured diagnostic scheme and diagnostic model to obtain the corresponding fault diagnosis data.

[0105] The above formulas are all numerical calculations after removing dimensions. The formulas are obtained by software simulation based on a large amount of data and are closest to the real situation. The preset parameters and preset thresholds in the formulas are set by those skilled in the art according to the actual situation or obtained by simulation based on a large amount of data.

[0106] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. An automated diagnostic analysis system based on computer vision and multimodal data fusion, characterized in that, Including cloud and user terminals; The cloud platform includes a fault database and a diagnostic database; the user terminal includes a monitoring and verification module, a monitoring module, a configuration module, and a diagnostic module. The fault database is used to store various faults in the metrology verification line and the corresponding fault characteristics. The diagnostic library is used to store various diagnostic schemes and corresponding diagnostic models for fault diagnosis. The monitoring and verification module verifies the monitoring scheme, obtains the monitoring scheme, connects to the fault database in the cloud, verifies the monitoring scheme based on the fault database, and obtains the corresponding verification results, which are either qualified or unqualified, along with the reasons for unqualification; and performs corresponding processing based on the verification results. The configuration module is used to perform configuration analysis, obtain the corresponding monitoring simulation model according to the application's monitoring scheme, connect to the cloud-based diagnostic library, and configure the corresponding diagnostic scheme and diagnostic model for the diagnostic module based on the diagnostic library and the monitoring simulation model. The monitoring module is used to monitor the metrological verification line in real time, obtain corresponding multimodal data, and send the multimodal data to the diagnostic module. The diagnostic module is used to perform diagnostic analysis, identify the received multimodal data, and analyze the multimodal data through the configured diagnostic scheme and diagnostic model to obtain corresponding fault diagnosis data.

2. The automated diagnostic analysis system based on computer vision and multimodal data fusion according to claim 1, characterized in that, The monitoring scheme is validated based on a fault database, including: A visual model is established for the metrological verification process. According to the monitoring plan, the corresponding monitoring arrangements are carried out in the visual model. The completed visual model is marked as a monitoring simulation model. Based on the fault database, generate fault simulation backgrounds; based on the fault simulation backgrounds, generate monitoring simulation data, and supplement the monitoring simulation model with the monitoring simulation data; Based on the monitoring simulation model, conduct simulation analysis to determine whether the monitoring simulation model can meet the fault diagnosis requirements under the corresponding fault simulation background; When there is a fault simulation background that does not meet the fault diagnosis requirements, the monitoring scheme is evaluated as unqualified, and corresponding reasons for non-compliance are generated based on the fault simulation background that does not meet the fault diagnosis requirements. When there is no fault simulation background that does not meet the fault diagnosis requirements, the monitoring scheme is evaluated and verified as qualified.

3. The automated diagnostic analysis system based on computer vision and multimodal data fusion according to claim 2, characterized in that, The fault simulation backgrounds generated based on the fault database include: Identify various faults in the fault database, and combine them according to whether they can occur simultaneously or independently to obtain several fault combinations; generate corresponding fault simulation backgrounds based on the fault combinations.

4. The automated diagnostic analysis system based on computer vision and multimodal data fusion according to claim 2, characterized in that, The fault simulation backgrounds generated based on the fault database include: Identify various faults in the fault database, obtain user monitoring requirements, filter faults according to monitoring requirements, generate several problem combinations based on the remaining faults after filtering, and generate corresponding fault simulation backgrounds based on the problem combinations.

5. The automated diagnostic analysis system based on computer vision and multimodal data fusion according to claim 2, characterized in that, Simulation analysis is conducted based on the monitoring simulation model, including: The monitoring simulation model is used to obtain monitoring simulation data, and the monitoring simulation data is used to identify all fault performance characteristics that can be collected during monitoring. Based on the identified fault performance characteristics, it is determined whether all fault problems corresponding to the fault simulation background can be identified. When it is determined that all fault problems corresponding to the fault simulation background can be identified, the fault simulation background is evaluated to meet the fault diagnosis requirements. If it is determined that not all fault problems corresponding to the fault simulation background can be identified, the fault simulation background is deemed not to meet the fault diagnosis requirements.

6. The automated diagnostic analysis system based on computer vision and multimodal data fusion according to claim 1, characterized in that, Based on the diagnostic library and monitoring simulation model, configure the corresponding diagnostic schemes and diagnostic models for the diagnostic module, including: Obtain the fault simulation background, and generate corresponding multimodal simulation data for the fault simulation background based on the monitoring simulation model and the fault simulation background; integrate the various multimodal simulation data into diagnostic simulation data; Identify the various diagnostic schemes corresponding to the diagnostic library, filter the diagnostic schemes based on the diagnostic simulation data, and determine the diagnostic schemes and diagnostic models to be configured.

7. The automated diagnostic analysis system based on computer vision and multimodal data fusion according to claim 6, characterized in that, Based on the diagnostic simulation data, various diagnostic options were screened, including: Evaluate whether each diagnostic solution can diagnose faults based on diagnostic simulation data, and eliminate diagnostic solutions that cannot diagnose faults based on diagnostic simulation data. The remaining diagnostic solutions after elimination are prioritized, and users select the configured diagnostic solution according to the priority.

Images