Equipment evaluation method and system based on digital twinning and multi-modal data fusion

Abstract

The invention discloses an equipment evaluation method and system based on digital twinning and multi-modal data fusion, and relates to the technical field of intelligent manufacturing and equipment health management, and the method comprises the steps: carrying out the integration based on data sequences corresponding to different types of monitoring data, and obtaining a multi-type monitoring data fusion sequence; performing feature fusion on the multi-category monitoring data fusion sequence to obtain a corresponding multi-channel tensor, a cross-modal weight and zoom dot product attention; adaptively adjusting health degree index thresholds corresponding to different types of monitoring data; and based on the multi-category monitoring data fusion sequence and the health degree index thresholds corresponding to the different categories of monitoring data, carrying out equipment evaluation. Through multi-source heterogeneous data fusion, a dynamic health assessment model and virtual-real collaborative decision optimization, an efficient and intelligent equipment health management system is constructed, a reliable basis is provided for equipment assessment work, and a guarantee is provided for the real-time performance and the working efficiency of assessment work.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent manufacturing and equipment health management, and specifically to an equipment evaluation method and system based on digital twin and multimodal data fusion. Background Art

[0002] With the deepening of Industry 4.0 and smart manufacturing strategies, equipment health management has become an important part of industrial digital transformation. According to market research data, the application of digital twin technology in China's industrial manufacturing field has grown rapidly. Equipment health management can effectively reduce unplanned downtime losses and improve production efficiency through real-time monitoring and predictive maintenance. However, traditional equipment health assessment systems generally have the following problems: serious data silos, equipment operation data is scattered across multimodal sensors (such as vibration, temperature, current) and business systems, and heterogeneous data is difficult to integrate; health assessment models are static and difficult to capture the timing characteristics of equipment degradation, resulting in a high false alarm rate; digital twin technology mostly remains in the visualization stage, lacks dynamic simulation and decision-making closed-loop support, and predictive maintenance response is delayed.

[0003] The combination of digital twins and artificial intelligence offers a new paradigm for equipment health management. Existing technologies typically perform health assessments based on a single data source or static model, such as using support vector machines (SVMs) or random forest algorithms to analyze equipment status. However, these methods have shortcomings in multimodal data fusion and dynamic modeling. First, multimodal data fusion relies on manual feature engineering, making it difficult to achieve semantic alignment of cross-modal data such as vibration spectra, thermal imaging, and work order records. Second, static twin models cannot update equipment degradation status in real time, resulting in large errors in motor gearbox wear predictions. Third, maintenance decisions often rely on manual experience and fail to dynamically optimize environmental parameters such as spare parts inventory and weather, leading to high operation and maintenance costs.

[0004] Currently, industrial equipment operation and maintenance faces core pain points such as difficulty in dynamic modeling, slow real-time response, and cost control. The maturity of digital twin and multimodal data fusion technologies offers a potential solution to these problems. Furthermore, the national intelligent manufacturing and green development strategy explicitly encourages technological innovation to improve the efficiency of equipment management throughout its lifecycle.

[0005] In response to the urgent needs of industrial practice, the feasibility of technology integration and the clarity of policy orientation, a device evaluation technology based on the fusion of digital twins and multimodal data is proposed. Summary of the Invention

[0006] In response to the defects in the existing technology, the purpose of this application is to provide an equipment evaluation method and system based on digital twin and multimodal data fusion. Through multi-source heterogeneous data fusion, dynamic health assessment model and virtual-reality collaborative decision optimization, an efficient and intelligent equipment health management system is constructed to provide a reliable basis for equipment evaluation work and to ensure the real-time and work efficiency of the evaluation work.

[0007] In order to achieve the above objectives, the technical solution adopted by this application is:

[0008] In a first aspect, the present application provides a device evaluation method based on digital twin and multimodal data fusion, the method comprising the following steps:

[0009] Based on the data sequences corresponding to different categories of monitoring data, a fusion sequence of multi-category monitoring data is obtained by integration;

[0010] Performing feature fusion on the multi-category monitoring data fusion sequence to obtain corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention;

[0011] Adaptively adjusting the health index thresholds corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weights, and the scaled dot product attention;

[0012] Based on the multi-category monitoring data fusion sequence and the health index thresholds corresponding to different categories of monitoring data, equipment evaluation is performed; wherein,

[0013] Different types of monitoring data correspond to different feature dimensions.

[0014] On the basis of the above technical solution, the data sequences corresponding to the monitoring data of different categories are integrated to obtain a fusion sequence of multi-category monitoring data, including the following steps:

[0015] Based on the data sequences corresponding to different categories of monitoring data, a cumulative distance matrix is ​​obtained;

[0016] Based on the cumulative distance matrix, obtaining an optimal alignment path through a state transfer equation;

[0017] Based on the optimal alignment path, spline difference processing is performed to generate the multi-category monitoring data fusion sequence.

[0018] Based on the above technical solution, the feature fusion of the multi-category monitoring data fusion sequence is performed to obtain the corresponding multi-channel tensor, cross-modal weights and scaled dot product attention, including the following steps:

[0019] Based on the multi-category monitoring data fusion sequence, a cross-modal feature matrix is ​​obtained, and features of different categories of monitoring data are spliced ​​to obtain the multi-channel tensor;

[0020] Performing attention-weighted fusion on the cross-modal feature matrix to obtain the cross-modal weight;

[0021] Based on the multi-category monitoring data fusion sequence, linear projection is performed to obtain the scaled dot product attention of different categories of monitoring data.

[0022] On the basis of the above technical solution, the linear projection is performed based on the multi-category monitoring data fusion sequence to obtain the scaled dot product attention of different categories of monitoring data, including the following steps:

[0023] Based on the multi-category monitoring data fusion sequence, linear projection is performed to obtain query, key, and value matrices of monitoring data of different categories;

[0024] Based on the query, key, and value matrices of the monitoring data of different categories, the scaled dot product attention of the monitoring data of different categories is obtained.

[0025] Based on the above technical solution, the method of adaptively adjusting the health index thresholds corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weights, and the scaled dot product attention includes the following steps:

[0026] Based on the multi-category monitoring data fusion sequence, combined with LSTM time series feature extraction, the time series feature information of different categories of monitoring data is obtained;

[0027] Based on the temporal feature information of the different categories of monitoring data, the corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention, combined with the Transformer multi-head self-attention mechanism, the spatial correlation between the different categories of monitoring data is obtained;

[0028] Based on the spatial correlation between different categories of monitoring data, the health index thresholds corresponding to different categories of monitoring data are adaptively adjusted through the fully connected layer.

[0029] In a second aspect, the present application provides an equipment evaluation system based on digital twin and multimodal data fusion, the system comprising:

[0030] A data integration module is used to integrate the data sequences corresponding to different categories of monitoring data to obtain a fusion sequence of multi-category monitoring data;

[0031] A data analysis module, which is used to perform feature fusion on the multi-category monitoring data fusion sequence to obtain corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention;

[0032] A threshold adjustment module, which is used to adaptively adjust the health index threshold corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weight and the scaled dot product attention;

[0033] A health assessment module is used to perform equipment assessment based on the multi-category monitoring data fusion sequence and the health index thresholds corresponding to different categories of monitoring data; wherein,

[0034] Different types of monitoring data correspond to different feature dimensions.

[0035] On the basis of the above technical solution, the data integration module is further used to obtain a cumulative distance matrix based on the data sequences corresponding to different categories of monitoring data;

[0036] The data integration module is further configured to obtain an optimal alignment path through a state transition equation based on the cumulative distance matrix;

[0037] The data integration module is further configured to perform spline difference processing based on the optimal alignment path to generate the multi-category monitoring data fusion sequence.

[0038] On the basis of the above technical solution, the data analysis module is further used to obtain a cross-modal feature matrix based on the multi-category monitoring data fusion sequence, and to splice the features of the monitoring data of different categories to obtain the multi-channel tensor;

[0039] The data analysis module is further configured to perform attention-weighted fusion on the cross-modal feature matrix to obtain the cross-modal weight;

[0040] The data analysis module is also used to perform linear projection based on the multi-category monitoring data fusion sequence to obtain the scaled dot product attention of different categories of monitoring data.

[0041] On the basis of the above technical solution, the data analysis module is further used to perform linear projection based on the multi-category monitoring data fusion sequence to obtain query, key, and value matrices of monitoring data of different categories;

[0042] The data analysis module is also used to obtain the scaled dot product attention of different categories of monitoring data based on the query, key, and value matrices of different categories of monitoring data.

[0043] On the basis of the above technical solution, the threshold adjustment module is further used to obtain time series feature information of monitoring data of different categories based on the multi-category monitoring data fusion sequence and combined with LSTM time series feature extraction;

[0044] The threshold adjustment module is further configured to obtain spatial correlation between different categories of monitoring data based on the temporal feature information of the different categories of monitoring data, the corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention, in combination with the Transformer multi-head self-attention mechanism;

[0045] The threshold adjustment module is further used to adaptively adjust the health index thresholds corresponding to different categories of monitoring data through a fully connected layer based on the spatial correlation between different categories of monitoring data.

[0046] Compared with the prior art, the advantages of this application are:

[0047] This application builds an efficient and intelligent equipment health management system through multi-source heterogeneous data fusion, dynamic health assessment model and virtual-reality collaborative decision optimization, providing a reliable basis for equipment assessment work and ensuring the real-time and work efficiency of the assessment work. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 This is a flowchart of the steps of the device evaluation method based on digital twin and multimodal data fusion according to an embodiment of the present application;

[0050] Figure 2 This is a principle framework diagram of the device evaluation method based on digital twin and multimodal data fusion in an embodiment of the present application;

[0051] Figure 3 This is a principle framework diagram of multi-source heterogeneous data fusion in the device evaluation method based on digital twin and multimodal data fusion in an embodiment of the present application;

[0052] Figure 4 This is a principle framework diagram of dynamic health assessment in the device assessment method based on digital twin and multimodal data fusion in an embodiment of the present application;

[0053] Figure 5 This is a principle framework diagram of virtual-reality collaborative decision optimization in the equipment evaluation method based on digital twin and multimodal data fusion in an embodiment of the present application;

[0054] Figure 6 This is a structural block diagram of the equipment evaluation system based on digital twin and multimodal data fusion in an embodiment of the present application. DETAILED DESCRIPTION

[0055] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] The embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0057] The embodiments of the present application provide an equipment evaluation method and system based on digital twin and multimodal data fusion. Through multi-source heterogeneous data fusion, dynamic health assessment model and virtual-reality collaborative decision optimization, an efficient and intelligent equipment health management system is constructed, providing a reliable basis for equipment evaluation work and ensuring the real-time and work efficiency of the evaluation work.

[0058] To achieve the above technical effects, the overall idea of ​​this application is as follows:

[0059] A device evaluation method based on digital twin and multimodal data fusion, the method comprising the following steps:

[0060] S1. Based on the data sequences corresponding to different categories of monitoring data, integrate and obtain a fusion sequence of multi-category monitoring data;

[0061] S2. Perform feature fusion on the multi-category monitoring data fusion sequence to obtain the corresponding multi-channel tensor, cross-modal weights, and scaled dot product attention;

[0062] S3, based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weight and the scaled dot product attention, adaptively adjusts the health index threshold corresponding to different categories of monitoring data;

[0063] S4. Perform equipment evaluation based on the fusion sequence of multi-category monitoring data and the health index thresholds corresponding to different categories of monitoring data;

[0064] Different types of monitoring data correspond to different feature dimensions.

[0065] The embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0066] First, see Figures 1 to 5 As shown, the embodiment of the present application provides a device evaluation method based on digital twin and multimodal data fusion, which includes the following steps:

[0067] S1. Based on the data sequences corresponding to different categories of monitoring data, integrate and obtain a fusion sequence of multi-category monitoring data;

[0068] S2. Perform feature fusion on the multi-category monitoring data fusion sequence to obtain the corresponding multi-channel tensor, cross-modal weights, and scaled dot product attention;

[0069] S3, based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weight and the scaled dot product attention, adaptively adjusts the health index threshold corresponding to different categories of monitoring data;

[0070] S4. Perform equipment evaluation based on the fusion sequence of multi-category monitoring data and the health index thresholds corresponding to different categories of monitoring data;

[0071] Different types of monitoring data correspond to different feature dimensions.

[0072] In summary, the core process of the embodiment of this application is as follows:

[0073] Collect multi-source heterogeneous data and perform semantic mapping and spatiotemporal alignment;

[0074] Build an LSTM-Transformer model to dynamically evaluate device health status;

[0075] Optimizing maintenance decisions through reinforcement learning and digital twin simulation.

[0076] Multimodal data, that is, different categories of monitoring data, including vibration, temperature, current and image data corresponding to the equipment to be evaluated.

[0077] It should be noted that the embodiments of the present application are applicable to dynamic modeling, real-time monitoring and predictive maintenance optimization in the full life cycle management of industrial equipment;

[0078] The core of the technical solution of the embodiment of this application is to overcome the following technical difficulties:

[0079] (1) It is difficult to efficiently fuse multi-source heterogeneous data, resulting in low feature extraction efficiency and limited health assessment accuracy;

[0080] (2) Traditional health assessment models cannot dynamically capture equipment degradation trends, and have high false positive and false negative rates;

[0081] (3) The digital twin is separated from the decision-making system, lacks the ability to optimize virtual-reality collaboration, and results in delayed maintenance responses and high comprehensive operation and maintenance costs.

[0082] In the embodiment of the present application, an efficient and intelligent equipment health management system is constructed through the fusion of multi-source heterogeneous data, dynamic health assessment model and virtual-real collaborative decision optimization, which provides a reliable basis for equipment assessment work and guarantees the real-time and work efficiency of the assessment work;

[0083] Through the integrated innovation of multimodal data fusion, dynamic health assessment and virtual-reality collaborative decision-making, the problems of data silos, model staticization and decision delay in traditional equipment health management have been solved, providing an efficient and intelligent technical solution for the full life cycle management of industrial equipment, with significant application value.

[0084] Furthermore, the data sequences corresponding to the monitoring data of different categories are integrated to obtain a fusion sequence of multi-category monitoring data, including the following steps:

[0085] Based on the data sequences corresponding to different categories of monitoring data, a cumulative distance matrix is ​​obtained;

[0086] Based on the cumulative distance matrix, obtaining an optimal alignment path through a state transfer equation;

[0087] Based on the optimal alignment path, spline difference processing is performed to generate the multi-category monitoring data fusion sequence.

[0088] Furthermore, the feature fusion of the multi-category monitoring data fusion sequence is performed to obtain corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention, including the following steps:

[0089] Based on the multi-category monitoring data fusion sequence, a cross-modal feature matrix is ​​obtained, and features of different categories of monitoring data are spliced ​​to obtain the multi-channel tensor;

[0090] Performing attention-weighted fusion on the cross-modal feature matrix to obtain the cross-modal weight;

[0091] Based on the multi-category monitoring data fusion sequence, linear projection is performed to obtain the scaled dot product attention of different categories of monitoring data.

[0092] Furthermore, performing linear projection based on the multi-category monitoring data fusion sequence to obtain the scaled dot product attention of different categories of monitoring data includes the following steps:

[0093] Based on the multi-category monitoring data fusion sequence, linear projection is performed to obtain query, key, and value matrices of monitoring data of different categories;

[0094] Based on the query, key, and value matrices of the monitoring data of different categories, the scaled dot product attention of the monitoring data of different categories is obtained.

[0095] Furthermore, the adaptive adjustment of the health index thresholds corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weights, and the scaled dot product attention includes the following steps:

[0096] Based on the multi-category monitoring data fusion sequence, combined with LSTM time series feature extraction, the time series feature information of different categories of monitoring data is obtained;

[0097] Based on the temporal feature information of the different categories of monitoring data, the corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention, combined with the Transformer multi-head self-attention mechanism, the spatial correlation between the different categories of monitoring data is obtained;

[0098] Based on the spatial correlation between different categories of monitoring data, the health index thresholds corresponding to different categories of monitoring data are adaptively adjusted through the fully connected layer.

[0099] Based on the technical solution of the embodiment of this application, a device assessment technology based on digital twin and multimodal data fusion is provided. Through the three core technologies of multi-source heterogeneous data fusion, dynamic health assessment model, and virtual-reality collaborative decision optimization, an efficient and intelligent device health management solution is constructed. The layered architecture includes a data acquisition layer, a data fusion layer, a health assessment layer, and a decision optimization layer. The edge computing nodes work in collaboration with the cloud to ensure real-time performance and computing efficiency. The following is a detailed description of the technical solution:

[0100] First, multi-source heterogeneous data fusion:

[0101] The semantic model of data is established using JSON-LD (JSON for Linking Data) semantic mapping rules, and the time series of heterogeneous data are aligned through the Dynamic Time Warping (DTW) algorithm to eliminate differences in sampling frequency and timestamps.

[0102] The specific operations of multi-source heterogeneous data fusion are as follows:

[0103] (1) JSON-LD semantic mapping rules:

[0104] Solve the semantic heterogeneity problem of cross-modal data and unify the label definitions of vibration, temperature, image and other data.

[0105] Context Mapping: Establishes a mapping relationship between data fields and standardized semantics to resolve semantic ambiguity between different systems. Using predefined rules, local data labels (such as "vibration frequency") are mapped to a globally unique standardized terminology (such as the Schema.org industrial sensor classification), clarifying the physical meaning, units, and data type of the field.

[0106] Entity association: Use entity type annotation (@type) to label type tags for entities such as devices, sensors, and events, declare the type of data entities, and clarify their role in the knowledge system. At the same time, assign a globally unique identifier (@id) to the entity to classify and associate cross-modal data, build an association network for cross-modal data, form a "device-subcomponent" topological relationship, and support hierarchical association of devices, sensors, and data.

[0107] (2) Space-time alignment:

[0108] Taking the difference in sampling rates between vibration signals (10kHz) and infrared images (0.5Hz) that leads to spatiotemporal misalignment as an example, local distance calculation is performed:

[0109] Define the vibration signal sequence X=[X1,X2,…,X m ] and the image temperature sequence Y=[Y1,Y2,…,Y n ]’s Euclidean distance matrix D:

[0110] The vibration signal is preprocessed by wavelet denoising, and the image temperature is extracted by thermal imager pixel clustering.

[0111] Dynamic programming path optimization: Calculate the cumulative distance matrix D and find the optimal alignment path W through the state transition equation. The formula is as follows:

[0112]

[0113] Multimodal data interpolation: Based on the alignment path W, cubic spline interpolation is performed on the low-frequency image data to generate a time-series temperature series synchronized with the vibration signal;

[0114] Among them, in the above operation, when there is missing data, the default data change is linear change, and the missing intermediate data is supplemented based on the law of linear change.

[0115] (3) Feature fusion:

[0116] Cross-modal feature matrix: The aligned vibration spectrum (time domain), temperature gradient (spatial domain), and image texture (frequency domain) features are concatenated into a multi-channel tensor. The formula is as follows:

[0117] F=[F vib ||F temp ||F img ]∈R T×d ;in,

[0118] T is the time step and d is the feature dimension.

[0119] Attention-weighted fusion: Calculate cross-modal weights through the Transformer multi-head attention mechanism.

[0120] Input and linear transformation:

[0121] First enter the sequence: (n is the sequence length, d model is the model dimension).

[0122] Then perform linear projection: for each head h (a total of H heads), generate an independent query (Query), key (Key), and value (Value) matrix. The formula is as follows:

[0123] in,

[0124]

[0125] It should be noted that each head, that is, each different category of monitoring data, is monitoring data of different dimensions.

[0126] Then, each head computes the scaled dot-product attention independently.

[0127] in,

[0128] Scaling factor d k Prevent the dot product result from being too large and causing the gradient to disappear.

[0129] Second, dynamic health assessment:

[0130] (1) Building an LSTM-Transformer hybrid architecture:

[0131] The long short-term memory network (LSTM) is used to extract temporal features, and the multi-head self-attention mechanism of Transformer is combined to analyze the spatial correlation of cross-modal features and adaptively adjust the health index threshold.

[0132] LSTM Time Series Feature Extraction: Utilizing bidirectional LSTM modules, this approach captures long-term dependencies in device degradation characteristics, such as the vibration spectrum shift trend associated with bearing wear. Each LSTM unit dynamically updates its state through a gating mechanism (forget gate, input gate, and output gate) to preserve key time series information. This generates a hidden state vector containing the time series degradation trend, which serves as the input feature for the Transformer.

[0133] Transformer multi-head self-attention mechanism: Cross-modal correlation analysis uses the time series features output by the LSTM along with cross-modal data such as images and temperature as query, key, and value inputs. Multi-head self-attention is used to calculate spatial correlations between different modalities. Spatial correlations are reflected in impact weights. For example, the correlation weight between periods of abnormal vibration and high-temperature areas in thermal imaging is increased. A health index is generated through a fully connected layer, and the threshold is dynamically adjusted.

[0134] (2) Model compression:

[0135] TensorFlow Lite quantization technology is used to compress the LSTM-Transformer hybrid model, reducing model size and inference latency, and adapting it to edge device deployment.

[0136] INT8 quantization: Maps FP32 weights to the INT8 range, preserving key parameter accuracy.

[0137] Calibration optimization: Calibrate the dynamic range of the activation layer using device historical data to reduce quantization errors.

[0138] TensorFlow Lite conversion: Use TFLiteConverter to optimize the model structure and enable the OPTIMIZE_FOR_LATENCY option to adapt to industrial computers.

[0139] (3) Incremental learning:

[0140] Model parameters are updated regularly through a sliding window to dynamically adapt to equipment degradation and changes in operating conditions.

[0141] Sliding window design: A window length of 60 sampling points (covering the typical equipment operating cycle) with a step size of 10 points ensures continuous capture of degradation features. Data filtering removes noise data (such as abnormal jumps in vibration signals) and retains valid degradation trend segments.

[0142] Online fine-tuning: Update LSTM-Transformer parameters with new data every 24 hours, and use a weighted loss function.

[0143] Knowledge distillation: The full-precision model on the cloud guides the update of the quantized model on the edge, reducing the accuracy loss in incremental learning.

[0144] Dynamic threshold adjustment: When the slope of the equipment degradation curve changes abnormally, such as a sudden change in bearing wear rate, the model is forced to update.

[0145] New fault mode detection: When an unlabeled fault type is found, such as high-frequency harmonics in the current waveform, incremental training is triggered.

[0146] Third, virtual-reality collaborative decision-making optimization:

[0147] (1) Digital twin environment modeling:

[0148] Geometric-physical joint modeling: Build three-dimensional equipment structures (such as gearboxes and circuit boards) based on BIM parametric tools, and integrate the ANSYS simulation engine to simulate physical properties such as thermal stress and mechanical fatigue.

[0149] Dynamic update of behavioral models: Use the LSTM network to predict equipment degradation curves (such as bearing wear rate) in real time and regularly update the status of the digital twin.

[0150] (2) Multi-source parameter input:

[0151] Health index: Output by the LSTM-Transformer hybrid model (0-1 range), with the threshold dynamically adjusted (for example, if it is < 0.6 for three consecutive cycles, an alert is triggered).

[0152] Environmental parameters: including spare parts inventory status (such as bearing inventory), weather conditions (such as wind speed affecting maintenance windows), and equipment priority (such as priority maintenance of key equipment on the production line).

[0153] (3) Double Q network generates Pareto solution set:

[0154] Dual-Q network architecture: The Q1 network evaluates the immediate cost of maintenance actions (such as spare parts consumption and labor costs), and the Q2 network predicts long-term benefits (such as remaining equipment life and downtime compression rate).

[0155] Alternating update mechanism: Q1 selects the action (such as "immediate repair" or "delay until the next shift"), and Q2 updates the value to avoid overestimation of benefits by a single network.

[0156] Pareto optimal solution screening: Generate multiple candidate strategies (e.g., 10 maintenance paths) through simulation, calculate the cost, time, and risk indicators for each strategy, and screen mutually non-dominated solution sets based on non-dominated sorting to form a Pareto frontier.

[0157] (4) Output the optimal maintenance plan

[0158] Virtual-reality collaborative verification: Simulate maintenance plans in a digital twin environment, such as changes in the equipment vibration spectrum after bearing replacement, to verify feasibility and predict actual effects.

[0159] Dynamic weight decision-making: Adjust target weights based on business needs, such as focusing on response time for emergency tasks and cost for routine maintenance, and select the optimal solution from the Pareto solution set.

[0160] In specific implementation, take industrial motors as an example:

[0161] Install vibration sensors (100Hz), temperature sensors, and current monitoring equipment to simultaneously collect data and obtain work order logs from the MES system;

[0162] The data is fused by JSON-LD and DTW algorithm to generate a feature matrix;

[0163] Input the LSTM-Transformer model and output the health index and remaining useful life (RUL) of gear wear;

[0164] The digital twin model combines spare parts inventory and weather data to generate maintenance plans through the dual-Q network. For example, if the gear is to be replaced in three days, spare parts will be allocated from warehouse A.

[0165] In summary, the technical solutions of the embodiments of the present application have the following technical advantages:

[0166] (1) Efficient data fusion: This paper proposes semantic and spatiotemporal alignment of multimodal data based on JSON-LD and DTW algorithm, supports the unification of spatiotemporal labels of heterogeneous data such as vibration, temperature, and images, and improves the efficiency of feature extraction.

[0167] (2) High-precision evaluation: The LSTM parameters are updated through the LSTM-Transformer model and sliding window mechanism to adapt to the dynamics of equipment degradation, significantly reduce the false alarm rate, and improve the accuracy of fault prediction.

[0168] (3) Intelligent Decision-Making: Digital twins drive the “simulation-decision-execution” cycle. They implement adaptive learning strategies, dynamically adjust batch size and learning rate, and maintain prediction accuracy in non-steady-state data streams. Virtual-reality collaborative optimization shortens maintenance response time and reduces overall operation and maintenance costs.

[0169] (4) Strong universality: supports a variety of industrial equipment (such as motors, gearboxes, fans), and can be deployed at the edge or expanded in the cloud.

[0170] Second, see Figure 6 As shown, an embodiment of the present application provides an equipment evaluation system based on digital twin and multimodal data fusion, the system comprising:

[0171] A data integration module is used to integrate the data sequences corresponding to different categories of monitoring data to obtain a fusion sequence of multi-category monitoring data;

[0172] A data analysis module, which is used to perform feature fusion on the multi-category monitoring data fusion sequence to obtain corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention;

[0173] A threshold adjustment module, which is used to adaptively adjust the health index threshold corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weight and the scaled dot product attention;

[0174] A health assessment module is used to perform equipment assessment based on the multi-category monitoring data fusion sequence and the health index thresholds corresponding to different categories of monitoring data; wherein,

[0175] Different types of monitoring data correspond to different feature dimensions.

[0176] In summary, the core technical operations of the embodiments of this application are as follows:

[0177] Data integration module, used to achieve spatiotemporal alignment of multimodal data through JSON-LD semantic mapping and DTW algorithm;

[0178] The threshold adjustment module uses an LSTM-Transformer hybrid architecture to generate the device health index;

[0179] The health assessment module integrates reinforcement learning and digital twin simulation engine to generate the optimal maintenance path.

[0180] In the embodiments of the present application, an efficient and intelligent equipment health management system is constructed through the fusion of multi-source heterogeneous data, dynamic health assessment model and virtual-reality collaborative decision optimization, which provides a reliable basis for equipment assessment work and guarantees the real-time and work efficiency of the assessment work.

[0181] Furthermore, the data integration module is further used to obtain a cumulative distance matrix based on data sequences corresponding to different categories of monitoring data;

[0182] The data integration module is further configured to obtain an optimal alignment path through a state transition equation based on the cumulative distance matrix;

[0183] The data integration module is further configured to perform spline difference processing based on the optimal alignment path to generate the multi-category monitoring data fusion sequence.

[0184] Furthermore, the data analysis module is further configured to obtain a cross-modal feature matrix based on the multi-category monitoring data fusion sequence, and to concatenate features of monitoring data of different categories to obtain the multi-channel tensor;

[0185] The data analysis module is further configured to perform attention-weighted fusion on the cross-modal feature matrix to obtain the cross-modal weight;

[0186] The data analysis module is also used to perform linear projection based on the multi-category monitoring data fusion sequence to obtain the scaled dot product attention of different categories of monitoring data.

[0187] Furthermore, the data analysis module is further configured to perform linear projection based on the multi-category monitoring data fusion sequence to obtain query, key, and value matrices of monitoring data of different categories;

[0188] The data analysis module is also used to obtain the scaled dot product attention of different categories of monitoring data based on the query, key, and value matrices of different categories of monitoring data.

[0189] Furthermore, the threshold adjustment module is further used to obtain time series feature information of monitoring data of different categories based on the multi-category monitoring data fusion sequence and in combination with LSTM time series feature extraction;

[0190] The threshold adjustment module is further configured to obtain spatial correlation between different categories of monitoring data based on the temporal feature information of the different categories of monitoring data, the corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention, in combination with the Transformer multi-head self-attention mechanism;

[0191] The threshold adjustment module is further used to adaptively adjust the health index thresholds corresponding to different categories of monitoring data through a fully connected layer based on the spatial correlation between different categories of monitoring data.

[0192] To sum up, the equipment evaluation system based on digital twin and multimodal data fusion provided in the embodiment of the present application has the same technical principles as the equipment evaluation method based on digital twin and multimodal data fusion provided in the first aspect in terms of technical problems, technical solutions and technical effects, so they will not be elaborated here.

[0193] In the description of this application, it should be noted that the terms "upper" and "lower" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be internal communication between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0194] It should be noted that, in this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0195] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A device evaluation method based on digital twin and multimodal data fusion, characterized in that: The method comprises the following steps: Based on the data sequences corresponding to different categories of monitoring data, a fusion sequence of multi-category monitoring data is obtained by integration; Performing feature fusion on the multi-category monitoring data fusion sequence to obtain corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention; Adaptively adjusting the health index thresholds corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weights, and the scaled dot product attention; Based on the multi-category monitoring data fusion sequence and the health index thresholds corresponding to different categories of monitoring data, equipment evaluation is performed; wherein, Different types of monitoring data correspond to different feature dimensions.

2. The device evaluation method based on digital twin and multimodal data fusion according to claim 1, characterized in that: The data sequences corresponding to the monitoring data of different categories are integrated to obtain a fusion sequence of multi-category monitoring data, including the following steps: Based on the data sequences corresponding to different categories of monitoring data, a cumulative distance matrix is ​​obtained; Based on the cumulative distance matrix, obtaining an optimal alignment path through a state transfer equation; Based on the optimal alignment path, spline difference processing is performed to generate the multi-category monitoring data fusion sequence.

3. The device evaluation method based on digital twin and multimodal data fusion according to claim 1, characterized in that: The step of performing feature fusion on the multi-category monitoring data fusion sequence to obtain corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention includes the following steps: Based on the multi-category monitoring data fusion sequence, a cross-modal feature matrix is ​​obtained, and features of different categories of monitoring data are spliced ​​to obtain the multi-channel tensor; Performing attention-weighted fusion on the cross-modal feature matrix to obtain the cross-modal weight; Based on the multi-category monitoring data fusion sequence, linear projection is performed to obtain the scaled dot product attention of different categories of monitoring data.

4. The device evaluation method based on digital twin and multimodal data fusion according to claim 3 is characterized in that: The method of performing linear projection based on the multi-category monitoring data fusion sequence to obtain the scaled dot product attention of different categories of monitoring data includes the following steps: Based on the multi-category monitoring data fusion sequence, linear projection is performed to obtain query, key, and value matrices of monitoring data of different categories; Based on the query, key, and value matrices of the monitoring data of different categories, the scaled dot product attention of the monitoring data of different categories is obtained.

5. The device evaluation method based on digital twin and multimodal data fusion according to claim 1, characterized in that: Adaptively adjusting the health index thresholds corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weights, and the scaled dot product attention includes the following steps: Based on the multi-category monitoring data fusion sequence, combined with LSTM time series feature extraction, the time series feature information of different categories of monitoring data is obtained; Based on the temporal feature information of the different categories of monitoring data, the corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention, combined with the Transformer multi-head self-attention mechanism, the spatial correlation between the different categories of monitoring data is obtained; Based on the spatial correlation between different categories of monitoring data, the health index thresholds corresponding to different categories of monitoring data are adaptively adjusted through the fully connected layer.

6. An equipment evaluation system based on digital twin and multimodal data fusion, characterized in that: The system comprises: A data integration module is used to integrate the data sequences corresponding to different categories of monitoring data to obtain a fusion sequence of multi-category monitoring data; A data analysis module, which is used to perform feature fusion on the multi-category monitoring data fusion sequence to obtain corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention; A threshold adjustment module, which is used to adaptively adjust the health index threshold corresponding to different categories of monitoring data based on the multi-category monitoring data fusion sequence, the corresponding multi-channel tensor, the cross-modal weight and the scaled dot product attention; A health assessment module is used to perform equipment assessment based on the multi-category monitoring data fusion sequence and the health index thresholds corresponding to different categories of monitoring data; wherein, Different types of monitoring data correspond to different feature dimensions.

7. The equipment evaluation system based on digital twin and multimodal data fusion according to claim 6, characterized in that: The data integration module is further used to obtain a cumulative distance matrix based on data sequences corresponding to different categories of monitoring data; The data integration module is further configured to obtain an optimal alignment path through a state transition equation based on the cumulative distance matrix; The data integration module is further configured to perform spline difference processing based on the optimal alignment path to generate the multi-category monitoring data fusion sequence.

8. The equipment evaluation system based on digital twin and multimodal data fusion according to claim 6, characterized in that: The data analysis module is further configured to obtain a cross-modal feature matrix based on the multi-category monitoring data fusion sequence, and to concatenate features of monitoring data of different categories to obtain the multi-channel tensor; The data analysis module is further configured to perform attention-weighted fusion on the cross-modal feature matrix to obtain the cross-modal weight; The data analysis module is also used to perform linear projection based on the multi-category monitoring data fusion sequence to obtain the scaled dot product attention of different categories of monitoring data.

9. The equipment evaluation system based on digital twin and multimodal data fusion according to claim 8, characterized in that: The data analysis module is further configured to perform linear projection based on the multi-category monitoring data fusion sequence to obtain query, key, and value matrices of monitoring data of different categories; The data analysis module is also used to obtain the scaled dot product attention of different categories of monitoring data based on the query, key, and value matrices of different categories of monitoring data.

10. The equipment evaluation system based on digital twin and multimodal data fusion according to claim 6, characterized in that: The threshold adjustment module is further used to obtain time series feature information of monitoring data of different categories based on the multi-category monitoring data fusion sequence and in combination with LSTM time series feature extraction; The threshold adjustment module is further configured to obtain spatial correlation between different categories of monitoring data based on the temporal feature information of the different categories of monitoring data, the corresponding multi-channel tensors, cross-modal weights, and scaled dot product attention, in combination with the Transformer multi-head self-attention mechanism; The threshold adjustment module is further used to adaptively adjust the health index thresholds corresponding to different categories of monitoring data through a fully connected layer based on the spatial correlation between different categories of monitoring data.

Images