Analysis method and device based on multi-modal time series data, equipment and medium

Abstract

The application relates to the technical field of data processing, and discloses a kind of analysis method, device, equipment and medium based on multi-modal time series data, comprising: collecting multi-source heterogeneous data and pre-processing and time synchronization, generating uniform format multi-modal time series data stream;Domain knowledge graph is constructed and knowledge big model is established;Each single mode feature is extracted and fused to generate a unified multi-modal time series feature vector;Analysis is carried out using time series big model combined with knowledge big model to generate analysis results;Show analysis results and generate early warning information;Decision suggestions are generated in combination with analysis results and knowledge big model.The application introduces the domain knowledge provided by the knowledge big model in the time series big model analysis process, so that the model has causal reasoning ability when analyzing multi-modal time series data, and can improve the accuracy of anomaly recognition while positioning the cause of the fault and intelligently outputing maintenance suggestions.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an analysis method, apparatus, device, and storage medium based on multimodal time series data. Background Technology

[0002] In the process of industrial manufacturing evolving towards intelligent manufacturing, real-time monitoring and fault diagnosis of equipment operating status have become crucial means to ensure product quality, improve production efficiency, and reduce maintenance costs. Traditional statistical process control (SPC) methods, as well as diagnostic methods based on machine learning and fundamental deep learning models, have been widely applied in industrial monitoring systems. However, faced with the increasingly complex operating environments in current industrial systems and the multi-source heterogeneity of equipment operating data, existing methods still have significant shortcomings in terms of diagnostic accuracy, adaptability, and knowledge utilization.

[0003] Current anomaly detection solutions largely rely on modeling and analyzing single-modal data, such as fault identification based solely on a single type of signal like vibration, temperature, or visual images. These methods typically employ models like support vector machines, autoencoders, or isolated forests, analyzing temporal or statistical features to determine the state. However, due to a lack of multi-source information fusion capabilities, single-modal data struggles to cover the complete operational state of equipment, especially in identifying early, weak faults caused by multi-factor coupling, often resulting in missed detections and false alarms. Furthermore, these methods heavily depend on manual feature extraction or static feature representation, leading to poor model generalization ability and difficulty adapting to changing operating conditions or novel fault modes, particularly exhibiting instability in scenarios with weak signals and low signal-to-noise ratios.

[0004] Furthermore, anomaly detection technologies for image data have also seen some development. Some systems use computer vision algorithms or basic convolutional neural networks to identify surface defects or detect appearance anomalies in industrial product images. However, static image analysis struggles to capture the evolution of equipment status over time and lacks the ability to understand process dynamics, resulting in limited effectiveness in identifying time-driven faults. Simultaneously, in complex industrial environments, image information is easily affected by factors such as lighting, occlusion, and background noise. Traditional models lack robustness and interpretability to image quality fluctuations, making it difficult to effectively support in-depth diagnostic reasoning.

[0005] Some emerging research has begun to explore the use of multimodal data for fusion modeling, such as combining sensor and image data for anomaly identification and introducing language models for auxiliary classification. However, most of these methods are still limited to the fusion of static features and shallow semantic label matching, and have not fully considered the dependencies between multimodal data over time. In real industrial environments, equipment failures often exhibit significant temporal evolution characteristics, and their diagnosis depends not only on the current state but also on a comprehensive analysis of the historical states before and after the anomaly. Therefore, existing methods are still insufficient in handling complex time-dependent multimodal data.

[0006] More critically, most current monitoring and diagnostic models have not yet established a systematic and structured knowledge representation mechanism. Although some methods attempt to use knowledge graphs and other forms for causal explanation during the diagnostic process, they usually remain at the level of static association, lacking dynamic reasoning capabilities and contextual semantic understanding, and thus failing to support the layer-by-layer tracing of fault root causes and the comprehensive generation of decision support. This limitation in knowledge utilization severely restricts the expansion of industrial intelligent diagnostic systems from "prediction" to "explanation" and "recommendation." These problems are particularly prominent in demanding manufacturing scenarios, becoming a core bottleneck affecting diagnostic efficiency and reliability. Summary of the Invention

[0007] The main objective of this invention is to provide an analysis method, apparatus, device, and storage medium based on multimodal time series data, aiming to solve the technical problems in existing industrial process monitoring technologies, such as insufficient fusion of multimodal data and lack of dynamic integration of domain knowledge into time series analysis, resulting in low accuracy of anomaly diagnosis and a lack of interpretable decision support.

[0008] To achieve the above objectives, the present invention provides an analysis method based on multimodal time series data, comprising: Collect multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data. Organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream with a unified format. Construct a domain knowledge graph, and build a large knowledge model based on the domain knowledge graph; Extract each single-modal feature from the multimodal time-series data stream, and fuse the single-modal features to generate a unified multimodal time-series feature vector; The unified multimodal temporal feature vector is analyzed using a time series big model, and the domain knowledge provided by the knowledge big model is incorporated into the analysis process of the time series big model to generate analysis results. Display the analysis results and generate early warning information based on the analysis results; Based on the analysis results and the knowledge model, decision recommendations are generated.

[0009] Furthermore, to achieve the above objectives, the present invention provides an analysis apparatus based on multimodal time series data, comprising: The data acquisition and synchronization module is used to acquire multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data, and organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream in a unified format. The knowledge graph construction and modeling module is used to construct a domain knowledge graph and build a large knowledge model based on the domain knowledge graph. The feature extraction and fusion module is used to extract each single-modal feature from the multimodal time-series data stream and fuse the single-modal features to generate a unified multimodal time-series feature vector; The knowledge-enhanced time series analysis module is used to analyze the unified multimodal time series feature vector through a large time series model, and to integrate the domain knowledge provided by the large knowledge model into the analysis process of the large time series model to generate analysis results; The display and early warning module is used to display the analysis results and generate early warning information based on the analysis results; The decision generation module is used to combine the analysis results with the knowledge model to generate decision recommendations.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an analysis program based on multimodal time series data stored in the memory and executable on the processor, wherein when the analysis program based on multimodal time series data is executed by the processor, it implements the steps of the analysis method based on multimodal time series data as described above.

[0011] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an analysis program based on multimodal time series data, wherein the analysis program based on multimodal time series data, when executed by a processor, implements the steps of the analysis method based on multimodal time series data as described above.

[0012] Beneficial Effects: This invention relates to the field of data processing technology and discloses an analysis method, apparatus, device, and medium based on multimodal time-series data. The method includes: collecting multi-source heterogeneous data, preprocessing and synchronizing it in time to generate a unified format multimodal time-series data stream; constructing a domain knowledge graph and establishing a large knowledge model; extracting and fusing features from each single modality to generate a unified multimodal time-series feature vector; analyzing the data using the large time-series model combined with the large knowledge model to generate analysis results; displaying the analysis results and generating early warning information; and generating decision suggestions by combining the analysis results with the large knowledge model. This invention introduces domain knowledge provided by the large knowledge model into the analysis process of the large time-series model, enabling the model to possess causal reasoning capabilities when performing multimodal time-series data analysis. This achieves semantic association between anomaly detection results and domain knowledge. Combined with visualization results and a decision generation process driven by early warning levels, it can improve the accuracy of anomaly identification while intelligently outputting fault location and maintenance suggestions. Attached Figure Description

[0013] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for an analysis method based on multimodal time series data according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the analysis method based on multimodal time series data according to the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the analysis device based on multimodal time series data of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0014] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0015] The analysis method based on multimodal time series data provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can collect multi-source heterogeneous data from the user terminal, perform preprocessing and time synchronization to generate a unified format multimodal time-series data stream; construct a domain knowledge graph and establish a large knowledge model; extract and fuse features from each single modality to generate a unified multimodal time-series feature vector; analyze the data using the large time-series model combined with the large knowledge model to generate analysis results; display the analysis results and generate early warning information; and generate decision suggestions by combining the analysis results with the large knowledge model. This invention introduces domain knowledge provided by the large knowledge model into the analysis process of the large time-series model, enabling the model to have causal reasoning capabilities when performing multimodal time-series data analysis, realizing the semantic association between anomaly detection results and domain knowledge. Combined with visualization results and a decision generation process driven by early warning levels, it can improve the accuracy of anomaly identification while achieving intelligent output of fault location and maintenance suggestions. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The invention will be described in detail below through specific embodiments.

[0016] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the analysis method based on multimodal time series data provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0017] like Figure 2 As shown, the analysis method based on multimodal time series data proposed in this invention includes the following steps: S10, collect multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data, and organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream in a unified format. In this embodiment, the process of acquiring multi-source heterogeneous data includes obtaining and uniformly managing industrial data from different sources with different data structures and sampling characteristics. Data sources typically include physical quantity sensors, visual image acquisition devices, and operation control systems. These data respectively reflect the dynamic physical response, visual representation, and operational behavior trajectory of the equipment's operating status. Sensor data generally includes continuously sampled time-series data such as acceleration, temperature, pressure, current, and voltage, with the sampling frequency configurable according to the industrial site settings. Image data is obtained by periodically acquiring image sequences through industrial cameras, with frame rate and resolution set according to processing requirements. Control parameter data comes from log records in the control system, PLC command streams, or instructions and process parameters in the MES system. The structural differences between various types of data manifest as inconsistent time-domain distribution, different data formats, and inconsistent missing fields, requiring parallel acquisition and caching through a unified acquisition framework.

[0018] Preprocessing involves multiple stages. For physical quantity data, noise filtering is required, such as using Kalman filtering and median filtering to suppress background jitter and high-frequency random disturbances, and using the sliding window mean method to supplement missing sampling points. For visual image sequences, spatial domain alignment and enhancement processing are needed, mainly including region of interest cropping, image registration, lens distortion correction, and pixel normalization. For structured control parameter data, invalid characters or abnormal fields must first be removed, and format standardization processing must be performed, such as converting the original tabular data into a hierarchical JSON structure for subsequent use. Preprocessing not only eliminates redundancy, conflicts, and inconsistencies in the data but also improves the collaborative expressive ability between multiple modalities.

[0019] The goal of time synchronization is to establish consistency across modal data in the time dimension. Due to differences in sampling mechanisms across different devices, a millisecond-level timestamp unification mechanism is required, along with setting a global time reference point and aligning data using a sliding time window. The sliding window size is dynamically adjusted based on the response speed and change cycle of each modality. For unaligned data sequences, unification is achieved through interpolation, delay correction, and time window sliding mechanisms. The synchronization process also needs to address sampling delays and trigger deviations, introducing a time delay compensation model to correct timing offsets when necessary.

[0020] After synchronization, the processed physical quantity data, visual image data, and control parameter data are organized into a multimodal time-series data stream. This encapsulation process requires structural fusion of different modal data under a unified time base, constructing a cross-modal, multi-field nested data format, such as using a dictionary structure with timestamp keying to encapsulate multimodal observations within the same time slice into composite data units. The structural organization method must be compatible with the input format of subsequent sequence modeling modules, and the data stream can be flow-progressed according to either time-based or event-triggered methods.

[0021] Multi-threaded or asynchronous management mechanisms can be used to concurrently acquire data from different types of data interfaces. Sensor sampling data is connected to the industrial bus, filtered and anomaly compensated by edge computing nodes, and then sent to the data buffer. Data from industrial cameras is streamed uniformly by the acquisition server, and registered, cropped, and standardized by the frame processing module. Control parameters obtain data logs through API connections with the MES system and call the data cleaning module to unify fields and convert formats. In the synchronization module, a global time synchronization protocol (such as PTP) is used to calibrate the time base of all data sources. A sliding time window is used to align the data in the buffer, and data with time drift exceeding a set threshold is dynamically adjusted or removed. The processed data is formatted by the structure encapsulation module to form a unified data stream, which is then stored in an in-memory database or message queue for subsequent sequence modeling and fusion analysis.

[0022] Edge computing modules can also be implemented on the device side to perform local preprocessing on each type of acquired data, and then package the preliminary processing results along with the sampling time and send them to the central platform. The platform then uniformly performs cross-modal alignment and structured encapsulation. A synchronization trigger mechanism can also be set at the acquisition front end, so that camera acquisition and sensor sampling are driven by a unified control signal to eliminate some sampling delay differences. During the format encapsulation stage, the unified multimodal time-series data can be organized into an event stream format, with each time point containing data tags, modality identifiers, original values, and structural attribute metadata to meet real-time inference requirements.

[0023] Example Description: Taking a production line for assembling battery modules for new energy vehicles as an example, sensors are installed on tightening equipment, temperature control nodes, and pressure devices, with a sampling frequency set to 100 times per second. Simultaneously, two industrial cameras are deployed to monitor the appearance of the battery modules in real time, acquiring five frames per second. The MES system records an operation log containing the operator's number, process parameters, and a timestamp after each operation step is completed. The data processing system acquires sensor data via the OPC UA protocol, performs image standardization processing using OpenCV, and extracts the MES logs through a RESTful interface. The system uses the PTP protocol for time base synchronization, constructing a time window every second, sliding and aligning sensor data to the median frame, and cropping image frames to the target area to correspond to the same time window. Operation parameters are assigned to the current window based on the most recently triggered principle. The aligned data is organized into a JSON structure and stored in Redis as key-value pairs for parallel reading and analysis in the subsequent modeling stage.

[0024] This embodiment achieves consistency in the time dimension and structural form of multimodal industrial data by constructing a unified data acquisition, preprocessing, and time synchronization mechanism. This allows data that was originally asynchronous, inconsistent, and full of interference to be uniformly packaged into a standardized time-series data stream before model analysis. This not only eliminates errors caused by data asynchrony in time-series modeling but also provides a clean, aligned, and structurally clear input foundation for subsequent feature extraction and model fusion. This helps improve diagnostic accuracy, reduce model misjudgments, and enhance the temporal correlation credibility of subsequent causal analysis.

[0025] S20, Construct a domain knowledge graph, and build a large knowledge model based on the domain knowledge graph; In this embodiment, the process of constructing a domain knowledge graph includes multi-level, multi-source data semantic extraction, structured representation, and knowledge relationship modeling. The goal is to explicitly represent the static and dynamic knowledge existing in the operation and maintenance of industrial equipment in a graph structure, supporting subsequent reasoning computation and semantic enhancement. Domain knowledge typically includes the structural relationships of equipment components, parameter constraint rules in the process flow, faults and handling behaviors reflected in historical maintenance records, standard operating procedures in operating protocols, and empirical rules. Each type of knowledge content can be collected from structured or unstructured sources.

[0026] Information related to equipment structure typically comes from documents such as equipment manuals, BOM lists, and CAD model annotations. Named entity recognition technology is used to extract the names of equipment components and their structural relationships, such as dependencies, connections, and functions. For example, "the screw is mounted on the bracket" is extracted and transformed into a structural triple "screw—connection—bracket". Process parameter knowledge comes from process specifications or process documents in the MES system. Relationship extraction and rule template mapping are used to obtain parameter entities and their value ranges, applicable conditions, and other constraint information. For example, "the tightening torque should be between 10 and 15 Nm" is extracted and transformed into a triple "tightening torque—constraint—10-15 Nm".

[0027] The sources of maintenance knowledge are historical work orders, maintenance records, or equipment logs. Information extraction algorithms are used to identify fault types, fault locations, and corresponding maintenance measures, constructing relationships such as "high temperature alarm - handling method - replacement of cooling module". After extraction, different knowledge types are integrated into an initial domain knowledge graph through a unified ontology design and semantic mapping mechanism. Then, operations such as entity deduplication, relation merging, hierarchical linking, and type normalization are performed to form a semantically closed structure of the graph.

[0028] Based on the initial graph, causal reasoning chains are constructed using rule definition and knowledge rule mining methods. These causal relationships can be automatically extracted or manually labeled based on equipment failure chain analysis, human factors engineering logic, or historical case patterns. For example, "filter blockage leads to abnormal air pressure, which in turn leads to fixture failure" can be constructed as a causal path of "filter blockage → abnormal air pressure → fixture failure" and expressed in the graph as a path.

[0029] Building upon a domain knowledge graph with a causal structure, a large-scale knowledge model is further constructed. This model is a framework that takes graph entities and their context as input and outputs inference results, classification labels, or semantically enhanced representations. It can be constructed using graph neural networks, graph attention networks, or by introducing graph bias or graph guidance mechanisms into the Transformer structure. Training data comes from real-world paths, subgraphs, and upstream / downstream influence information between nodes within the domain graph. During training, the model not only learns entity embedding representations but also forms semantically structure-guided inference logic, possessing the capabilities for semantic completion, causal prediction, contextual recall, and semantic correction of new inputs.

[0030] A two-way interface is established between the knowledge big model and the knowledge graph. On the one hand, the graph is used as an input embedding source for entity context modeling. On the other hand, the reasoning results are subject to structural constraints or explanation paths, enabling the knowledge graph to have dynamic enhancement, semantic association and logical tracing capabilities, providing structural priors and interpretable support for subsequent multimodal analysis tasks.

[0031] Pre-trained language models such as BERT or RoBERTa, combined with rule templates, can be used to perform entity recognition and relation extraction on textual information in equipment manuals, process standard documents, and maintenance logs. A custom dictionary is used to expand the recognition capability of industrial terms, and a CRF layer is used for fine-grained entity boundary localization. The extracted structural triples are semantically fused with a unified namespace and ontology standards, and a graph structure is built using a graph database (such as Neo4j), configuring node attributes, edge types, and graph indexing rules.

[0032] Sample data can also be constructed based on paths and subgraphs established from the graph, and node classification training can be performed using graph neural networks (such as GCN and GAT), or a Transformer model based on graph attention can be trained using path sampling. Graph constraints can be introduced during the inference stage to achieve graph-text semantic linkage. If the data volume is large, a knowledge embedding module can be constructed to pre-compute entity representations for fast querying during the inference stage, and the dynamic evolution state of the graph can be managed based on time labels.

[0033] In the causal reasoning part, the causal paths co-constructed by humans and machines can be organized into a set of causal chains that can be reasoned through a rule graph representation by a structural rule encoder. Graph neural networks or logical neural networks can be introduced to realize path tracing and fault explanation. At the same time, in order to enhance the reasoning ability, a fault propagation probability model can be established or a reinforcement learning mechanism can be introduced to dynamically optimize the selection of causal paths.

[0034] Example Description: Taking a motor drive system as an example, the system identifies structural triples such as "rotor-connection-bearing" and "bearing-connection-motor housing" from equipment drawings; extracts process knowledge such as "bearing temperature-threshold-85 degrees Celsius" and "lubrication cycle-constraint-100 hours" from process documents; and extracts maintenance behaviors such as "bearing overheating-treatment method-lubricating oil replacement" and "rotor jamming-treatment method-shoulder replacement" from maintenance records. A complete knowledge graph is formed through graph fusion. Based on this, using "bearing overheating" as the observation node, the knowledge model can identify upstream causes that may lead to this event based on path learning, such as "insufficient lubrication → increased bearing friction → bearing overheating," and provides credibility scores for multiple fault explanation paths. Simultaneously, combining solutions and their success rates from existing maintenance records, the system recommends the optimal handling strategy and explains its matching basis and potential impact, thereby achieving a collaborative diagnostic analysis process of data-driven and knowledge-based reasoning.

[0035] This embodiment constructs a structured domain knowledge graph and trains a graph-driven knowledge model, enabling the introduction of semantic and causal prior knowledge structures beyond data-driven analysis. This enhances semantic understanding and path interpretability during fault diagnosis. Furthermore, the graph structure unifies the organization of equipment components, fault characteristics, and maintenance behaviors, improving the model's adaptability to limited samples, rare faults, and changing operating conditions. This provides a traceable and verifiable causal basis for the decision-making logic of downstream models.

[0036] S30, extract each single-modal feature from the multimodal time-series data stream, and fuse the single-modal features to generate a unified multimodal time-series feature vector; In this embodiment, the process of extracting individual modal features from a multimodal time-series data stream and fusing these features to generate a unified multimodal time-series feature vector requires modeling and mapping the differences in temporal representation, feature dimension, structural form, and information density of different modal data to achieve effective joint representation. Multimodal time-series data streams typically include standardized and aligned physical quantity time-series data, visual image sequence data, and structured operational parameter data. These data originate from different systems such as device state perception, environmental perception, and control command execution, and exhibit inherent heterogeneity in data space and temporal sequence structure.

[0037] When extracting single-modal features, for time-series data of physical quantities, time-series modeling networks based on gating mechanisms are typically used. Gated recurrent unit networks (GRUNs) are a commonly used network structure for modeling dynamic evolution patterns. They can capture small fluctuation trends over long periods and are suitable for modeling the dynamic features of continuously changing signals such as temperature, vibration, and pressure. This network takes a time-series sequence as input and iteratively updates the hidden state, ultimately outputting a dynamic mode feature vector corresponding to each time window, reflecting the evolution trend and deviation of the current state from historical states.

[0038] For visual image sequence data, a 3D convolutional network is used to model joint spatial-temporal features. When processing images, the 3D convolutional kernel not only captures local texture, edge, and shape features in the spatial dimension, but also models inter-frame changes along the temporal dimension, such as the procedural manifestations of rotation, vibration, or surface wear. This structure is suitable for extracting sequential features from visually dominant modalities such as equipment movement, surface defect evolution, and liquid motion. Its output is a visual spatial and temporal feature vector, typically possessing high dimensionality and rich contextual structure information.

[0039] After cleaning and format conversion, the structured operation parameter data undergoes feature extraction. Categorical information is transformed into vector representations through one-hot encoding, which are then input into a temporal convolutional network. This network models the local continuous patterns of operation instructions, such as the temporal relationships between multiple operation steps and high-frequency linkage patterns in operation combinations. This generates statistical feature vectors that reflect the distribution and changes in control behavior over different time periods.

[0040] After extracting the three types of single-modal features, they need to be fused. The fusion process uses an attention mechanism to allocate feature weights, dynamically generating a weight distribution based on the discriminative power of each modality feature in the current diagnostic task. The weight generation mechanism can be guided by self-attention, gating networks, or external supervision signals to form a soft allocation structure between modalities, which is then further subjected to a weighted concatenation operation. The concatenation result constitutes a multimodal joint feature vector, containing the joint representation and interaction information of the three modalities within the time window.

[0041] To match the downstream task's requirement for a unified representation structure, a linear transformation and dimensionality reduction process using a fully connected layer is performed on the multimodal joint feature vector. This step achieves feature compression and information centralization, eliminates redundant dimensions, and generates a unified multimodal temporal feature vector while maintaining discriminative power, providing structured input for subsequent anomaly detection, classification, and interpretation analysis.

[0042] A two-layer time-series modeling structure can be constructed using a gated recurrent unit network. Each layer contains several hidden units, and the time step truncation length and sliding window size are configured to handle time-series physical quantity data with different sampling frequencies. The input data is Z-score normalized and residual connections and Dropout control are performed in the network to generate stable dynamic pattern features.

[0043] For visual image sequences, a visual encoding network can be constructed by stacking multiple convolutional-normalization-pooling modules with three-frame × three-row × three-column convolutional kernels. During the training phase, labeled video clips are used for target task pre-training, enabling the network to have prior perception capabilities of device visual events, and feature fine-tuning is performed in the actual task.

[0044] The encoding process for operational parameter data involves vectorizing categorical operational states into vectors and mapping them to a fixed-dimensional vector space. Short-term operational combinations and periodic structures are extracted using multiple one-dimensional convolutional kernels with varying widths. The output, after time pooling, generates a high-order feature vector representing the statistical pattern.

[0045] In the feature fusion stage, a self-attention mechanism with a shared attention weight matrix is ​​introduced. For each time window, the cross-correlation value between modalities is calculated to determine the modal weights during fusion. The weights are applied to the modal feature concatenation result, and the output joint feature vector is then passed through a fully connected network containing a ReLU activation function to complete the dimensionality reduction mapping, outputting a standard feature representation for inference tasks.

[0046] Example Description: In a continuous casting steel equipment, temperature sensors capture real-time temperature changes in the billet cooling section, industrial cameras acquire image sequences of the continuous casting plate surface, and the operation control system records the casting speed and the opening and closing status of the cooling valves. Physical quantity data is input into a GRU model to extract temperature fluctuation patterns; image sequences are input into a 3D CNN to extract surface anomaly evolution features; and operation status is encoded and then input into a TCN to extract operation strategy switching modes. These three data points are fused and dimensionality reduced using an attention mechanism to generate a unified feature vector. When the system exhibits a combined pattern of "sudden high temperature + image blurring + frequent control operation switching," this joint feature vector can accurately capture and indicate the critical evolution trend of the abnormal event.

[0047] This embodiment maximizes the complementarity of each modality in terms of perceptual granularity, temporal coverage, and contextual structure by extracting and dynamically fusing single-modal features from multimodal temporal data streams. This enhances the discriminativeness and stability of the joint representation in anomaly detection, state classification, and root cause analysis. The introduction of an attention mechanism makes feature fusion task-relevant and scene-adaptive, avoiding information dilution or modal redundancy issues and providing a higher-confidence input representation for downstream analysis.

[0048] S40, the unified multimodal time series feature vector is analyzed through the time series big model, and the domain knowledge provided by the knowledge big model is integrated into the analysis process of the time series big model to generate analysis results; In this embodiment, the analysis of a unified multimodal temporal feature vector first requires inputting this feature vector into a network structure capable of temporal modeling, typically a large temporal model with an encoder and decoder. This model takes the multimodal features within the current time window as input and, through a deep sequence modeling module, such as a multi-layered structure based on self-attention, captures the dynamic correlation between evolutionary patterns and features over time. The encoder part converts the input feature sequence into a set of hidden state vectors, which encode the temporal evolution information of the device or process state within the current time period.

[0049] The key mechanism for integrating domain knowledge into the analysis process lies in the interaction between the knowledge context and the temporal modeling process. The content provided by the large knowledge model is typically organized in the form of a knowledge graph, where nodes represent entities, such as equipment components, failure modes, and operating states, and edges represent semantic relationships between entities, such as causality, constraints, evolution, and structural dependencies. Entity vectors related to the feature patterns of the current time window are retrieved from the large knowledge model, and a knowledge context representation is constructed. This representation includes not only the embedded features of nodes but also structural path information and causal rule chains.

[0050] A cross-modal attention mechanism can be used to interact with knowledge context and temporal features. In this mechanism, the hidden state vector in the temporal model serves as the query vector, and the knowledge context vector serves as the key and value. Attention weights are calculated to achieve knowledge-guided representation reconstruction. The output is a knowledge-enhanced temporal feature representation with stronger structural interpretability and diagnostic specificity.

[0051] This enhancement is then input into the decoder module to perform anomaly detection and root cause analysis. The decoder calculates the anomaly score sequence by measuring the reconstruction error between the current temporal features and historical normal behavior patterns. The reconstruction error is an indirect measure used to identify the degree of deviation between the current state and the normal state.

[0052] To classify and interpret anomalies, root cause path reasoning is performed using causal rule chains from a large knowledge model. Reasoning methods include node propagation path simulation in graph neural networks and conditional probability chain analysis in Bayesian causal networks. Based on the reasoning results, possible fault types and their key propagation paths in the causal chain are determined, thus integrating anomaly scoring, fault type, and causal chain interpretation into the final analysis result.

[0053] A multi-head self-attention mechanism can be used to construct the encoder part of a large temporal model. This mechanism splits the unified multimodal temporal feature vector into sequential data in the form of time windows, inputs them into a Transformer encoding layer, and outputs a sequence of hidden states for each time step. A positional encoding module is configured in the encoder to enhance the temporal sequence modeling capability.

[0054] The knowledge context representation consists of the k entity vectors in the knowledge graph that are most similar to the current feature space. Similarity can be calculated using cosine distance or Euclidean distance, or a semantic alignment network can be introduced for modality matching and mapping. After selecting entities, their adjacency structure is truncated and sampled to form a context graph structure, and a vector representation that can be used for attention interactions is generated through graph embedding methods.

[0055] The cross-modal attention interaction module matches the hidden state sequence with the knowledge context vectors, and uses a scaled dot product attention mechanism to assign weights to each knowledge vector, generating a weighted combined knowledge-enhanced representation. This representation maintains the original temporal dimension structure and incorporates knowledge-guided bias information into the temporal representation.

[0056] The decoder section employs a bidirectional GRU network or a Transformer decoding layer, taking the knowledge-enhanced temporal representation as input and comparing it with historical normal behavior patterns to generate reconstructed values. Anomaly scores for each time step are obtained by calculating the mean squared error or Kullback-Leibler divergence between the current representation and the reconstructed representation.

[0057] In root cause path backtracking, a graph search algorithm based on path scores is introduced to traverse the causal paths in the knowledge graph starting from the current failure mode. The path is evaluated and sorted by combining high-frequency nodes with anomaly scores, and finally the most likely failure type and its causal propagation chain are output.

[0058] Example Description: During the operation of a high-voltage motor, a multi-modal sensor system collects information such as vibration frequency, operating current, stator temperature, and stator surface images, generating a unified multi-modal time-series feature vector. When input into a large-scale time-series model, the model, through an encoder, captures the temporal evolution characteristics of a slow temperature increase accompanied by an enhancement of high-frequency components in the vibration spectrum. The system further retrieves entities related to "cooling airflow blockage" from the large-scale knowledge model, discovering a causal path between this entity and "winding temperature rise" and "abnormal current fluctuations," inferring that the anomaly may originate from a cooling fan failure. The final analysis results include the high-risk time window of the anomaly scoring sequence, the fault type "cooling system failure," and its causal path explanation chain: "fan blockage → winding temperature rise → power oscillation." This result directly provides maintenance guidance, assisting maintenance personnel in accurately locating and handling problems.

[0059] This embodiment combines unified multimodal temporal feature vectors with domain knowledge for analysis, enabling the diagnostic system to possess both high-resolution temporal perception capabilities and structured semantic reasoning capabilities, significantly improving the sensitivity and accuracy of anomaly detection. Knowledge injection enhances the model's ability to perceive low-frequency faults and small-sample anomalies, supporting robust anomaly identification even in scenarios with incomplete data. Causal chain-based reasoning further enhances the visibility of diagnostic interpretation, providing a basis for subsequent strategy selection and human decision-making.

[0060] S50, Display the analysis results and generate early warning information based on the analysis results; In this embodiment, the presentation of analysis results requires a structured representation of the results generated by the multimodal perception model and diagnostic model, transforming them into easily understandable and operable visual output. The analysis results consist of anomaly scoring sequences, fault type classifications, and causal chain explanations. Anomaly scoring sequences are typically one-dimensional score vectors corresponding to a time window, with values ​​reflecting the degree of anomaly at each moment. Fault type classifications correspond to the labels of potential faults diagnosed during the current time period, usually output by a multi-classification network or knowledge graph inference mechanism. Causal chain explanations combine path information from the knowledge graph to obtain event association chains, reflecting the evolutionary logic and potential propagation mechanisms of the faults.

[0061] To improve the efficiency of diagnostic results and enhance user perception, the abnormal scoring sequence needs to be transformed into a time-stamped trend chart to visually represent the evolution of the abnormality. This process can utilize a dual-axis line chart, with the horizontal axis representing timestamps and the vertical axis representing the degree of abnormality. Fault type labels and threshold indicator lines can be overlaid in areas where peak values ​​occur.

[0062] A 3D equipment model is used to visually display the location of the fault. After locating the fault type, the corresponding physical component is found in the equipment structure model, and the abnormal component is marked using techniques such as highlighting, flashing, and thermal rendering. This 3D model should have interactive attributes, supporting user operations such as rotation, zooming, and focusing.

[0063] The fault propagation path topology graph is generated based on the causal chain interpretation. The graph consists of nodes and directed edges; nodes represent critical states or events, and edges represent causal or temporal dependencies between them. The graph structure can adopt a hierarchical layout, allowing the fault propagation path to unfold from top to bottom, facilitating the observation of event evolution trends.

[0064] The generation of early warning information is based on an anomaly scoring sequence, combined with preset scoring threshold ranges for level determination. The levels can be divided into four ranges: "Normal," "Observation," "Early Warning," and "Emergency," each corresponding to a different response strategy. When the emergency level is reached, an SMS, email, or app push notification is sent via an external communication module, simultaneously triggering an audible and visual alarm signal on the local device. When the system is in the observation range, a list of tasks to be processed and suggested action times are generated.

[0065] Finally, the trend chart, 3D model, propagation topology, and warning level icons are overlaid and displayed on the monitoring terminal through a split-screen interface, window switching, or interactive controls, improving the response efficiency and information acquisition capabilities of maintenance personnel.

[0066] Anomaly scoring sequences can be dynamically mapped to time curves using data visualization engines (such as Plotly or D3.js), with a dynamic refresh mechanism to update the curve trend in real time. Fault points are marked in red, abnormal intervals are highlighted with transparent masks, and auxiliary lines display alarm thresholds.

[0067] The 3D device model is built based on CAD design files, converted to a web-loadable format (such as glTF), and deployed in a 3D engine (such as Three.js). During fault localization, the model component identifiers are matched according to the current classification results, and their material properties are dynamically changed to achieve flashing or color changes.

[0068] The causal path topology graph selects the set of nodes and edges corresponding to the current reasoning chain from the knowledge graph and generates an interactive graph structure using Graphviz or ECharts. Clicking on any node can expand its upstream and downstream events.

[0069] The warning level determination uses a simple interval matching logic. For example, an anomaly score below one threshold is considered a normal zone, and a score above another threshold is considered an emergency zone. Once an entry into the emergency zone is detected, the warning mechanism is triggered via SMS gateway, HTTP interface, or industrial bus communication module, and suggested task entries are generated simultaneously.

[0070] The warning display interface can adopt a multi-view layout, with a trend chart on the left, a 3D device diagram in the middle, a cause-and-effect path diagram on the right, and warning levels and strategy suggestions at the bottom. Users can customize the display items and set the view refresh cycle.

[0071] Example Description: In the equipment monitoring system of the injection molding workshop, the model outputs an abnormal score curve over a period of time, showing an abnormal temperature rise trend in its cooling system. The trend graph shows a continuous rise in the score in the middle region, exceeding the preset "warning" threshold. The system highlights the cooling pump component in the 3D injection molding machine model and simultaneously displays its causal path: "Increased pump resistance → Decreased cooling flow rate → Increased mold cavity temperature → Increased risk of product deformation." Because the score enters the "emergency" level range, the system immediately triggers an audible and visual alarm via the industrial Ethernet, simultaneously sending a warning message to the manager's mobile phone and generating a maintenance suggestion at the bottom of the monitoring screen: "It is recommended to replace the cooling pump impeller after this injection cycle. The estimated repair time is approximately 15 minutes. The current inventory of this part is sufficient." Based on this, maintenance personnel dispatched personnel and spare parts to complete the repair, effectively avoiding potential quality problems.

[0072] This embodiment presents the analysis results from multiple perspectives through graphics, models, and reasoning paths, significantly improving the interpretability and response efficiency of fault diagnosis results. A multi-level early warning mechanism automatically identifies severe abnormal states and triggers remote or local notifications, enabling maintenance personnel to obtain risk information and take intervention measures immediately. By overlaying multi-dimensional graphical information, the intuitiveness and operational guidance of the analysis results are enhanced, thereby reducing the reliance on manual experience in maintenance and improving system stability and operational efficiency.

[0073] S60, Combining the analysis results with the knowledge model, generate decision recommendations.

[0074] In this embodiment, the process of generating decision recommendations by combining analysis results with the knowledge-based big data model first requires constructing a structured description of the current fault situation based on the fault type classification, anomaly scoring sequence, and causal chain explanation information output in the analysis results. This description includes the current equipment status, fault label, anomaly severity, and potential propagation paths. These elements constitute the input index for matching with the maintenance knowledge in the knowledge-based big data model.

[0075] A knowledge-based large-scale model typically consists of a maintenance case library and an operating procedure library. The maintenance case library records the relationships between historical equipment failures and corresponding handling solutions, while the operating procedure library contains information such as standard operating procedures, maintenance sequences, required tools, and procedural specifications. First, by performing similarity matching between the constructed failure scenario and historical records in the maintenance case library, the historical handling strategy most closely related to the current failure type classification is retrieved, generating a set of candidate maintenance strategies. Similarity matching can be calculated comprehensively based on dimensions such as failure label, component level, and anomaly propagation path similarity.

[0076] Next, obtain the real-time spare parts inventory status data of the current equipment to identify whether the material resources required to execute the alternative maintenance strategy are available under the current operating conditions. The inventory status can be obtained from the material management module or asset monitoring module in the enterprise resource management system. The availability status of equipment identification and component model can be queried in real time through the interface to return whether the required parts are available to perform the operation.

[0077] Subsequently, the time cost of each maintenance strategy needs to be evaluated. The time cost assessment depends not only on the standard execution time of the maintenance operation itself, but also on factors such as replacement cycle, required manpower hours, and whether it affects other production tasks. Therefore, a maintenance time prediction model is introduced to predict the execution time of each strategy, resulting in a time cost assessment result corresponding to each strategy.

[0078] The set of alternative maintenance strategies, corresponding spare parts availability indicators, and time cost assessment results are input into the multi-objective optimization module. This module comprehensively considers objectives such as response speed, resource consumption, and recovery efficiency, and uses constrained programming or heuristic algorithms to perform weight balancing calculations, generating a priority-ordered maintenance strategy queue. The priority ranking reflects the comprehensive evaluation result between the feasibility, economy, and urgency of the strategies.

[0079] For maintenance strategies prioritized in the system, their standard operating procedures are retrieved from the operating procedure library, including the order of steps, precautions, supporting tools, and safety constraints. If the system supports augmented reality environments, these operating procedures can be converted into an AR-recognizable format to generate an AR-executable instruction set, which can be used to support on-site personnel in step-by-step guided operation via AR glasses or tablet devices.

[0080] Finally, the maintenance strategy queue and the augmented reality executable instruction set are integrated into a decision recommendation report. The report structure includes: recommended strategies, executable steps, resource status assessment, estimated recovery time, and visualized operation process. It can be exported to HTML, PDF, or mobile display formats for operation and maintenance decision-making.

[0081] A semantic matching mechanism between historical faults and maintenance plans can be constructed using a graph embedding model. An embedding vector is generated for each case in the maintenance case library. The cosine similarity between the current diagnostic result vector and the historical vectors is calculated, and the closest strategies are selected as a candidate set. Equipment model and operating parameters can be introduced as auxiliary matching conditions to improve accuracy. Spare parts inventory status is queried from the warehouse management platform via an API interface. A binary status identifier is formed based on the spare parts number and the matching strategy, marked as "available" or "unavailable." If unavailable, the possibility of scheduled delivery or temporary allocation can be further evaluated.

[0082] Maintenance time assessment models can employ regression models trained on historical data, such as XGBoost or neural networks, or rule-based models that cumulatively estimate time based on operational steps. The estimated results, combined with the strategy execution sequence, form a timeline for subsequent optimization.

[0083] The multi-objective optimization module can employ a weighted linear programming model or an evolutionary algorithm such as NSGA-II. Factors such as response time, resource constraints, and reliability can be set in the objective function. The output policy queue includes a score, facilitating the visualization of the sorting logic on the interface.

[0084] The process of generating an augmented reality executable instruction set includes converting standard operating procedures into step-by-step graphical instructions, adding semantic anchors and key interaction area markers, and locating the execution position on the field device using SLAM mapping technology. The final generated AR instruction set is compatible with HoloLens or Android terminal visualization platforms.

[0085] Example Description: In a smart pumping station, after the monitoring system diagnoses a fault type of "abnormal impeller vibration, tending towards bearing wear," the system searches its knowledge base for historical impeller fault maintenance plans, obtaining three strategies: "bearing replacement," "calibration center," and "temporary load reduction operation." A check of the inventory reveals available bearings but no spare couplings in the spare parts warehouse. The estimated downtime for the three strategies is assessed at 35 minutes, 65 minutes, and 20 minutes, respectively. A multi-objective optimization algorithm calculates the priority: "bearing replacement" > "temporary load reduction operation" > "calibration center." The system then retrieves the standard procedure for "bearing replacement" from the operating procedure library and generates step-by-step AR instructions, including steps such as "disassembling the protective cover," "removing the old bearing," "installing the new bearing," and "lubrication injection," which are projected onto the equipment in real time via AR glasses. Finally, a strategy recommendation report is generated, showing the recommended execution strategy, the estimated recovery time of 35 minutes, and the estimated consumption of one bearing. An AR instruction QR code is included; maintenance engineers can scan the code to access the AR guidance interface, significantly improving response speed and maintenance success rate.

[0086] This embodiment links diagnostic results with a knowledge model to achieve a closed-loop process from anomaly identification to maintenance decision-making, enhancing the system's automation and intelligence. By integrating historical cases, inventory resources, and time costs into an optimization mechanism, the feasibility and response efficiency of strategies are improved. Through AR command conversion, complex maintenance tasks are streamlined, step-by-step, and visualized, reducing human judgment and operational errors and effectively supporting the consistency and standardization of decision execution. The strategy sorting and command display formats are rich and flexible, and can be embedded into various platforms such as operation and maintenance workbench, mobile terminals, and production control panels.

[0087] This invention relates to the field of data processing technology and discloses an analysis method, apparatus, device, and medium based on multimodal time-series data. The method includes: collecting multi-source heterogeneous data, preprocessing and synchronizing it in time to generate a unified format multimodal time-series data stream; constructing a domain knowledge graph and establishing a large knowledge model; extracting and fusing features from each single modality to generate a unified multimodal time-series feature vector; analyzing the data using the large time-series model combined with the large knowledge model to generate analysis results; displaying the analysis results and generating early warning information; and generating decision suggestions by combining the analysis results with the large knowledge model. This invention introduces domain knowledge provided by the large knowledge model into the analysis process of the large time-series model, enabling the model to possess causal reasoning capabilities when performing multimodal time-series data analysis. This achieves semantic association between anomaly detection results and domain knowledge. Combined with visualization results and a decision generation process driven by early warning levels, it can improve the accuracy of anomaly identification while intelligently outputting fault location and maintenance suggestions.

[0088] In one embodiment, step S10 above includes: S101 acquires physical quantity time-series data through vibration sensors, temperature sensors, and pressure sensors, collects visual image sequences of the equipment surface state through an industrial camera, and exports structured operation parameters containing process numbers from the manufacturing execution system. S102, Perform Kalman filtering on the physical quantity time series data to generate denoised physical quantity time series data, and perform neighbor data imputation on the denoised physical quantity time series data to generate physical quantity time series data with missing values ​​imputed. S103, Perform a standardization operation on the physical quantity time series data after the missing values ​​are imputed to generate standardized physical quantity time series data; S104, Perform a region of interest extraction operation on the visual image sequence to generate a visual image sequence after region cropping, and perform an image registration operation based on feature points on the visual image sequence after region cropping to generate a visual image sequence with coordinate alignment. S105, Perform lens distortion correction on the coordinate-aligned visual image sequence to generate a geometrically corrected visual image sequence; S106, Perform pixel value normalization on the geometrically corrected visual image sequence to generate a standardized visual image sequence; S107, Perform invalid character cleaning operation on the structured operation parameters to generate cleaned structured operation parameters; S108, Perform a format conversion operation on the cleaned structured operation parameters to generate structured operation parameters with a unified format; S109, Establish a data buffer to store the standardized physical quantity time-series data, standardized visual image sequences, and structured operation parameters with unified format; S110, Perform a sliding window alignment operation based on millisecond-level timestamps on each modal data in the data buffer to generate time-aligned multimodal data; S111, Perform an abnormal data segment removal operation on the time-aligned multimodal data to generate multimodal data within the effective time window; S112, encapsulate the multimodal data within the effective time window into a time-series data packet format to generate a multimodal time-series data stream with a unified format.

[0089] In this embodiment, the acquisition of multi-source heterogeneous data encompasses three data types: physical quantities, visual images, and structured operational parameters. These data originate from various sensing devices and information systems deployed in the industrial field. Time-series physical quantity data is acquired through vibration sensors, temperature sensors, and pressure sensors to reflect dynamic indicators such as equipment operating status, ambient temperature changes, and pressure fluctuations. Visual image sequences are acquired periodically or based on trigger conditions by industrial cameras installed at key locations to capture images of the equipment surface, identifying structural integrity, surface condition, or latent defects. Structured operational parameters are exported from the manufacturing execution system, and the data format is generally a table or JSON structure containing fields such as process number, equipment code, and job task.

[0090] The time-series data of physical quantities contains a certain amount of noise and missing data. First, a Kalman filter is used for denoising. This filter is a dynamic system state estimation method that adaptively corrects outliers based on the residual between observed data and predicted states. If there are missing values ​​in the filtered data due to acquisition interruptions or sampling errors, they are filled by interpolating neighboring data to restore its integrity. Finally, the processed data is standardized to bring its mean to zero and its variance to one, thus avoiding bias in the model caused by different dimensions of physical quantities.

[0091] Visual image sequences are preprocessed to improve their spatial alignment and quality for subsequent modeling. First, a region of interest (ROI) extraction operation is performed using convolutional kernels or region labeling mechanisms to crop out image regions containing key components, preventing the model from focusing on redundant background areas. Next, an image registration algorithm based on feature point matching is used to align image sequences acquired at different times along the coordinate dimension, eliminating feature errors caused by image displacement. Registered images often suffer from geometric distortion due to optical systems, requiring lens distortion correction to restore the true image proportions. Finally, pixel normalization is performed on the images to unify grayscale values ​​or pixel channel ranges within the [0,1] interval, facilitating network input standardization.

[0092] Structured operation parameters may contain redundant symbols and encoding errors due to human input or interface malfunctions. Therefore, invalid character cleanup is necessary to retain valid process numbers and operation fields. After cleanup, the parameters should be uniformly converted to a standard format, such as JSON or key-value pairs, to facilitate subsequent data parsing and mapping operations.

[0093] To achieve unified and collaborative analysis of data from various modalities, standardized physical quantity time-series data, standardized visual image sequences, and uniformly formatted structured operation parameters are stored in memory data buffers for parallel alignment processing. A sliding window alignment operation based on millisecond-level timestamps is performed on each modal data to ensure accurate alignment of corresponding observation points within the same time range across different modalities, forming time-consistent data blocks. Sampling breakpoints or delay segments are detected in the window-aligned data, and data segments that do not meet the time accuracy threshold are removed, retaining only data within the valid time window for subsequent feature extraction. Finally, the valid time window data is encapsulated into structured data packets, each packet including uniformly formatted physical quantity vectors, image frame sequences, and process parameter fragments, forming a standardized multimodal time-series data stream as input for subsequent deep learning models.

[0094] A data acquisition module is deployed in the equipment monitoring platform, configured with a B&K 4535-B vibration sensor at a sampling frequency of 5 kHz; a PT100 standard input temperature sensor with a sampling period of 500 ms; and an industrial camera with a resolution of 2048x1536 and an image frame rate of 30fps. Sensor data is acquired via the Modbus RTU protocol, and camera images are transmitted to the edge computing module via the GigE interface.

[0095] The filter is a Kalman filter implemented in Python using pykalman, which automatically adjusts the state covariance when there are abrupt jumps in the sampled data. Missing value imputation is performed using nearest neighbor interpolation (K=1), which has high repair accuracy for consecutive missing segments of less than 5 points. Image registration uses the SIFT algorithm from the OpenCV library to extract feature points and the RANSAC algorithm to fit the affine matrix. Distortion correction is achieved by calibrating the camera's intrinsic parameter matrix and distortion coefficients.

[0096] The timestamp alignment operation employs a sliding window strategy with a window width of 500ms and a step size of 250ms. This, combined with a frame numbering mechanism, filters and removes data segments that were not sampled on time. In the in-memory data structure, a doubly linked list-based buffer structure is constructed, encapsulating three fields in each data packet: a physical quantity tensor (M×T), an image sequence frame array (T×H×W×C), and a parameter dictionary object.

[0097] This embodiment achieves temporal alignment and unified format organization of three types of heterogeneous data—physical, visual, and structural parameters—through this sequential processing flow. It addresses the modeling difficulties caused by differences in sampling rates, data quality, and dimensional scales, and enhances the downstream model's ability to perceive data temporal consistency and modal correlation. Standardization reduces distribution bias between modalities, improving the convergence stability of the model during training. Buffering and windowing mechanisms ensure time sliding and data integrity during processing, thus establishing the entire multimodal modeling process on a high-quality, structured, and synchronized data foundation.

[0098] In one embodiment, step S20 above includes: S201, extract component entities and assembly relationships from equipment manual text to generate equipment structure knowledge triplet; S202, parse the historical maintenance work order records of the manufacturing execution system and generate a triplet of the relationship between fault type and maintenance action; S203, Identify process parameter threshold entities from unstructured text of process standard documents and generate parameter constraint rule triples; S204, perform a knowledge fusion operation on the equipment structure knowledge triplet, the fault type and maintenance measure relationship triplet, and the parameter constraint rule triplet to generate an initial domain knowledge graph; S205, establish a causal reasoning rule chain in the initial domain knowledge graph to generate a knowledge graph containing fault propagation paths; S206, Based on the entities and relationships in the knowledge graph, train a graph neural network model to generate a knowledge reasoning model; S207, Integrate the knowledge graph with the knowledge reasoning model to generate a large knowledge model.

[0099] In this embodiment, the process of constructing a domain knowledge graph is first based on multi-source text and structured record information from the industrial field. Equipment manual texts contain descriptions of equipment components, interfaces, electrical connections, and mechanical mating relationships; their original carriers may be PDF manuals, CAD model document annotations, or HTML-formatted technical web pages. Specific terms for equipment components are identified using named entity recognition methods. Combined with syntactic analysis and relation extraction models, assembly logic is identified, such as "component A is installed within component B" or "cable X is connected to terminal Y." These entity and relational structures are abstracted into triples, forming equipment structural knowledge triples, which include equipment component names, connection structures, and assembly logic.

[0100] The historical maintenance work orders stored in the Manufacturing Execution System (MES) are structured logs that record equipment failure phenomena, handling strategies, and final states, typically in database tables or CSV format. Fault types, such as "bearing noise" or "motor overload," are extracted from these logs, along with corresponding maintenance measures, such as "changing lubricating oil" or "adjusting belt tension." One-to-one or one-to-many maintenance strategies are identified through the logical relationships between work order fields. Each fault-maintenance pair constitutes a relation triple, recorded in the fault type and maintenance measure relation triple.

[0101] Process standards documents are often published in the form of normative documents, industry standards, or technical white papers, with content structured as unstructured natural language text. Using an entity recognition method combining rule-driven and machine learning approaches, parameter constraint descriptions such as "temperature upper limit" and "pressure tolerance range" are identified. Combined with syntactic dependency relations or a domain dictionary, the parameter name, comparison operator, and numerical range are extracted to generate parameter constraint rule triples. This knowledge is used to supplement the boundary conditions and triggering limits during equipment operation.

[0102] The three types of triples mentioned above exhibit heterogeneity in semantic structure, data source, and granularity. To integrate them into a unified knowledge graph, knowledge fusion operations such as entity disambiguation, attribute alignment, and relation merging are required. By setting standardized entity naming conventions (such as a unified namespace), designing rules, or training mapping models, "motor" and "drive motor" are merged into a unified entity. Redundant nodes are removed, and relation labels are normalized to form an initial domain knowledge graph. The nodes of this graph include equipment components, fault types, parameter items, and process constraints; edges represent connection relationships, causal paths, and constraint chains.

[0103] To enable reasoning capabilities in knowledge graphs, causal reasoning rule chains need to be constructed based on their topological structure. Typical fault propagation paths, such as "abnormal temperature → lubrication failure → bearing wear → abnormal motor current," are extracted from the graph paths through clustering or manually created rules. These paths are then encoded as directed edge sequences or rule templates to enhance the graph's causal expressive power.

[0104] Entities and relationships in the graph are vectorized and then input into a graph neural network model for training. This model learns latent patterns in the graph structure, such as high-frequency co-occurrence relationships between components and influence propagation mechanisms in maintenance paths. The graph neural network can employ structures such as GAT, GCN, or R-GCN. During training, it aggregates and updates based on the graph adjacency matrix and node features, ultimately resulting in a knowledge reasoning model capable of performing reasoning tasks.

[0105] The generated knowledge reasoning model is integrated with the existing knowledge graph at the structural and interface levels to form a large knowledge model. This model can accept external input, such as specific equipment status or fault sequences, and perform tasks such as entity query, path tracing, causal reasoning, and policy generation based on existing knowledge entities, relation edges, and trained semantic embeddings, becoming the basic support capability for subsequent diagnosis and decision-making modules.

[0106] The BERT-CRF model combined with rule templates can be used to annotate technical terms in equipment manuals. For example, from the sentence "This motor consists of a rotor, stator, and housing," entities such as "motor," "rotor," and "stator" can be extracted and their compositional relationships can be labeled. The relationship extraction part uses a dual-channel BiLSTM-CNN network to encode the component relationships in the sentence, and improves the ability to process long sentences through an attention mechanism.

[0107] Work order records are extracted and processed in batches in a structured log table using Spark SQL, establishing matching rules from the "fault description" and "handling method" fields. The correspondence between fault types and maintenance measures is then formed into triples through cluster analysis and manual confirmation.

[0108] In the process documentation, guided by domain rules (such as "when the temperature exceeds..." or "the minimum allowable gap is..."), parameter boundaries are identified through text regular expression matching, and syntactic analysis using Spacy or HanLP is performed to form a parameter-threshold structure. The fusion operation utilizes the Entity Linking system to achieve unified naming, and uses the Neo4j graph database to record all entity nodes and triplet relationships. The graph fusion module performs Jaccard similarity clustering to align synonym nodes.

[0109] The training portion of the graph neural network uses the DGL framework to construct the graph structure input, with entity vector dimensions set to 256, 200 training epochs, and negative sampling contrastive loss as the loss function. Causal chains are obtained by depth-first search in the graph, yielding path templates that are stored in a path rule base. Integration operations connect the knowledge graph and model services via a REST interface. When calling the model, an abnormal node can be input, and the system outputs its possible propagation paths and suggested repair nodes.

[0110] This embodiment integrates static knowledge with a reasoning model to create a dynamic knowledge platform capable of causal tracing, knowledge-based question answering, and strategy recommendation. This significantly improves the accuracy and response speed of anomaly diagnosis and decision suggestion generation, while reducing the complexity of manually maintaining knowledge rules. The trained model can be shared across multiple devices, demonstrating potential for cross-device knowledge transfer.

[0111] In one embodiment, step S30 above includes: S301, Separate physical quantity time-series data, visual image sequence data, and structured operation parameter data from the multimodal time-series data stream; S302, Perform gated recurrent unit network processing on the physical quantity time series data to generate physical quantity dynamic mode feature vector; S303, Perform three-dimensional convolutional network processing on the visual image sequence data to generate visual spatial and temporal feature vectors; S304, the structured operation parameter data is encoded into a one-hot vector sequence and input into a temporal convolutional network to extract operation statistical feature vectors; S305, perform attention weight allocation operation on the physical quantity dynamic mode feature vector, visual space and time feature vector and operation statistics feature vector to generate the feature weight distribution of each modality; S306, Based on the modal feature weight distribution, perform a weighted concatenation operation on the physical quantity dynamic mode feature vector, visual space and time feature vector, and operational statistics feature vector to generate a multimodal joint feature vector; S307, Perform a fully connected layer dimensionality reduction operation on the multimodal joint feature vector to generate a unified multimodal temporal feature vector.

[0112] In this embodiment, to extract feature representations for subsequent analysis from a structurally unified multimodal time-series data stream, the data stream first needs to be decoupled according to its modal attributes. A multimodal time-series data stream typically contains data units organized by timestamps, each unit incorporating data items from multiple sources, such as physical sensor data, industrial vision image frames, and structured logs from equipment control systems. At the structural level, a decoder or feature separator needs to be designed to identify, reconstruct, and divide each type of field in the stream into its own modal input channels, thereby obtaining independent physical quantity time-series data, visual image sequence data, and structured operational parameter data.

[0113] Physical quantity time-series data exhibits significant temporal continuity and dynamic fluctuations, typically including sensor outputs such as temperature, current, and vibration acceleration. To capture its evolution patterns and contextual dependencies, this data needs to be processed by a gated recurrent unit network (GRU) model. GRUs, with their gate mechanism structure, can control the updating and forgetting of memory states, and their ability to capture long dependencies in time-series pattern recognition is superior to that of basic RNNs. This processing yields a dynamic pattern feature vector of physical quantities describing the state changes of the physical system; this vector preserves the trend and magnitude of change of the original sequence in the time dimension.

[0114] Visual image sequence data, acquired by industrial cameras, reflects changes in the external condition of equipment, such as surface wear, liquid leakage, or abnormal displacement. When processing this type of data, a three-dimensional convolutional neural network (3D CNN) should be utilized because it can simultaneously model motion information and structural features in the image sequence in both spatial and temporal dimensions. Short image segments are generated by sliding time windows, input into the 3D CNN model, and the output is a visual spatial and temporal feature vector. This vector reflects the dynamic changes in the equipment's appearance over a short period, while preserving important image details and semantic context.

[0115] Structured operational parameter data refers to process parameter logs from the control system, such as process number, operation type, operator ID, and execution time, typically existing in discrete encoding or label form. For this type of discrete data, one-hot encoding is first required to convert each category value into a high-dimensional sparse vector, making it suitable for subsequent neural network input. To extract temporal statistical features, a Temporal Convolutional Network (TCN) should be used. This model captures the operation frequency, mode switching, and periodic behavior within a time period through dilated convolutional structures, thereby generating operational statistical feature vectors.

[0116] The three types of feature vectors mentioned above differ in their content, dimension, and feature granularity. To achieve semantic collaboration and joint modeling, an attention mechanism is needed to assign weights to each modality feature. The attention weight assignment operation involves constructing an attention scoring function, such as using scaled dot product attention, to calculate the contribution of each modality feature to the joint objective, generating a modality weight distribution. The weight generation process depends on the importance assessment within the current task or time window.

[0117] Based on the generated feature weight distribution, a weighted concatenation operation is performed on the physical quantity dynamic mode feature vector, the visual spatial and temporal feature vector, and the operational statistics feature vector. This involves linearly combining each feature vector according to its corresponding weight and concatenating them dimensionally to form a unified multimodal joint feature vector. This vector can retain the key information of each modality and provides advantages in dimensional uniformity and representational coordination.

[0118] Finally, a fully connected layer is applied to reduce the dimensionality of the joint multimodal feature vector. The fully connected network maps the high-dimensional input to the target dimension space through a parameter matrix, and automatically selects the most discriminative feature combination during training. This process outputs a unified multimodal temporal feature vector, which serves as the input for subsequent temporal analysis and diagnosis, possessing the capabilities of cross-modal alignment, information compression, and expression enhancement.

[0119] This embodiment enhances the understanding of various types of industrial state data by constructing a specific neural network structure to extract time-series information from multiple data modalities. Furthermore, it achieves dynamic information fusion and importance allocation between modalities through an attention mechanism. Compared to traditional simple concatenation or averaging fusion methods, this approach better highlights the contributions of key modalities while preserving the dynamic attributes of the original modalities, significantly improving the accuracy and stability of subsequent models in tasks such as anomaly identification, root cause localization, and trend prediction.

[0120] In one embodiment, step S40 above includes: S401, Input the unified multimodal temporal feature vector into the encoder of the temporal large model to generate a multimodal temporal hidden state sequence; S402, retrieve the fault mode entity vector corresponding to the current time window from the knowledge big model, and generate a knowledge context vector; S403, perform cross-modal attention interaction operation on the knowledge context vector and the multimodal temporal hidden state sequence to generate a knowledge-enhanced temporal feature representation; S404, in the decoder of the large temporal model, the reconstruction error between the knowledge-enhanced temporal feature representation and the historical normal pattern is calculated to generate an anomaly scoring sequence; S405, Based on the causal reasoning rules in the knowledge model, perform root cause path backtracking operation on the abnormal scoring sequence to generate fault type classification and causal chain explanation; S406, integrate the abnormal scoring sequence, fault type classification and causal chain explanation into the analysis results.

[0121] In this embodiment, to achieve in-depth analysis of the unified multimodal temporal feature vector, a structure with strong temporal modeling capabilities is needed, combined with prior knowledge to provide semantic supplementation and causal constraints. The input unified multimodal temporal feature vector represents the result of the previous fusion processing, possessing high comprehensiveness in terms of information dimension and semantic expression. It contains data content from multiple modalities while maintaining its temporal organization. This feature vector sequence is input into the encoder part of the temporal large-scale model to extract the dynamic dependency structure and contextual semantics within the current time period. This encoder is constructed using stacked attention modules or a gating structure, which can be a multi-layer Transformer encoder structure or a deep convolutional temporal network, used to capture long-distance temporal correlations.

[0122] To incorporate semantic priors and domain causal information, it is necessary to retrieve knowledge representations matching the current time window from the large knowledge model, especially fault mode entity vectors. These entity vectors typically originate from a knowledge graph embedding vector set trained on a graph neural network, capable of expressing the correlation between equipment structure, abnormal symptoms, and evolutionary processes. By querying the labels, parameter states, and structural component nodes corresponding to the current input features, a knowledge context vector semantically related to the input data is constructed.

[0123] To effectively fuse knowledge vectors with the multimodal temporal hidden state sequence generated by the encoder, a cross-modal attention interaction operation needs to be performed. This operation constructs a query-key-value attention mechanism, using the knowledge context vector as the value vector to weight and adjust the temporal hidden states, thereby enhancing the output representation with domain knowledge while preserving its temporal structure. This process can employ a multi-head attention structure, allowing semantic relationships to be modeled simultaneously in multiple subspaces, thus improving the interactive fusion effect.

[0124] The fused result is called a knowledge-enhanced temporal feature representation. This representation embeds both the temporal structure information of the input data and the semantic content of the knowledge graph, while retaining traceable fault-related entity labels. In the decoder part of the large-scale temporal model, it is necessary to determine whether anomalies exist in the current time period through reconstruction. The decoder receives the knowledge-enhanced feature representation and attempts to reconstruct the sequence features that should appear under normal historical conditions. Then, it calculates the reconstruction error based on the difference between the reconstruction and the actual input, thus obtaining an anomaly scoring sequence. This sequence marks the degree of deviation at each moment in the time dimension and is the basis for judging the health of the current state.

[0125] To further explain the root cause of the anomalies, root cause analysis based on the anomaly scoring sequence is required. This analysis is based on the causal reasoning rules embedded in the knowledge model, constructing a set of paths tracing back from the anomaly representation to the potential causes by traversing the fault propagation path in reverse. This operation is typically implemented using causal chain propagation algorithms in graph neural networks, such as those based on shortest path search, maximum confidence propagation, or weighted information flow backtracking mechanisms. The output is a fault type classification result and a causal chain explanation, the latter being a set of nodes and edges in a directed graph describing the event chain and the derivation process.

[0126] Finally, the abnormal scoring sequences, fault type classifications, and causal chain explanations are integrated to generate analysis results, forming a structured output for subsequent visualization and decision recommendation modules. During the integration process, the correspondence between data items must be maintained to ensure spatiotemporal consistency between anomaly locations, fault labels, and explanation paths.

[0127] This embodiment achieves deep perception and interpretable modeling of complex multimodal input data by organically combining the dynamic evolution of temporal information with the semantic structure in prior knowledge. Introducing knowledge context vectors and constructing a cross-modal interaction mechanism not only improves the accuracy of anomaly detection but also provides a causal chain-level explanation path for diagnosis. This solves the problems of high false alarm rates and unexplainable root causes in traditional models under low signal-to-noise ratio environments, thereby enhancing the model's deployment credibility and automated operation and maintenance support capabilities in real-world industrial environments.

[0128] In one embodiment, step S50 above includes: S501, map the abnormal score sequence in the analysis results into a real-time trend graph to generate an abnormal fluctuation curve with time stamp; S502, Locate the physical components corresponding to the fault type classification in the analysis results in the three-dimensional equipment model, and generate a highlighted visualization model; S503, extract key node events from the causal chain explanation in the analysis results, and generate a fault propagation path topology map; S504, Match a preset warning level with the numerical range of the abnormal scoring sequence to generate a warning level identifier; S505, When the warning level reaches the preset emergency threshold, a text message notification and an audible and visual alarm signal are triggered. S506, When the warning level is within the preset observation range, a maintenance plan suggestion list is generated; S507, the visualization model, fault propagation path topology map and early warning level identifier are overlaid and displayed on the monitoring interface.

[0129] In this embodiment, the analysis results include outputs such as anomaly scoring sequences, fault type classifications, and causal chain explanations, all presented in a structured format. The anomaly scoring sequence represents a quantitative assessment of the deviation at each time point within a time window, and its numerical changes reflect the changing trend of the system state. Mapping this scoring sequence to a trend graph involves constructing a two-dimensional coordinate graph using a time axis, presenting the scoring values ​​as line graphs, bars, or heatmaps, allowing users to intuitively grasp the location and intensity of abnormal fluctuations. This trend graph needs to include time stamps to correspond to the specific equipment operating time.

[0130] To achieve spatial anomaly localization, the fault type classification in the analysis results needs to be integrated with the equipment model. The equipment model represents physical components in a three-dimensional digital structure. The classification results need to be mapped to predefined identifiers in the equipment structure. By matching tags, the corresponding physical components are highlighted in the three-dimensional model, achieving spatial visualization. Highlighting can use color enhancement, transparency adjustment, or flashing animations to guide users to focus on the fault location.

[0131] In the semantic reasoning dimension, to help users understand the propagation chain of anomalies, key node events are extracted from the causal chain explanation to construct a topology graph structure, displaying the directional path of fault propagation. The topology graph is represented in graph structure form, where nodes represent events and edges represent causal relationships, and can be visualized using graph layout algorithms (such as force-directed graphs).

[0132] The anomaly scoring sequence is used not only for visualization but also to trigger early warning logic. Based on the numerical range and preset thresholds, different risk levels are defined as early warning level identifiers. These level identifiers are represented by structured labels, such as high risk, medium risk, and observation status, with different levels mapping to different response mechanisms.

[0133] When the warning level indicator reaches the emergency response threshold, such as when the score remains above the set upper limit for a certain period of time, the system automatically invokes a preset trigger module to execute immediate response methods such as SMS notification and audible / visual alarms, ensuring that the abnormal status is communicated to on-duty personnel as soon as possible. SMS notifications are sent via the enterprise SMS platform interface, and alarm signals are triggered via the device's audible / visual module or the HMI interface.

[0134] If the system determines that the situation falls within a non-urgent but still requires attention observation period, it will call the maintenance rule base to generate maintenance suggestions, forming a maintenance plan suggestion list, which includes fields such as suggested inspection content, troubleshooting order, tool list, and responsible person assignment.

[0135] Finally, the above outputs are merged and integrated into a unified monitoring interface. This interface supports a layered overlay display mechanism, presenting trend charts, device models, and topology diagrams in the main view, and overlaying the current warning level indicator to ensure spatial, temporal, and semantic consistency between data, thereby improving the interactivity and readability of the monitoring system.

[0136] This embodiment achieves multi-dimensional integration of time, space, and semantics in anomaly localization by visualizing the analysis results from multiple perspectives and dimensions; it effectively improves alarm timeliness by responding promptly to high-risk states through a hierarchical early warning mechanism; it provides maintenance personnel with a clear basis for judgment by combining the causal reasoning display of the topology map with the visual annotation of the equipment model; and it further improves processing efficiency and accuracy by automatically generating executable suggestions in conjunction with the maintenance knowledge base, thereby reducing the false judgment rate and response delay.

[0137] In one embodiment, step S60 above includes: S601, retrieve historical handling solutions that match the fault type classification in the analysis results from the maintenance case library of the knowledge big model, and generate a set of alternative maintenance strategies; S602, obtain the current equipment spare parts inventory status data and generate a spare parts availability identifier; S603 calculates the estimated downtime for each maintenance strategy based on the maintenance time estimation model and generates time cost assessment results; S604, The alternative maintenance strategy set, spare parts availability identifier and time cost assessment result are input into a multi-objective optimization algorithm to generate a priority-ordered maintenance strategy queue; S605, extract the standard operation steps corresponding to the selected maintenance strategy from the operation procedure library of the knowledge big model, and generate an augmented reality executable instruction set; S606, integrate the maintenance strategy queue with the augmented reality executable instruction set into a decision recommendation report.

[0138] In this embodiment, during fault diagnosis and decision support, historical handling solutions matching the fault type classification are first retrieved from the maintenance case library in the knowledge model. The maintenance case library contains a large number of historical maintenance records, indexed and categorized by fault type, with each record corresponding to a specific maintenance measure or operation step. When the system detects a fault type, it retrieves historical handling solutions similar to or consistent with the current fault type classification from the case library based on a matching algorithm (such as content-based recommendation or similarity matching). This allows for the development of targeted maintenance strategies based on historical experience, reducing diagnosis time and error rate.

[0139] Next, spare parts inventory status data for the equipment is obtained to ensure sufficient spare parts support when the recommended maintenance strategy is implemented. Spare parts inventory status is obtained through real-time monitoring of the equipment's spare parts management system, reflecting the quantity, type, and current inventory status of available spare parts. Spare parts availability indicators are generated in this step, indicating which spare parts are immediately usable in the current fault repair and which require procurement or delay. This forms the basis for developing practical operational plans, ensuring the smooth progress of maintenance work.

[0140] Based on the maintenance time prediction model, the estimated downtime for each maintenance strategy is further calculated. Each maintenance strategy may involve different operational steps, and the length of downtime directly affects production efficiency. By introducing the maintenance time prediction model, the required downtime for each maintenance strategy can be estimated based on historical data, equipment complexity, and maintenance difficulty. This process helps production schedulers to rationally arrange production plans, reduce disruptions to the production line, and optimize production efficiency.

[0141] Guided by a multi-objective optimization algorithm, the algorithm comprehensively considers the set of alternative maintenance strategies, spare parts availability indicators, and time cost assessment results to generate a priority ranking queue for maintenance strategies. The multi-objective optimization algorithm ranks each strategy based on multiple factors (such as spare parts availability, time cost, and maintenance difficulty). Through weight adjustment and calculation, the optimization algorithm ensures that the selected maintenance strategies are not only efficient and timely but also minimize production line downtime and spare parts waste.

[0142] Standard operating procedures related to the selected maintenance strategy are extracted from the operating procedure library of the knowledge model to generate an augmented reality (AR) executable instruction set. The operating procedure library contains detailed equipment maintenance manuals and operating procedures to guide maintenance personnel in performing specific operations. By integrating augmented reality technology, intuitive and visual operating guidance can be provided to maintenance personnel, such as displaying maintenance steps, equipment component locations, tool requirements, etc., through smart glasses or mobile devices, further improving maintenance efficiency and accuracy.

[0143] Finally, the prioritized maintenance strategy queue is integrated with the augmented reality executable instruction set to form a complete decision recommendation report. This report provides a clear sequence of maintenance operations, detailed instructions for each step, a list of required tools and spare parts, and relevant timelines. Maintenance personnel can operate according to the report, ensuring a smooth workflow and reducing errors and wasted time.

[0144] In its implementation, the system first extracts historical maintenance records from the maintenance case library within the large knowledge model and generates a set of alternative maintenance strategies based on the current equipment's fault type. Based on real-time equipment data (such as spare parts inventory data obtained via PLC), the system automatically generates spare parts availability indicators to determine if there are sufficient spare parts for maintenance. If inventory is insufficient, the system automatically recommends appropriate procurement operations or schedules spare parts. The maintenance time prediction model, based on historical maintenance records and equipment operation data, uses machine learning algorithms (such as regression analysis or deep learning networks) to predict the downtime for each maintenance strategy. A multi-objective optimization algorithm combines multiple decision criteria (such as spare parts availability, maintenance cost, and time cost) to generate a priority queue through weighted and sorted calculations. Different optimization algorithms, such as genetic algorithms and particle swarm optimization algorithms, can be selected according to actual application requirements. The generation of the AR instruction set relies on an operation procedure library, which contains detailed descriptions of maintenance manuals and procedures. The AR system transforms this information into visual operation guidance, displayed through AR glasses or mobile devices on the equipment, allowing maintenance personnel to directly view operation steps, equipment components, and tool information. Finally, the system combines the generated priority maintenance strategy queue with the AR instruction set to form a decision recommendation report, which is then presented to maintenance personnel through a graphical user interface (GUI), providing clear operating procedures and maintenance suggestions.

[0145] Example description: In the process of health status monitoring and fault diagnosis of CNC machine tool spindle system, the system achieves high-precision detection and accurate diagnosis of faults in complex industrial environments by collecting, preprocessing and fusing multi-source heterogeneous data, and combining the support of knowledge big model.

[0146] Multiple sensors are installed in the CNC machine tool spindle system to collect physical quantity signals such as vibration, temperature, and current, and to acquire real-time visual image sequences of the equipment's surface condition via an industrial camera. Specifically, vibration sensors collect vibration acceleration data in three axes, temperature sensors monitor temperature changes in the spindle bearings, current sensors monitor the operating current of the spindle motor, and the industrial camera provides image sequences of the spindle components. All sensor data is precisely timestamped and undergoes initial noise reduction and time synchronization via an edge computing gateway. The data is uploaded to a central server via the MQTT protocol and stored as a unified format multimodal time-series data stream, ensuring that all modal data can be processed synchronously for subsequent operations.

[0147] Supported by a large knowledge model, a deep analysis of the spindle system's health status is performed by constructing a domain knowledge graph. First, the entity and assembly relationships of each spindle component are extracted from the equipment manual, generating equipment structure knowledge triples. Second, historical maintenance work order records from the manufacturing execution system are parsed to extract the relationship between fault types and maintenance measures, forming fault type-maintenance measure triples. Simultaneously, process parameter thresholds are identified through process standard documents, generating corresponding parameter constraint rule triples. By fusing knowledge from these triples, the system can generate a complete domain knowledge graph containing fault propagation paths. This graph is then combined with a knowledge reasoning model trained on a graph neural network for deep reasoning, further enhancing the model's analytical and diagnostic capabilities.

[0148] In the data processing module, vibration data undergoes frequency and time domain feature extraction using a one-dimensional convolutional neural network (1D-CNN), temperature and current data are processed using a long short-term memory network (LSTM) to extract their long-term trend features, and image data is processed using a pre-trained ResNet-3D model to extract spatiotemporal features. All extracted features are merged into a unified format multimodal temporal feature vector, and a weighted concatenation method is used to weight the features based on the importance of each modality, generating the final multimodal temporal feature vector. These feature vectors will serve as the input to the large-scale temporal model.

[0149] In the temporal large-scale model, the Transformer Encoder architecture is used to encode multimodal temporal feature vectors, generating a multimodal temporal hidden state sequence. Through cross-modal attention interaction, combined with domain knowledge from the knowledge large-scale model (such as fault modes and their causal relationships), knowledge-enhanced temporal feature representations are generated. By reconstructing error calculation, an anomaly score sequence is generated; if the anomaly score exceeds a preset threshold, it is judged as an anomaly. The system simultaneously outputs the probability distribution of fault categories and fault severity assessment, and backtracks key fault paths through causal inference rules to generate detailed fault type classifications and causal chain explanations.

[0150] The system maps abnormal scoring sequences to real-time trend charts and generates time-stamped abnormal fluctuation curves. These curves visually demonstrate the dynamic changes in fault development. In the 3D equipment model, the system locates faulty components based on diagnostic results, generating a highlighted visual model for easy identification of faulty locations by operators. Simultaneously, the system extracts fault propagation paths from causal chain interpretation, generates a fault propagation path topology map, and matches preset warning levels to the numerical ranges of the scoring sequences. When the warning level reaches the emergency threshold, the system triggers SMS notifications and audible and visual alarms; when the warning level is within the observation range, it generates a maintenance plan suggestion list to guide maintenance personnel on how to address potential problems.

[0151] Supported by a large knowledge model, the system retrieves historical handling solutions matching the current fault type from the maintenance case library, generating a set of alternative maintenance strategies. Combined with spare parts inventory status, it generates spare parts availability indicators to ensure sufficient spare parts support for recommended maintenance strategies. Simultaneously, it calculates the estimated downtime for each maintenance strategy using a maintenance time prediction model, optimizing downtime. Finally, the system inputs the alternative maintenance strategies, spare parts availability, and time cost assessment results into a multi-objective optimization algorithm to generate a priority-ranked maintenance strategy queue. It also extracts operational steps from the operation procedure library to generate an augmented reality executable instruction set, ultimately integrating them into a complete decision recommendation report.

[0152] This embodiment comprehensively considers multiple factors (spare parts availability, repair time, historical experience, etc.) and ranks them using a multi-objective optimization algorithm to select the optimal maintenance strategy, greatly improving the efficiency and accuracy of maintenance decisions. With the help of augmented reality technology, maintenance personnel can more intuitively understand operating procedures and equipment status, reducing human error and improving the quality and speed of maintenance work. Furthermore, the integration of a large knowledge model can provide personalized decision support based on historical experience and current equipment status, further optimizing the maintenance process and ensuring the efficient operation of equipment during production.

[0153] In one embodiment, an analysis apparatus based on multimodal time series data is provided, which corresponds one-to-one with the analysis method based on multimodal time series data in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the analysis device based on multimodal time-series data of the present invention. The modules include: data acquisition and synchronization module 10, knowledge graph construction and modeling module 20, feature extraction and fusion module 30, knowledge-enhanced time-series analysis module 40, display and early warning module 50, and decision generation module 60. Detailed descriptions of each functional module are as follows: The data acquisition and synchronization module 10 is used to acquire multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data, and organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream in a unified format. The knowledge graph construction and modeling module 20 is used to construct a domain knowledge graph and build a large knowledge model based on the domain knowledge graph. The feature extraction and fusion module 30 is used to extract each single-modal feature from the multimodal time-series data stream and fuse the single-modal features to generate a unified multimodal time-series feature vector; The knowledge-enhanced time series analysis module 40 is used to analyze the unified multimodal time series feature vector through a large time series model, and to integrate the domain knowledge provided by the knowledge model into the analysis process of the large time series model to generate analysis results. The display and early warning module 50 is used to display the analysis results and generate early warning information based on the analysis results; The decision generation module 60 is used to combine the analysis results with the knowledge model to generate decision recommendations.

[0154] In one embodiment, the data acquisition and synchronization module 10 is specifically used for: Physical quantity time-series data are acquired through vibration sensors, temperature sensors, and pressure sensors. Visual image sequences of the equipment surface state are collected through industrial cameras. Structured operation parameters containing process numbers are exported from the manufacturing execution system. Perform a Kalman filter operation on the physical quantity time series data to generate denoised physical quantity time series data, and perform a neighbor data imputation operation on the denoised physical quantity time series data to generate physical quantity time series data with missing values ​​imputed. Perform a standardization operation on the physical quantity time series data after the missing values ​​are imputed to generate standardized physical quantity time series data; Perform a region of interest extraction operation on the visual image sequence to generate a visual image sequence with cropped regions, and perform a feature point-based image registration operation on the visual image sequence with cropped regions to generate a visual image sequence with aligned coordinates. Perform lens distortion correction on the coordinate-aligned visual image sequence to generate a geometrically corrected visual image sequence; Pixel value normalization is performed on the geometrically corrected visual image sequence to generate a standardized visual image sequence; An invalid character cleanup operation is performed on the structured operation parameters to generate cleaned structured operation parameters; Perform a format conversion operation on the cleaned structured operation parameters to generate structured operation parameters with a unified format; A data buffer is established to store the standardized physical quantity time-series data, standardized visual image sequences, and structured operation parameters with a unified format; Perform a sliding window alignment operation based on millisecond-level timestamps on each modal data in the data buffer to generate time-aligned multimodal data; Anomaly segment removal operation is performed on the time-aligned multimodal data to generate multimodal data within the effective time window; The multimodal data within the effective time window is encapsulated into a time-series data packet format to generate a multimodal time-series data stream with a unified format.

[0155] In one embodiment, the knowledge graph construction and modeling module 20 is specifically used for: Extract component entities and assembly relationships from equipment manual text to generate equipment structure knowledge triples; Analyze the historical maintenance work order records of the Manufacturing Execution System to generate triplet pairs of relationships between fault types and maintenance measures; Identify process parameter threshold entities from unstructured text of process standard documents and generate parameter constraint rule triples; A knowledge fusion operation is performed on the equipment structure knowledge triplet, the fault type and maintenance measure relationship triplet, and the parameter constraint rule triplet to generate an initial domain knowledge graph. A causal reasoning rule chain is established in the initial domain knowledge graph to generate a knowledge graph containing fault propagation paths; A knowledge reasoning model is generated by training a graph neural network model based on the entities and relationships in the knowledge graph. The knowledge graph is integrated with the knowledge reasoning model to generate a large knowledge model.

[0156] In one embodiment, the feature extraction and fusion module 30 is specifically used for: Separate physical quantity time-series data, visual image sequence data, and structured operation parameter data from the multimodal time-series data stream; The time-series data of the physical quantities are processed by a gated recurrent unit network to generate a dynamic pattern feature vector of the physical quantities. The visual image sequence data is processed by a three-dimensional convolutional network to generate visual spatial and temporal feature vectors; The structured operation parameter data is encoded into a one-hot vector sequence and input into a temporal convolutional network to extract operation statistical feature vectors; An attention weight allocation operation is performed on the physical quantity dynamic mode feature vector, visual spatial and temporal feature vector, and operational statistical feature vector to generate the feature weight distribution of each modality. Based on the modal feature weight distribution, a weighted concatenation operation is performed on the physical quantity dynamic mode feature vector, visual space and time feature vector, and operational statistics feature vector to generate a multimodal joint feature vector; A fully connected layer is used to reduce the dimensionality of the multimodal joint feature vector to generate a unified multimodal temporal feature vector.

[0157] In one embodiment, the knowledge-enhanced time-series analysis module 40 is specifically used for: The unified multimodal temporal feature vector is input into the encoder of the temporal large model to generate a multimodal temporal hidden state sequence; Retrieve the fault mode entity vector corresponding to the current time window from the knowledge model to generate a knowledge context vector; Perform cross-modal attention interaction operation between the knowledge context vector and the multimodal temporal hidden state sequence to generate a knowledge-enhanced temporal feature representation; In the decoder of the large temporal model, the reconstruction error between the knowledge-enhanced temporal feature representation and the historical normal pattern is calculated to generate an anomaly scoring sequence; Based on the causal reasoning rules in the aforementioned knowledge model, a root cause path backtracking operation is performed on the abnormal scoring sequence to generate a fault type classification and causal chain explanation. The abnormal scoring sequence, fault type classification, and causal chain explanation are integrated into the analysis results.

[0158] In one embodiment, the display and early warning module 50 is specifically used for: The abnormal score sequences in the analysis results are mapped to real-time trend graphs to generate time-stamped abnormal fluctuation curves; Locate the physical components corresponding to the fault type classifications in the analysis results within the 3D device model, and generate a highlighted visualization model; Key node events are extracted from the causal chain explanation in the analysis results to generate a fault propagation path topology graph. A warning level identifier is generated by matching the numerical range of the abnormal scoring sequence with a preset warning level. When the warning level reaches the preset emergency threshold, an SMS notification and an audible and visual alarm signal are triggered. When the warning level is within the preset observation range, a maintenance plan suggestion list is generated; The visualization model, fault propagation path topology map, and early warning level indicators are overlaid and displayed on the monitoring interface.

[0159] In one embodiment, the decision generation module 60 is specifically used for: Retrieve historical handling solutions that match the fault type classification in the analysis results from the maintenance case library of the knowledge model to generate a set of alternative maintenance strategies; Obtain the current equipment's spare parts inventory status data and generate a spare parts availability identifier; The estimated downtime for each maintenance strategy is calculated based on the maintenance time prediction model, and the time cost assessment results are generated. The alternative maintenance strategy set, spare parts availability identifier, and time cost assessment results are input into a multi-objective optimization algorithm to generate a priority-ranked maintenance strategy queue. Extract standard operating steps corresponding to the selected maintenance strategy from the operating procedure library of the knowledge model, and generate an augmented reality executable instruction set. The maintenance strategy queue is integrated with the augmented reality executable instruction set into a decision recommendation report.

[0160] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side analysis method based on multimodal time-series data.

[0161] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the user-side functions or steps of an analysis method based on multimodal time series data. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Collect multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data. Organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream with a unified format. Construct a domain knowledge graph, and build a large knowledge model based on the domain knowledge graph; Extract each single-modal feature from the multimodal time-series data stream, and fuse the single-modal features to generate a unified multimodal time-series feature vector; The unified multimodal temporal feature vector is analyzed using a time series big model, and the domain knowledge provided by the knowledge big model is incorporated into the analysis process of the time series big model to generate analysis results. Display the analysis results and generate early warning information based on the analysis results; Based on the analysis results and the knowledge model, decision recommendations are generated.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Collect multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data. Organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream with a unified format. Construct a domain knowledge graph, and build a large knowledge model based on the domain knowledge graph; Extract each single-modal feature from the multimodal time-series data stream, and fuse the single-modal features to generate a unified multimodal time-series feature vector; The unified multimodal temporal feature vector is analyzed using a time series big model, and the domain knowledge provided by the knowledge big model is incorporated into the analysis process of the time series big model to generate analysis results. Display the analysis results and generate early warning information based on the analysis results; Based on the analysis results and the knowledge model, decision recommendations are generated.

[0163] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0166] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An analysis method based on multimodal time series data, characterized in that, Includes the following steps: Collect multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data. Organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream with a unified format. Construct a domain knowledge graph, and build a large knowledge model based on the domain knowledge graph; Extract each single-modal feature from the multimodal time-series data stream, and fuse the single-modal features to generate a unified multimodal time-series feature vector; The unified multimodal temporal feature vector is analyzed using a time-series big model, and the domain knowledge provided by the knowledge big model is incorporated into the analysis process of the time-series big model to generate analysis results. Display the analysis results and generate early warning information based on the analysis results; Based on the analysis results and the knowledge model, decision recommendations are generated.

2. The analysis method based on multimodal time series data as described in claim 1, characterized in that, Collect multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data. Organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream with a unified format, including: Physical quantity time-series data are obtained through vibration sensors, temperature sensors, and pressure sensors. Visual image sequences of the equipment surface state are collected through industrial cameras. Structured operation parameters containing process numbers are exported from the manufacturing execution system. Perform a Kalman filter operation on the physical quantity time series data to generate denoised physical quantity time series data, and perform a neighbor data imputation operation on the denoised physical quantity time series data to generate physical quantity time series data with missing values ​​imputed. Perform a standardization operation on the physical quantity time series data after the missing values ​​are imputed to generate standardized physical quantity time series data; Perform a region of interest extraction operation on the visual image sequence to generate a visual image sequence with cropped regions, and perform a feature point-based image registration operation on the visual image sequence with cropped regions to generate a visual image sequence with aligned coordinates. Perform lens distortion correction on the coordinate-aligned visual image sequence to generate a geometrically corrected visual image sequence; Pixel value normalization is performed on the geometrically corrected visual image sequence to generate a standardized visual image sequence; An invalid character cleanup operation is performed on the structured operation parameters to generate cleaned structured operation parameters; Perform a format conversion operation on the cleaned structured operation parameters to generate structured operation parameters with a unified format; A data buffer is established to store the standardized physical quantity time-series data, standardized visual image sequences, and structured operation parameters with a unified format; Perform a sliding window alignment operation based on millisecond-level timestamps on each modal data in the data buffer to generate time-aligned multimodal data; Anomaly segment removal operation is performed on the time-aligned multimodal data to generate multimodal data within the effective time window; The multimodal data within the effective time window is encapsulated into a time-series data packet format to generate a multimodal time-series data stream with a unified format.

3. The analysis method based on multimodal time series data as described in claim 1, characterized in that, Constructing a domain knowledge graph and building a large knowledge model based on the domain knowledge graph, including: Extract component entities and assembly relationships from equipment manual text to generate equipment structure knowledge triples; Analyze the historical maintenance work order records of the Manufacturing Execution System to generate triplet pairs of relationships between fault types and maintenance measures; Identify process parameter threshold entities from unstructured text of process standard documents and generate parameter constraint rule triples; A knowledge fusion operation is performed on the equipment structure knowledge triplet, the fault type and maintenance measure relationship triplet, and the parameter constraint rule triplet to generate an initial domain knowledge graph. A causal reasoning rule chain is established in the initial domain knowledge graph to generate a knowledge graph containing fault propagation paths; A knowledge reasoning model is generated by training a graph neural network model based on the entities and relationships in the knowledge graph. The knowledge graph is integrated with the knowledge reasoning model to generate a large knowledge model.

4. The analysis method based on multimodal time series data as described in claim 1, characterized in that, Extracting individual modal features from the multimodal time-series data stream and fusing these individual modal features to generate a unified multimodal time-series feature vector, including: Separate physical quantity time-series data, visual image sequence data, and structured operation parameter data from the multimodal time-series data stream; The time-series data of the physical quantities are processed by a gated recurrent unit network to generate a dynamic pattern feature vector of the physical quantities. The visual image sequence data is processed by a three-dimensional convolutional network to generate visual spatial and temporal feature vectors; The structured operation parameter data is encoded into a one-hot vector sequence and input into a temporal convolutional network to extract operation statistical feature vectors; An attention weight allocation operation is performed on the physical quantity dynamic mode feature vector, visual spatial and temporal feature vector, and operational statistical feature vector to generate the feature weight distribution of each modality. Based on the modal feature weight distribution, a weighted concatenation operation is performed on the physical quantity dynamic mode feature vector, visual space and time feature vector, and operational statistics feature vector to generate a multimodal joint feature vector; A fully connected layer is used to reduce the dimensionality of the multimodal joint feature vector to generate a unified multimodal temporal feature vector.

5. The analysis method based on multimodal time series data as described in claim 1, characterized in that, The unified multimodal time-series feature vectors are analyzed using a time-series large-scale model. During the analysis process, domain knowledge provided by the knowledge large-scale model is incorporated to generate analysis results, including: The unified multimodal temporal feature vector is input into the encoder of the temporal large model to generate a multimodal temporal hidden state sequence; Retrieve the fault mode entity vector corresponding to the current time window from the knowledge model to generate a knowledge context vector; Perform cross-modal attention interaction operation between the knowledge context vector and the multimodal temporal hidden state sequence to generate a knowledge-enhanced temporal feature representation; In the decoder of the large temporal model, the reconstruction error between the knowledge-enhanced temporal feature representation and the historical normal pattern is calculated to generate an anomaly scoring sequence; Based on the causal reasoning rules in the aforementioned knowledge model, a root cause path backtracking operation is performed on the abnormal scoring sequence to generate a fault type classification and causal chain explanation. The abnormal scoring sequence, fault type classification, and causal chain explanation are integrated into the analysis results.

6. The analysis method based on multimodal time series data as described in claim 1, characterized in that, Display the analysis results and generate early warning information based on the analysis results, including: The abnormal score sequences in the analysis results are mapped to real-time trend graphs to generate time-stamped abnormal fluctuation curves; Locate the physical components corresponding to the fault type classifications in the analysis results within the 3D device model, and generate a highlighted visualization model; Key node events are extracted from the causal chain explanation in the analysis results to generate a fault propagation path topology graph. A warning level identifier is generated by matching the numerical range of the abnormal scoring sequence with a preset warning level. When the warning level reaches the preset emergency threshold, an SMS notification and an audible and visual alarm signal are triggered. When the warning level is within the preset observation range, a maintenance plan suggestion list is generated; The visualization model, fault propagation path topology map, and early warning level indicators are overlaid and displayed on the monitoring interface.

7. The analysis method based on multimodal time series data as described in claim 1, characterized in that, Based on the analysis results and the knowledge model, decision recommendations are generated, including: Retrieve historical handling solutions that match the fault type classification in the analysis results from the maintenance case library of the knowledge model to generate a set of alternative maintenance strategies; Obtain the current equipment's spare parts inventory status data and generate a spare parts availability identifier; The estimated downtime for each maintenance strategy is calculated based on the maintenance time prediction model, and the time cost assessment results are generated. The alternative maintenance strategy set, spare parts availability identifier, and time cost assessment results are input into a multi-objective optimization algorithm to generate a priority-ranked maintenance strategy queue. Extract standard operating steps corresponding to the selected maintenance strategy from the operating procedure library of the knowledge model, and generate an augmented reality executable instruction set. The maintenance strategy queue is integrated with the augmented reality executable instruction set into a decision recommendation report.

8. An analysis device based on multimodal time series data, characterized in that, The analysis device based on multimodal time series data includes: The data acquisition and synchronization module is used to acquire multi-source heterogeneous data, and perform preprocessing and time synchronization operations on the multi-source heterogeneous data to obtain preprocessed and time-synchronized multimodal data, and organize the preprocessed and time-synchronized multimodal data into a multimodal time-series data stream in a unified format. The knowledge graph construction and modeling module is used to construct a domain knowledge graph and build a large knowledge model based on the domain knowledge graph. The feature extraction and fusion module is used to extract each single-modal feature from the multimodal time-series data stream and fuse the single-modal features to generate a unified multimodal time-series feature vector; The knowledge-enhanced time series analysis module is used to analyze the unified multimodal time series feature vector through a large time series model, and to integrate the domain knowledge provided by the large knowledge model into the analysis process of the large time series model to generate analysis results; The display and early warning module is used to display the analysis results and generate early warning information based on the analysis results; The decision generation module is used to combine the analysis results with the knowledge model to generate decision recommendations.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and an analysis program based on multimodal time series data stored in the memory and executable on the processor. When the analysis program based on multimodal time series data is executed by the processor, it implements the steps of the analysis method based on multimodal time series data as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores an analysis program based on multimodal time series data, which, when executed by a processor, implements the steps of the analysis method based on multimodal time series data as described in any one of claims 1-7.

Images